1. IT BEST PRACTICES, WHITE PAPERS/INTERVIEWS
Focus on Patrick O’Toole , Process Improvement Guru
Matthew Kabik | September 20, 2012 |
Could you tell us a little bit about yourself, your
background and what you are working on today.
PATRICK O’TOOLE: I started out in IT back in the late 70’s. During that time period I
worked as a systems analyst, project manager, and ultimately program manager. I then
made a career switch and assumed more of a systems engineering role working on high-
speed imaging products that were being released to the financial services marketplace. By
the early 90’s, we were capturing images of checks at 30 images or 30 checks per second,
which was pretty high speed at that time. We were counting on Intel’s 486 chips to be
released on time so we could actually do the processing – that’s how long ago this was.
After that, I decided that I really wanted to consult in the area of project management, so
I left the company I was with and started up my own consulting company. I found that I
could get jobs as a contractual project manager; nevertheless, I wasn’t really doing
consulting in the area of project management. Then one day I attended a conference at the
Software Engineering Institute that focused on risk management. I was fortunate enough
to be seated next to the president of a risk management consulting company and we
stayed in touch over the Internet. About three years later, he gave me a call and asked if I
was interested in working with him. That’s when I joined Teraquest, which is Bill
Curtis’s company. Bill is one of the authors of the CMM for software.
I worked with Teraquest for about 4 and a half years, became a lead appraiser and
worked with several organizations to help them achieve a higher level of maturity.
However, when I noticed that I was spending 95 percent of my time on the road, I
decided that with 2 small kids at home I had better make an adjustment. So I gave
Teraquest about a year’s notice, made a separation plan, and started up my own company
in 2000. Since that time, I have essentially been focusing on training, assessing, and
consulting primarily in the area of CMMI, essentially same things that I was doing with
Teraquest.
Can you talk a little bit about your status as a visiting
scientist at the SEI?
PATRICK O’TOOLE: Essentially, the SEI uses part-time employees to help with
certain areas of specialization. I teach the intermediate concepts course which is a 5 day
upper level CMMI course that can only be offered by the SEI. It’s a simple pass fail class

2. that is a prerequisite for becoming an authorized lead appraiser or CMMI instructor. I
also teach the intro course on behalf of the SEI. In fact, I just spent two weeks teaching
this class in Brazil. Another thing that I do at the SEI is observe first time instructors.
Could you tell us a little bit about the historical
background of the CMM and the CMMI – how it
started, how it evolved and any changes that are
currently taking place today?
PATRICK O’TOOLE: The story, as I’ve been told it, is that Watts Humphrey went to
Phil Crosby’s quality college down in Florida, and as he was heading back home to
Pittsburgh he thought, “How can we apply something like the Crosby model, which is a
5-scale model, to the wonderful world of software?” So on the proverbial back of an
envelope, he sketched out the 5 levels of the Crosby scale and what they would mean in
terms of software, and this ultimately evolved into the CMM.
The CMM is simply a collection of good practices that organizations have found to be
successful. When they were putting the model together, they invited several highquality
software development organizations to congregate in Pittsburgh. They all sat around in a
conference room and wrote the practices that had made them successful on yellow
stickies. After they pasted the walls with these things, they went through an infinity
diagram exercise and clumped them into groups. They found a large group of project
management practices, a large group of engineering practices and a large group of
support type activities. These groups ultimately became the key process series.
That resulting set of practices ultimately turned into the CMM for software, which served
many companies very well for several years. The CMM was primarily a software model
because companies were finding that they were having problems not so much with
hardware, but more with software. They were more and more dependent on software and
software was increasingly letting them down. The need for software improvement was
what ultimately led to the funding of the SEI and the development of the CMM for
software.
After several years of use, the CMM started getting crusty around the edges. It was a
good first attempt at a model, but the SEI decided that they needed to upgrade it and
began revising it and developing the CMM for Software. The other thing that was
happening was that more and more maturity models were being produced, including a
maturity model for system engineering and another one for people. At one point, the SEI
claims they had 17 maturity models either published or on the drawing board to be
developed.
In response to this, the Department of Defense said, “Wait a minute; every time the SEI
releases a new model we have to train people on it and we need another appraisal method
and another appraisal to go through. Let’s just get it down to one model.” That’s when

3. the SEI determined that it was time to do a consolidation. They brought together three
models: the CMM for software, the System Engineering CMM, and the IPTD that was
just about to be released (the integrated product team CMM). They consolidated these
three into what is now known as the CMMI, which was published in early 2000.
The CMMI went through a couple of trial periods, then ultimately fell into version 1.1,
which became the official release for the model. Last August, it went through another
fairly significant revision and now we’re up to version 1.2.
What is your opinion of the progress that has been
made to date in terms of CMMI penetration in large
organizations?
PATRICK O’TOOLE: I think one of the things we have found, especially in the
Department of Defense area, is that the CMMI is almost a rite of passage. Organizations
have to adopt the model, because the Department of Defense gives preferred nation status
to any organization that has achieved maturity level 2 or maturity level 3, depending on
what kind of software or engineering systems need to be developed. The use of the
CMMI has now extended into hardware engineering, system engineering, electrical
engineering and mechanical engineering. At this point, any project oriented engineering
activity can use the CMMI to help guide its activities. The Department of Defense is still
on the leading edge of pushing it, but it’s also expanding into other areas. For example,
the automotive industry is very adamant that software suppliers and other engineering
suppliers achieve a certain level of maturity, or at least get on the path to doing so.
General Motors has a requirement for their suppliers to be at maturity level 2 by a certain
date and maturity level 3 by another date. Chrysler has done the same thing, but it has
taken a slightly different approach. Rather than focusing on maturity levels, which is a
staged representation of the model, Chrysler focused on the continuous representation of
the model. They take the process areas they believe benefit them as a customer the most,
and then they insist that their suppliers achieve certain capability levels in just those
process areas. In some respects, I think this is a good trend because it’s pushing
organizations to do what they should be doing on their own. In other respects, however, I
think it’s a bad trend because engineering organizations will wind up striving for the
number, not necessarily for the improvement. Many organizations approach CMMI-
based initiatives thinking, ”Let’s do anything we can to get to level 3, whether or not we
achieve any improvement.”
We hear a lot about Six Sigma, ITIL and of course the
CMM. How do you differentiate between these different
improvement programs and models and how do
organizations know where they should be putting their

4. energies? What, if any, is the interrelationship between
all of these things?
PATRICK O’TOOLE: There are a lot of models out there and it can be very confusing
to people. If you’re doing data center or maintenance type support – a kind of flow
activity where it’s not really a project oriented activity – then ITIL is probably the proper
choice. Six Sigma can be very beneficial, but it suffers from some of the problems that
we see in the low maturity organizations. That is, if they are not doing things in a
reasonably consistent and repeatable way, Six Sigma can be applied, and can even be
beneficial, but the real benefit won’t materialize until a greater consistency is achieved.
My typical recommendation is to get to CMMI maturity level 2, stabilize your projects,
and then start thinking about how you might be able to apply a little more sophistication
to take those stable projects to the next level. At this point, I think Six Sigma and CMMI
work very nicely hand-in-hand, especially at higher maturity levels. And once you are
looking at maturity level 4 and 5, Six Sigma is a perfectly natural fit. The one thing I
would suggest to organizations that have multiple quality initiatives going – Six Sigma,
CMMI and ITIL – is that they make sure to pick one as the master. I don’t care which one
it is, but you’re going to get into bizarre turf wars over various quality initiatives if you
do not designate a primary initiative. For instance, you might say, “We’re doing Six
Sigma, and underneath that umbrella we’ve got an ITIL portion that we have applied to
the flow activities and a CMMI portion going on for the projectoriented activities, but it’s
all being done under the umbrella of Six Sigma.” In other words, one intiative is master
and the others are subbordinate to it.
For those organizations that have had a reasonably
successful CMM program in place, how would you
characterize their return on investment?
PATRICK O’TOOLE: That is always a tricky question, because it is kind of a
selfselecting group. Who is going to report their data? Well, it is going to be the
organizations that had glorious success. Nevertheless, the numbers that they show are
compelling. They can show five-to-one payback – for every dollar invested you get a five
dollar return – and that is very good.
I think the ones that have the data and are reporting it are being honest; nevertheless, they
are self-selected and that is something to keep in mind. What I would be more interested
in finding out is what occurred within those organizations that have not achieved success.
What have been the failure points? Why did they abandon the effort or settle for less? I
think part of the issue, and this is addressed a little bit in the CMMI, is that when
organizations measure the benefits achieved or the progress that they’ve made, they
measure this through the accomplishment of maturity levels. But maturity levels are
nothing more than a recognition that you’ve achieved a certain level of process maturity.
They do not reflect anything about the value that you have derived from that stabilization.

5. Organizations would be well served to say, “In order for us to be successful in the
marketplace, we need to be a high quality producer, so the process we’re going to build
and the CMMI that we’re going to use as a guideline will have to be focused on
achieving high quality.” Too often organizations say, “Let’s get to maturity level 3,” and
they don’t consider what they are really trying to achieve as a business.
There are some statistics out there that suggest that
80% of IT money is spent on maintaining existing
systems and infrastructure, whereas 20% of the total
investment goes toward development. To your
knowledge, is the SEI addressing maintenance at all? Is
there any possibility that we might see a CMM for
maintenance in the future?
PATRICK O’TOOLE: Well, the CMMI that was released in August was called CMMI
for development, so this presupposes that the SEI is recognizing the need for different
flavors, or what they call constellations. The CMMI for development is the first of three
that they have planned, the other two being the CMMI for services and the CMMI for
acquisitions. To the best of my knowledge, though, they do not have maintenance on the
drawing board yet. A lot of organizations treat their maintenance projects. Consequently,
you would leverage the CMMI for development, because teh CMMI for development is
designed for project-oriented development involving things like minor enhancements.
I think that the thinking around maintenance is that if we were to develop higher quality
products in our development activities and if we were to support these with better
documentation, better designs, and better architectures, then the maintenance effort over
the long term would naturally decrease. So rather than trying to fix the systems that we
already feel are eating up 80% of our resources, the best we can hope for is to reduce
those same mistakes in the future and to release higher quality, more maintainable, more
supportable products in the beginning of the lifecycle.
For organizations that are undertaking the CMMI and
implementing it, what are the critical success factors?
PATRICK O’TOOLE: I think something that really helps a CMMI implementation be
successful is the burning platform: the last release that went out was a major disaster, it
blew up in the field, the customers were irate and it resulted in lawsuits. That failure
motivates an organization to say, “We can’t keep doing things the way we have been
doing them. It isn’t working.” Barring that, I have actually recommended to clients that
they would be well served to have a major disaster, because that is what they need to
convince the workforce that things really need to change. They usually don’t listen to that
kind of advice, however.

6. Other than that, strong management support is crucial. And this goes beyond just
sponsorship. To me sponsorship means we put a team together, we fund the team and
after a month we get together with the team and listen to what they have to say. My
preference, however, is that have the executives actively get involved. I want them to
document some of their processes and to ensure that they are following their processes
and make sure that they are improving them over time. I want the executive who can
stand up and say, “Look, I’m becoming more process disciplined. It appears to be
working for me and my team, therefore I think it would be beneficial to the engineers and
the project managers as well.” We need executives who can stand up and lead by
example.
For organizations that are attempting to work with the
CMM and are having problems, could you outline some
of the biggest pitfalls and some of the most common
mistakes made?
PATRICK O’TOOLE: I think the biggest problem is the level. They may start off well
and say, “We really need to improve and we’re using the CMM as a guidebook to help us
do that.” Then they go at it for six months, maybe twelve months, and because the
changes are gradual, the benefits that they’re achieving is not really evident unless they
step back and assess how far they have come. If they can take that longer term
perspective they will see that things are getting better. But at some point the executive
will say, “This is going way too slow. I know we’re in it for improvement, but darn if we
are going to make level 2 six months from now.” And so the pendulum swings from
doing it for the right reasons to getting to the level. The level unfortunately becomes the
end all and be all, and whatever process improvement group you have in place starts
putting their energies into generating templates and forms to show that they have
achieved the level, rather than staying focused on gaining the day-to-day value. When I
went to college, I didn’t get smarter the day I was handed my diploma. I got smarter over
the four years that I was at the University, went to the classes and did the homework.
Achieving the diploma was just the recognition of the accomplishment; it was not the
accomplishment itself.
What advice would you give to a Level 1 organization if
they had 9-12 months to demonstrate value through a
process improvement effort? What should they be
focusing on and how should they undertake this?
PATRICK O’TOOLE: I would take two different perspectives on the work being
performed. The first question I would ask is, “Where’s the pain?” In other words, what
are the problems the organization is facing and how can we use the CMMI as a medical

7. reference book to find practices that we believe will address the pain? After that I would
ask, “Where do you need to be successful?”
For example, if you’re trying to achieve high quality in order to be successful, why not
look at the CMMI for practices that will lead to higher quality? Peer reviews are one
example of a practice that will lead to higher quality, but they happen to be staged at
maturity level 3. But who cares? If peer reviews are going to lead to higher quality, and
higher quality is what is going to make you successful, you should get implemented as
early as you can in order to get the value as quickly as possible.
Is there anything else that you’d like to talk about, any
area that we didn’t cover?
PATRICK O’TOOLE: The one final thing I want to comment on is the appraisal
process. The SEI has now come out with a family of appraisal methods. There are class A
methods, where you’re actually going for the gold, but there are also class B and C
methods that are less rigourous. We typically use class B and C methods for gap analysis
or progress analysis. These kinds of less rigourous appraisal methods can really be used
as a health check. We can not only look for compliance to the model, which I think is a
rather bizarre term anyway, but we can also determine how well the practices of the
model are being adopted by the organization. An organization is trying to become the
high quality producer. Are they achieving that? Is the organization achieving that? Let’s
use the appraisal method not just to look for conformance to the CMMI, but also to
explore personnel’s ideas on how they can achieve higher quality and win in the
marketplace.
I think appraisals are an underutilized tool. Most organizations look to them to determine
whether or not they have hit their number. But I think there is a whole other use for these
things. They’re expensive, they’re time consuming and they’re somewhat disruptive, but
we’ve got to find a way to optimize their value.
Biography of Patrick O’Toole
Pat O’Toole is the Principal Consultant at Process Assessment, Consulting & Training
(PACT) where he provides a full range of services to his process improvement clients.
Pat is one of the most active CMMI lead appraisers, and has led appraisals spanning all
maturity levels, including one of the largest and most complex CMM Level 5 assessment
conducted to date. He is an SEI authorized instructor for the “Intro to CMMI” course who
has taught this course more than 40 times in 6 countries. Pat is a Visiting Scientist at the
SEI, and teaches the “Intermediate Concepts of CMMI” course on their behalf.