Project Estimation Presentation at DrupalCamps. Presentation discussing risks, methods and recommendations when estimating software estimation with a focus on web CMS (drupal)
2. Andy Kucharski
2
• CEO & Founder
• Promet Source:
• Drupal Development
& Support
• Chicago, IL.
3. “The typical software organization is not struggling to improve its estimates from +-10% to +-5%
accuracy. The typical software organization is struggling to avoid estimates that are incorrect by
100% or more.” -- Steve McConnell
What is this presentation about?
3
What is the presentation NOT about?
• Fundamentals (overlooked)
• Big Risk factors
• Selling your estimate (Team)
• 20%-100% over budget
• Specific estimates (content type)
• Magical formulas
• Fine tuning error from 20% to 5%
• Selling your estimate (client)
4. 4
Agenda
What is the Problem with Software Estimation?1
2
3
Avoiding Risk
Estimation Techniques
5. What is the Problem with Software Estimation?
5
6. The greatest discrepancy between what the client thinks, an executive thinks, and
a developer thinks is often the definition of ESTIMATE
6
7. 7
Estimates, Targets, and Commitments
1 Estimates
Prediction
• A Prediction of how
long a project will
take or how much it
will cost
2 Targets
Statement/Desire
• A Statement of a
desirable business
objective
3 Commitments
Promise
• A Promise to deliver
defined
functionality at a
specific level of
quality and by a
certain date
10. 10
The Solution
Architect
About your unicorn:
• Knows the most about
Drupal
• Experience in
Ecommerce
Migration
Integration
Site building & Theming
• Understands business value
• Can magically debug
anything
Things that may be missing:
How fast can a DEVELOPER do the work?
How much time will be spent training, instructing,
reviewing, and doing oversight?
Avoiding unfounded optimism
11. What is the definition of a “GOOD” estimate?
11
12. 90% confidence
12
After all, the process is called
Estimation, not Exactimation.
- Phillip Armour
What is the surface temperature of the
Sun? (Give a range in Fahrenheit)
Make your range wide enough
that you feel you are 90% confident
that your range includes the answer
10,000 Degrees F (9,941 F)
13. When given the option of a shorter average schedule with higher variability or a
longer average schedule with lower variability, 8 out of 10 executives will choose
the second option
13
14. 90% confidence
14
How confident is 90% confident?
The average number of correct
answers was 2.8. Only 2% of quiz
takers scored 8 or more correctly.
Most peoples intuitive sense
of 90% confidence is in fact
closer to 30% confidence.
16. 16
EXTERNAL FACTORS ARE OFTEN UNACCOUNTED FOR
LESS EXPERIENCED
STAFF
STAFF NOT READY
REQUIREMENTS
REMOVED
STAFF DIVERTED TO
SUPPORT OLD
PROJECT
UNSTABLE
FUNCTIONALITY
REMOVED
SERVER NOT READY
STAFF DIVERTED
TO SUPPORT
TRADE SHOW
REQUIREMENTS
ADDED
17. 17
Events that happen during the project nearly always invalidate the assumptions
that were used to estimate the project in the first place.
1
Functional
Assumptions
2
Staffing
Assumptions
3
Priority
Assumptions
4
Input
Assumptions
18. 18
Project
Dynamics
The activities of a late or compressed
project create destructive “late-
project” dynamics that make the
project even WORSE than nominal.
More
Status
Meetings
Frequent Re-
estimation
Apologizing to
key customers
Preparing
interim
releases
More
discussion on
priority &
requirements
Fixing
problems that
arise from
quick and
dirty
workarounds
Going back to
the contract
21. “Software does not have a neutral estimation problem. The industry data shows
clearly that the software industry has an underestimation problem.”
-- Steve McConnell
21
23. Major Areas of Risk
23
• What kind of content
type/field/thing
• Scope w/o specs
• Specific definitions
• Setup Costs
• Management Costs
• Other (non-dev)
Costs
• 90% confidence
• Best Estimator
• Terminology
• External Factors
• Lack of end-user
involvement or
production owner
• Timeline
• Poor wires/design
24. Consider your project’s current position within the cone of uncertainty, this will help
you gauge the level of accuracy in your estimate.
24
25. 25
The Cone of
Uncertainty
Initial Product
Concept
“I want a website”
Product
Definition
“I know the scope of
services”
Requirements
Complete
“I know the specs
of the project”
User Interface
Design
Complete
“I have wireframes”
Detailed Design
Complete
“I have full
documented specs
and design files”
26. Leverage the power of well defined products to move your way through the cone of
uncertainty
26
27. 27
Products
Training
Workshops
Drupal Concept Education as
an entry into the product
Discovery Process
Discovery
Technical and UI/UX planning
for a large scale development
projects
Evaluation of the inner
workings of a current
Drupal site, for upgrade
and migration planning
Audits
28. Know when you are in a chaos project and how to reset
28
29. Chaos Projects
How does BAD go to TERRIBLE?
29
External
Factors
Re-estimation
or Planning
Poor Project
Dynamics
Pure Chaos
“Chaos projects introduce variability for each following step. The answer is not to
continue to re-estimate but instead address each issue through project control.”
30. NEVER, let your estimator forget (or your sales team remove) these important
activities
30
33. Estimating “Things”
33
Lets assume that you are all
expert estimators and that all the
process and risk areas we have
spoken today you know
PERFECTLY how to avoid.
How many people are in this
room right now?
34. 34
APPROACHES TO ESTIMATING “THINGS”
The Hobbyist
“I’m guessing
75 people”
The Factor
Guy
“15 rows 5
per row”
The Percentage
Guy
“150 person
capacity and its
half full”
The Data Guy
“The last 3
sessions have
had XX”
Judgement Count Compute Calibration
35. Count, Compute, Judge
35
Count if at all possible.
Compute when you can’t count.
Use Judgement alone only
as a last resort.
COUNT (& Calibrate)
Find something to count that has a direct
correlation to what you are building1
COMPUTE
Multiply by a statistically
meaningful (proven) average2
JUDGE
Only make assumptions
as a final option (document)3
36. • Click to edit Master text styles
Items to Consider
• Size of Project 5-25 people, 25-50, etc.
• Sequential vs. Scrum
• Stage of project
36
37. 37
Common Items in these areas
1 2 3Count/Calibrate Compute Judge
Site Specs
Content types
Taxonomies
Menus
Views
Wireframes
Designs
Migration
“features”
Non-build items
QA
Deployment
Project Management
Training
Governance
Engagement Management
Adjustment Factors
Risk Multipliers
Contingency
Unknown
38. 38
Count &
Calibrate
1) Determine an item that’s
highly correlated with the
size of the software
2) COUNT how many you
have
3) CALIBRATE your count
with data from
- Industry Data
- Historical Data
- Project Data
Factor Factor Count Calibration Gross Estimate
# Content Types 8 4 32
How many hours for feature “X”?
39. 39
Compute
1) Determine a defined
value to base your
computation
2) Determine a multiplier
that is a statistically
meaningful average
3) COMPUTE subtotal for
your line item
Development Subtotal 1200 Hours
Subtotal PM % PM SubTotal
1200 20% 240
How many hours for Project Management?
40. 40
Judgement
1) JUDGE the rating of a
specific multiplier (factor)
2) Apply multipliers based
on rating of your estimate
3) Determine factors
influence on the total
project
4) COMPUTE
Site Build PM %
Gross
Estimate
Rating Influence
Net
Estimate
1200 240 1440 Low = 1.25 1.1 1980
Site Building Definition 2.0 1.25 1.0 .95 .85 .75 1.1
41. Overcoming Judgement
41
Judgement is the most
hazardous kind of estimation
due to subjectivity and
unfounded optimism.
Magnitude of Relative Error
MRE = (Actual Result - Estimated) / (Actual)
Expected Case
EC = (Best Case + (3 x Most Likely Case) + (2 x Worst Case)) / 6
42. Wide Delphi Method
• Coordinator presents each expert with a specification and an estimation form.
• Coordinator calls a group meeting in which the experts discuss estimation issues with the coordinator and each
other.
• Experts fill out forms anonymously.
• Coordinator prepares and distributes a summary of the estimates
• Coordinator calls a group meeting, specifically focusing on having the experts discuss points where their
estimates vary widely
• Experts fill out forms, again anonymously, and steps 4 to 6 are iterated for as many rounds as appropriate.
42
43. Subtitle for slide
Guide for Standardized Estimation Procedure
• Focus on counting and computing when possible, rather than using judgment - create tools to facilitate this and
make it easy
• Encourage multiple estimation approaches and comparison of results
• Define toll gates and encourage re-estimation over the course of a project
• Contains a clear description of an estimate’s inaccuracy
• Defines when an estimate can be used as the basis for a project budget
• Defines when an estimate can be used as the basis for internal and external commitments
• Calls for archiving estimation data and reviewing effectiveness of the procedure
• McConnell, Steve (2006-02-22). Software Estimation: Demystifying the Black Art (Developer Best Practices)
(Kindle Locations 5142-5149). Pearson Education. Kindle Edition.
43
45. Thank You
45
CHICAGO, ILLINOIS
1802 W. Berteau Ave. / Suite 209 / Chicago IL, 60613
773-525-8255
D.O / Twitter
@akucharski or @prometsource
Email:
andy@prometsource.com
Problems: Common misconceptions or thought provoking statements
Avoiding Risk: Lessons learned
Techniques: Very high overview of where to get started
This may be the number one issue that organizations have when internally discussing project costs
-- This may be the number one issue that organizations have when internally discussing project costs
-- understanding the differences in these terms can greatly reduce internal tension when it comes to pricing a project and setting developer expectations
“5 months! we need this ready in 3 months”
“Here is what we can deliver in 3 months”
“well i have to have this one piece you didnt include. Lets add that and settle on 4 months”
-- In fact, no one is going to be good at estimating something NEW
“well be more effective on this project than the last time we did this”
“A lot of things went wrong last project when it took us longer to do this item”
“We started the project slowly due to a learning curve, but now will have a faster velocity with the same team”
-- In fact, no one is going to be good at estimating something NEW
-- The best estimator is often the one with access to the most historical data and comparables
-- Activity time!
year Alexander the Great was born
Total volume of great lakes 5,493
Year and month of registration of 231557 - Feb 2008
-- Even more shocking!
-- Why dont we ask? Why do we continue to give handouts before they are asked for
-- Because we dont have these reasons we just discussed to arm ourselves in those conversations!
-- This test was given in many industries, many question types, many people
-- The results are staggering
-- 10 questions means you should have 9 right
-- Of the people with 7 or 8 right they claimed afterwards that their ranges are too wide, thats how they did well
-- They felt BAD about their estimate-- We are conditioned to believe that estimates expressed as narrow ranges are more accurate or appealing than estimates with wider ranges. We believe wider ranges make us appear ignorant or incompetent.
-- We are laughing because we KNOW these happen on almost all projects
-- Even if you do not pass all these costs onto the client, they MUST be accounted for
1Staff not ready when planned
2Less experienced staff than planned
3requirements removed
4staff divered to support old project
5staff diverted to support trade show
6more requirements added
7unstable funcality removed
8Server hardware not setup to support pre-prod testing
-- There isnt a 3rd party data lookup to support functionality. There isnt exact department definition of all content. There are events that start at 15 min after or 45 min after an hour.
-- SA got put on another project that closed first
-- supporting past project is higher priority
-- Wireframes delivered at a different time. Client didnt complete their rebranding in time for design phase
Use example of a project that has an issue, that starts to delay a project
Project Dynamics aren’t considered or are ignored --
-- if you cant see this
--- showing that most projects are delivered late, very few on time, and non ahead of schedule
--- shows that only 20% of projects are on time AND on budget, 50% are late or over, 30% FAIL
The Seattle Mariners’ new baseball stadium was estimated in 1995 to cost $ 250 million. It was finally completed in 1999 at a cost of $ 517 million— an estimation error of more than 100% (Withers 1999). The most massive cost overrun in recent times was probably Boston’s Big Dig highway construction project. Originally estimated to cost $ 2.6 billion, costs eventually totaled about $ 15 billion— an estimation error of more than 400% (Associated Press 2003). Of course, the software world has its own dramatic estimation problems. The Irish Personnel, Payroll and Related Systems (PPARS) system was cancelled after it overran its € 8.8 million system by € 140 million (The Irish Times 2005). The FBI’s Virtual Case File (VCF) project was shelved in March 2005 after costing $ 170 million and delivering only one-tenth of its planned capability (Arnone 2005). The software contractor for VCF complained that the FBI went through 5 different CIOs and 10 different project managers, not to mention 36 contract changes (Knorr 2005).
McConnell, Steve (2006-02-22). Software Estimation: Demystifying the Black Art (Developer Best Practices) (Kindle Locations 1191-1199). Pearson Education. Kindle Edition.
http://www.dispatch.com/content/stories/local/2014/12/06/daunting-drilling.html
Project to bore tunnel under Columbus faces $29.5 million cost overrun
The $26 million custom machine, like something out of science fiction, was built to drill through mostly dry bedrock. But city engineers’ estimate of water levels didn’t account for millions of gallons of water that travel underground, below the Scioto River.
Instead of grinding through dry rock, Marsha was choking on a water/rock/mud mixture called slurry, which is kind of like wet concrete.
-- Let this sink in
-- All the data out there CLEARLY supports that we are drastically underestimating our projects
**** this doesnt mean we have to price differently
-- we need this info for expectation setting, true costing, and predicting how things are going to go
-- ideally we would also learn to pass on some of these costs to the client
-- discussed #1 -- Estimation process is a large portion of the risk of estimation.
(that is why we spent the most time just talking about it)
-- Would like to move into a few of the other areas in less detail and discuss some lessons learned
This is about Project Info or project specs
This is about Project Info or project specs
This is also about project info
this is also about project info
-- this is a great way to NARROW your cone of uncertainty through paid services/products
-- get paid to lower your estimation risk
Project chaos
-- discussed partially earlier regarding project dynamics and external factors
-- “we are past the hard part now our velocity will increase”
-- Understanding how to “pause and restart” is key to keeping your estimates meaningful
-- AVOID THE SPIRAL “how the heck did we get here” at the end of your project
Omitted activities is a very common mistake
-- even worse is when they are REMOVED
-- Dont reduce your estimators estimate, we just proved before that it is OFTEN already too optimistic
Omitted activities is a very common mistake
-- even worse is when they are REMOVED
-- Dont reduce your estimators estimate, we just proved before that it is OFTEN already too optimistic
-- doesnt mean you have to have the client pay for these all
72% of software projects were reported to be done by “expert opinion” also known as judgement
https://en.wikipedia.org/wiki/Wideband_delphi
EVALUATIONS
Calibraion - discuss the example of similar project
T-Shirt Sizing
-- The COST of underestimation is exponentially worse than the COST of overestimation (parkinsons Law -- is a predicable loss)