Presentation delivered at the Open Government and Innovations (OGI) Conference in Washington, DC, on July 22, 2009. Outlines the ways in which government has measured its web presence in a "1.0" context, including an overview of the measurement activities conducted by Brookings Institution, Foresee, Forrester and the e-Government Act of 2002.

1. Measuring Gov 2.0
(But First, Lessons from Gov 1.0!)
Andrew Krzmarzick
Senior Project Coordinator
The Graduate School
http://www.graduateschool.edu
http://generationshift.blogspot.com
Twitter: @krazykriz

2. AGENDA
 Gov 1.0 Measurement
 Advancing Social Media Survey
 Gov 2.0 Measurement Hallmarks

3. 3

4. “In many ways,
the Internet
more resembles
an ancient
bazaar…”
4

5. 5

6. Ordinary people…possess information –
serious, expert, fact-based,
scientific information.
[They] are ready and willing to share
that information across geographic,
disciplinary, and institutional boundaries.
- Beth Noveck, Wiki Government
6

8. GOV 1.0 MEASUREMENT
1. Brookings
2. Forrester
3. OMB’s E-Gov Initiative
4. ForeSee

9. GOV 1.0 MEASUREMENT
1. Brookings Institution E-Government Study
Since 2000:
• Reviewed > 1,500 State / Federal Websites
•0 – 100 Point Scale

10. GOV 1.0 MEASUREMENT
1. Brookings Institution E-Government Study
4 points
4 points +1
4 points
4 points +2
4 points per feature (72 total possible points) + 28 more for frequency

11. GOV 1.0 MEASUREMENT
1. Brookings Institution E-Government Study
You got it…
or you don’t.

12. GOV 1.0 MEASUREMENT
1. Brookings Institution E-Government Study
Applied to Gov 2.0:
Should we count
the number of social
media tools an
agency deploys?

13. GOV 1.0 MEASUREMENT
2. Forrester Website Benchmark Survey
User Goals Point Scale
2 = Strong Pass (Best Practice)
 Value
1 = Pass (No problems)
 Navigation -1 = Fail (One major or many minor problems)
-2 = Strong Fail (2+ Major…)
 Presentation
 Trust

14. GOV 1.0 MEASUREMENT
2. Forrester Website Benchmark Survey
Value
• Is essential content available where needed?
• Is essential function available where needed?
• Are essential content and function given priority in display?

15. GOV 1.0 MEASUREMENT
2. Forrester Website Benchmark Survey
Navigation
• Are menu categories/sub-categories clear?
• Is wording in hyperlinks clear and informative?
• Are keyword searches comprehensive and precise?

16. GOV 1.0 MEASUREMENT
2. Forrester Website Benchmark Survey
Presentation
• Does site use language that’s easy to understand?
• Does the site use graphics, icons, and symbols “ “ ?
• Is text legible? Text format/layout support easy scanning?
• Are form fields and interactive elements placed well?

17. GOV 1.0 MEASUREMENT
2. Forrester Website Benchmark Survey
Trust
• Does the site present privacy and security policies?
• Do location cues orient the user?
• Is contextual help available at key points?
• Does the site help users avoid and recover from errors?

18. GOV 1.0 MEASUREMENT
2. Forrester Website Benchmark Survey
Trust the
experts?

19. GOV 1.0 MEASUREMENT
2. Forrester Website Benchmark Survey
Applied to Gov 2.0
Blog Review Tool?
Tool Placement?

20. GOV 1.0 MEASUREMENT
3. OMB E-Government Initiatives
E-Government Act of 2002
“a broad framework of measures
that require using Internet-based IT
to enhance citizen access
to Government information and services...”

21. WEB 1.0 MEASUREMENT
3. OMB E-Gov Initiatives

22. WEB 1.0 MEASUREMENT
3. OMB E-Gov Initiatives
• Business.gov
• Regulations.gov As of January 2009:
• USALearning.gov • 28/28 agencies had
• Grants.gov implementation plans
• FedBizOpps.gov • 87% of milestones met
• Recreation.gov
• USAJOBS.gov

23. WEB 1.0 MEASUREMENT
3. OMB E-Gov Initiatives
d) Improve E-Gov Initiatives Performance Measures
• Adoption/Participation – Is the relevant community participating?
• Usage – What’s the level of use by the target community?
• Customer Satisfaction – Is the community satisfied w/ products/services?
• Cost Savings/Avoidance – What’s $ value for government /citizens?
• Efficiency - Any decreases in time and/or increases in productivity?

24. WEB 1.0 MEASUREMENT
3. OMB E-Gov Initiatives
Peer
Pressure?

25. GOV 2.0
DASHBOARD?

26. GOV 2.0 DASHBOARD?

27. GOV 1.0 MEASUREMENT
4. ForeSee: E-Government Satisfaction Index

28. GOV 1.0 MEASUREMENT
4. ForeSee: E-Government Satisfaction Index
E-Gov – What is it?
• Key question:
• Measures 94 e-gov sites on:
How satisfied are citizens?
1. Navigation
• University of Michigan’s
2. Functionality
American Customer
3. Search
Satisfaction Index (ACSI)
4. Look and Feel

29. GOV 1.0 MEASUREMENT
4. ForeSee: E-Government Satisfaction Index
E-Gov Outcomes as of Q1 2009
• All-time high: 74.1% satisfaction overall
• Most successful sites:
• Citizens find information quickly and easily
• E-Commerce and Transaction functions

30. WEB 1.0 MEASUREMENT
3. ForeSee: E-Government Satisfaction Index
E-Gov Top Performers
• http://www.ssa.gov/estimator (Score: 89)
• http://medlineplus.gov (Score: 86)
• http://www.niams.nih.gov/index.htm (Score: 82)
• http://www.cia.gov/employment (Score: 81)

31. WEB 1.0 MEASUREMENT
4. ForeSee: E-Government Satisfaction Index
The
customers’
always
right?

32. WEB 1.0 MEASUREMENT
4. ForeSee: E-Government Satisfaction Index
Applied to Gov 2.o
Add/Alter a series of
questions related to
social media?
Real-time polling data
(shared right away)?

33. “ADVANCING SOCIAL MEDIA”
SURVEY
• 10 questions
• 4,000 potential respondents
• Web Manager’s Forum
• GovLoop
• GovTwit Directory
•International Contacts
• 105 responses
• 7 countries

34. “ADVANCING SOCIAL MEDIA” SURVEY
Questions
1. What enables or hinders you from using social media?
2. What social media tools does your agency use?
3. Rate tools per value/importance in achieving mission.
1. Do you establish metrics prior to implementation?
1. If yes, for which tools and what variables do you measure?
2. Is privacy, security and monitoring social tools important?
3. How often do you use social media in your job?
4. Thoughts re: gov standards w/browsers, software, etc.?
5. Is CTO/CIO actively involved in social media initiatives?
6. Where are you from/what agency do you represent?

35. Advancing Social Media in Government
Do you establish metrics prior to implementing any of the
above social media tools at your agency?
Answer Options Response Response
Frequency Count
Yes 43.8% 46
No 56.2% 59
answered question 105
skipped question 0

36. “ADVANCING SOCIAL MEDIA” SURVEY
“…we don't have benchmark data,
nor do we have measurable objectives
for any of our tools.
We basically just put them out there
and hope they work.
It's kind of annoying.”

37. Advancing Social Media in Government
If you answered YES for #4, what tools are you measuring?
Answer Options Response Response
Frequency Count
1 Blogs 60.4% 29
2 E-mail 58.3% 28
3 Twitter 56.3% 27
4 RSS 43.8% 21
5 YouTube 33.3% 16
6 Facebook 31.3% 15
7 Podcasts 25.0% 12
8 Mobile devices 16.7% 8
9 Wikis 14.6% 7
10 Delicious 8.3% 4
11 Z-other social networking (e.g. Flickr, Govloop, LinkedIn, Ning) 8.3% 4
12 LinkedIn 6.3% 3
13 Z-other video sharing (e.g. Hulu, Vimeo, Viddler) 6.3% 3
14 Z-other social bookmarking (e.g. Digg, StumbleUpon) 4.2% 2
15 Gaming 2.1% 1
16 MySpace 2.1% 1
17 Second Life 0.0% 0
18 Z-other virtual worlds/3D Web (e.g. YooWalk) 0.0% 0
answered question 48

38. GENERAL BLOGS
• Views • Posts
 Number of views/page • Views
PODCASTS
• Comments (quantitative)
 Time on site/page • Subscriptions
• People
• Visitors • Downloads
Google • Complaints
• Visits
 Unique Analytics • Comments (qualitative)
 Overall • Subscriptions
 Paths to site • Paths
• Links elsewhere?
 Paths on site
WIKIS
 Geographic distribution •Users
•Adoption rate RSS
 Searches
• Subscriptions
 Satisfaction •Edits
 Comments
TWITTER
•Followers
•Growth
YOUTUBE/VIMEO •Link click-throughs (when, what types of
• Views content)
• Comments •Retweets
• Downloads •Rankings (Twinfluence, Twitter Grader)
• Page placement traffic •Comparison to similar organizations
• Ratings •Friends
• Click-throughs •Conversations

39. “ADVANCING SOCIAL MEDIA” SURVEY
“Metrics only tell part of the story...
it is difficult to directly measure
influence…”

40. 40

41. TRANSPARENCY

42. ENGAGEMENT

44. For More Information:
Andrew Krzmarzick
Senior Project Coordinator
(202) 821-6288
Andrew_Krzmarzick@grad.usda.gov
Thank You! GraduateSchool.edu
GenerationShift.blogspot.com
IAmPublicsService.org
Twitter.com/krazykriz
LinkedIn.com/in/AndrewKrzmarzick
Delicious.com/akrzmarzick
Slideshare.net/akrzmarzick
44

45. RESOURCES
1. Ari Herzog: http://www.ariwriter.com, @ariherzog
2. Andrew Krzmarzick: http://generationshift.blogspot.com, @krazykriz
3. Brookings Institution E-Government Study: http://snurl.com/crpxn
4. Forrester Website Benchmark Survey: http://www.forrester.com/cxpbenchmark
5. ForeSee: E-Government Satisfaction Index: http://snurl.com/crpyu
6. OMB E-Gov Initiative and Report: http://www.whitehouse.gov/omb/e-gov/
7. Social Media and Government (Jeffrey Levy): http://snurl.com/crq0x
8. Federal Web Manager’s “Putting Citizens First:” http://snurl.com/crndj
9. IBM’s “Leveraging Web 2.0 in Government”: http://snurl.com/crq3e
10. Air Force Blog Assessment: http://is.gd/eAYo