1. Codelescence:
Engineering Comes
of Age
Data-Driven
Government
in Action
Socrata
2013 Highlights
Designing the Future:
Code for America
Makes an Impact
WINTER 2014
Fixing the Federal IT
Procurement Process

2. WINTER 2014
Editor-in-Chief
Bridget Quigg
Managing Editor
Alida Moore
Dear Fellow Innovator,
Contributing Editors
Tim Cashman
Patrick Hasseries
Thank you for picking up our Winter 2014 edition of Open Innovation magazine. In this issue
we take a look back at the most exciting year in open data history, and share some thoughts on
what’s to come in 2014.
As part of our look back, we’ll review the civic hacking events and conferences Socrata had
the honor of attending and awards our customers received. Were you part of the National Day
of Civic Hacking? We highlight the group that played a central role in its success, Code for
America.
16 Fixing the Federal IT Procurement Process
By Kevin Merritt
OPEN DATA IN FOCUS
Looking more closely at stories from the open data movement, Socrata Director of GovStat
Beth Blauer features three government organizations that are bringing more data to their
decisions. And, Socrata VP of Worldwide Markets, Erika Smith, gives us a snapshot of how
open data is expanding in Europe.
Thanks to the exponential growth of the movement, there are important issues the open data
community needs to address. Socrata Director of Open Data Ian Kalin calls for establishing
data standards and soon, while longtime thought leader David Eaves reminds us to engage in
the debate on how open data will be put to use in organizations around the world. And, you’ll
hear my suggestions for cleaning up the broken federal government procurement process.
For the tech enthusiasts in the group, Socrata’s Director of Product, Ben McInnis describes
how the demands on and uses of open data are evolving quickly towards bigger datasets, faster
data movement, and even greater accessibility. Data artist Thomas Levine asks us to consider
the benefits of open data metadata analysis. And, Chris Whong, a Socrata Data Solutions
Architect and geodata specialist, describes how OpenStreetMap is helping data flow from the
citizens of New York back into the city’s data stores.
4 Why We Need Open Data
Standards Right Now
By Ian Kalin
22 Designing the Future: Code for
America Makes an Impact
By Alida Moore
Promotion
Steven Gottlieb
Published By
Socrata
83 S. King Street
Seattle, Wa. 98104
info@socrata.com
(206) 340-8008
www.socrata.com
8 The Importance of Engaged
Open Data Advocates
By David Eaves
28 Data-Driven Government
in Action
10 Open Data in Europe Leaps
Beyond Transparency
By Erika Smith
By Beth Blauer
12 Making the Two-Way Street
of Open Data a Reality
By Chris Whong
I am honored to be a part of the exciting time in the open data movement. I hope you enjoy the
magazine and wish you the best in 2014.
Kevin Merritt
Socrata Founder and CEO
6 Bigger, Faster, Broader: How
Open Data Use Is Changing
By Ben McInnis
Plus, you get to meet our engineering team, and learn about their hobbies and habits that keep
our office full of home-brewed beer and hand-built electronics.
Sincerely,
Design/Art Direction
Corey Smith
14 Open Data Had Better Be
Data-Driven
By Thomas Levine
34 Codelescence: Engineering
Comes of Age
By Patrick Hasseries
Subscribe to future issues of Open Innovation by going to www.socrata.com/magazine

3. OPEN DATA
IN
FOCUS
OPEN DATA IN FOCUS
In today’s world of data standards, we
are operating in pre-mainstream chaos,
similar to that which existed with
electricity in Edison and Tesla’s time.
The potential for data-as-a-fuel exists as
a natural resource in much the same
way as magnets and copper did in the
industrial revolution. As this next wave
of economic opportunity takes root, the
same kind of arguments are being made.
Why We Need
Open Data
Standards
Right Now
GREAT THINGS ARE POSSIBLE
WITH OPEN DATA
By Ian Kalin
Socrata Director of Open Data
As the open data movement gains
momentum, more organizations, businesses,
and citizens are looking to share data and
collaborate on projects. What is essential
to them doing so? Data standards. Data
standards are a topic that deserves our
community’s full attention right now. The
government innovation movement must
address their importance across industries,
across borders, and even between
departments.
4
OPEN INNOVATION
•
WINTER 2014
THE DIFFERENCE
STANDARDS MAKE
First, let’s discuss the importance of
standards in familiar industries. Imagine
if the electrified world we live in today
had no standards; you might have to
replace your toaster every time you move
to a new house. When electricity first
made its way into the households of the
world, standards did not yet exist. To
that point, two of the world’s greatest
inventors, Nikola Tesla and Thomas
Edison, became engaged in an epic debate
between alternating-current versus directcurrent. Each argument had strengths and
weaknesses but Tesla made the innovative
choice to back his argument up with
household appliances, guaranteeing his
victory in the debate and establishing our
universal outlets today.
We benefit from technical standards
daily. They allow drivers to purchase gas
from any station without fear of using
incompatible fuels. They allow a text
message from a Verizon phone to be
transmitted to an AT&T phone. Data
standards operate in a similar manner,
but their use within modern applications
is relatively new. The most popular one
in use today is the General Transit Feed
Specification (GTFS). GTFS allows users of
Google Maps to know when the next bus is
going to arrive within participating cities.
Looking to evolving data standards that
deserve greater adoption, a great example
at the civic level is HouseFacts. There
are many businesses that help people
very different formats. The solution is
a uniform format for reporting things
like asbestos, pest infestations, and
even abusive landlords. As standards
like HouseFacts are adopted by city
governments, businesses can aggregate the
data that most cities are already collecting
As open data moves into our mainstream
world, making it as available as possible, in a
consistent, efficient manner, is essential.
buy and sell homes (e.g. Trulia, Zillow,
most commercial banks, etc.), but the
information on the safety and health of
those homes is messy and reported in
and integrate that information into the
websites people are using to make housing
decisions. The map for standard adoption
becomes a map for business growth.
The challenge lies in developing these
standards. Development can be difficult
because of the complexity of the data. Data
reflects the world in which it is created.
The maintenance alone of engineered
standards is hard, as demonstrated by the
huge network of organizations that deal
with standards, like the International
Organization for Standardization (ISO),
the American National Standards Institute
(ANSI), and UL.
As open data moves into our mainstream
world, making it as available as possible, in
a consistent, efficient manner, is essential. I
encourage you to engage in the discussion
of defined data standards so that public,
open data truly serves the world of citizens
who own it.
OPEN INNOVATION
•
WINTER 2014
5

4. OPEN DATA IN FOCUS
OPEN DATA IN FOCUS
Bigger, Faster,
Broader: How
Open Data Use
Is Changing
By Ben McInnis
Socrata Director of Product
Imagine a state that tracks K-8 student consumption
of free and reduced lunch and puts that data
alongside test scores and attendance information.
Imagine this data is then summarized in an internal
dashboard and sent to teachers via a mobile app.
In this scenario, millions of data points updated
regularly could help with resource allocation on a
monthly, weekly, or even daily basis.
T
his way of tracking and
reporting on school programs
via web and mobile apps is just
one example of the many that
will emerge as government organizations
offer more public data as open data.
sites, and decision support systems for
government.” says Socrata VP of Product
Saf Rabah.
Having hosted millions and millions of
rows of public data since 2007, Socrata
has seen data use change dramatically,
particularly over the last two years. Here
are three of the strongest trends we’ve
noticed.
BIGGER DATA
Two years ago, the average dataset stored
on Socrata was under 10,000 rows. Today’s
common datasets fall somewhere between
10 - 40 million rows. Socrata stores
datasets as large as 100,000,000 rows and,
by end of 2014, we expect to store datasets
of 1 billion rows or more.
•
WINTER 2014
As data scales in size, two interesting
things happen. First, data becomes
more useful for indirect and unforeseen
purposes because, within larger datasets,
common keys are more likely to exist.
Second, with more and more correlated
records across datasets, the need to
precisely clean and format data decreases.
“We’re evolving past the catalog phase of
open data and into the data-as-a-platform
era, where value is measured by the realtime flow and distribution of data through
application ecosystems, consumer internet
OPEN INNOVATION
Data Served by Socrata 2011-2013
(polynomial trend line)
Businesses, governments, and citizens
now use open data on a daily basis, and
they want to look at as much of it at
once as possible. For example, the City
of Chicago offers a traffic dataset that is
more than one million rows long. And,
San Francisco’s app showing parking
availability is updated in near-real-time.
Since its founding in 2007, Socrata has
seen open data move in larger quantities,
faster, and to more places than ever before.
6
and manage their consumption, public
officials accurately plan for capacity
needs and seasonal variance, and utility
personnel pinpoint leaks and service
interruptions.
Because of these characteristics,
analysis of large datasets can often yield
surprising and profound results. For
example, the City of Chicago’s analytics
team discovered a relationship between
A study by technology market research
firm Hurwitz & Associates titled “The
Benefits of APIs in the App Economy”
found that organizations offering APIs,
as opposed to those that didn’t, increased
customer reach by 70 percent, number of
apps created by 50 percent, and number of
mobile platforms supported by 58 percent.
2011
2012
streetlight outages and petty theft-correlations that were always latent within
their massive 311 and crime datasets.
While the connection might have seemed
obvious, having the data to clearly
understand this relationship helped the
city strategically address the problem.
2013
synchronized to optimize traffic flows,
and less obvious connections can drive
business efficiencies. A coffee vendor
might synchronize the preparation of fresh
coffee and warm pastries to correspond
with the anticipated arrival of a commuter
train - not the scheduled arrival, but the
arrival time according to data coming
from the train’s computer system.
By transforming data from a historical
record to a dynamic feed, it can be utilized
on an on-going basis to make programs
and decision-making more efficient.
And, the total volume of data served by
Socrata via API has grown more than
1,500 percent in the last two years. This
explosion in API usage across the open
data movement comes from innovative
citizen apps like WasMyCarTowed. This
app uses public data APIs to tell motorists,
who return to their parking spot but find
no car, if their car was towed and, if so,
where.
MACHINE-TO-MACHINE DATA
FROM ARCHIVE TO FUEL
Data is also increasingly accessed, not by
humans, but by systems and applications
via application programming interfaces
(API). They make data assets useful by
integrating them as an input for other
systems. For example, a new generation of
connected and sensor-enabled municipal
water systems provide data on water usage.
This information lets citizens understand
As the size of datasets grows, frequency
of dataset updates increases, and more
machine-to-machine interactions occur,
data’s usefulness is being tapped and
novel uses and reuses are coming to
light. Publishing data for the sake of
transparency is a noble, yet shortsighted,
goal. In the era of big, fast, useful data,
we are challenged to do more.
FASTER DATA
High-frequency data (data that’s updated
very often) has seen similar growth. In
the last two years, as data from sensor
networks and operational systems has
been published as open data, the number
of real-time datasets Socrata hosts has
grown more than 2,000 percent. (See
chart above.)
While static data is useful primarily for
archival and analytical purposes, realtime or streaming data can be leveraged
as an input into other systems. Snowplow
locations can be fed in real-time to
citizens and the media, bus locations and
traffic signal state information can be
Today, nearly every dataset is accessed
via API and many thousands of times
per day. Socrata is the largest provider of
government data APIs in the world, with
more than 77,000 datasets available
via API.
OPEN INNOVATION
•
WINTER 2014
7

5. OPEN DATA IN FOCUS
OPEN DATA IN FOCUS
The Importance
of Engaged Open
Data Advocates
By David Eaves
Open Innovation Expert
During my 2011 keynote at Open Government Data
camp I talked about how the open data movement was
at an inflection point. For years we have been on the
outside, yelling that open data matters. Now we are
being invited inside and we have a great responsibility
to be of service. Once you have world leaders talking
about things like a G8 Open Data Charter you are no
longer on the fringes - not even remotely.
R
ob Kitchin, Professor of
Geography at the National
University of Ireland at
Maynooth, recently wrote
a list of open data critiques for his blog
Programmable City. His post inspired
me to remind the open data community
– particularly the advocates - of our
responsibility to take part in the debates
around open data, right now. We need to
engage in the discussions on a number of
topics if we want open data to reach its full
potential for effecting positive change in
the world. Specifically, I will address two
critiques that Professor Kitchin raised:
using data to empower the less powerful
and how to improve utility and usability of
that data.
8
OPEN INNOVATION
•
WINTER 2014
WHY IMPROVING DATA LITERACY
IS IMPORTANT
We must address the reality that data, even
while open to all, can be used to by the
most powerful to gain more power. There
are definitely cases where data can serve to
further marginalize at-risk communities.
For example, we should never publish
publicly the locations of women’s shelters
or worse, the list of families taking refuge
in them.
There are two things that give me some
hope in this space. The first is that,
when it comes to open data, the axis of
competition among providers usually
centers on accessibility. For example,
the Socrata platform (a provider of open
data portals to government) invests heavily
in creating tools that make government
data accessible and usable to the broadest
possible audience. This is not a claim that
all communities are being engaged and
that a great deal more work cannot be
done, but there is a desire to show greater
use, which drives some data providers to
find ways to engage new communities.
The second is that if we want to create a
data literate society— and I think we do,
for reasons of good citizenship, social
justice, and economic competitiveness—
we need the data first for people to learn
and play with. One of the best ways to help
people become data literate is to give them
more interesting data to play with. We did
not build libraries after everyone knew
how to read, we built them beforehand
with the goal of having them as a place that
could facilitate learning and education.
There are also things that often depress
me. I struggle to think of technologies that
did not empower the empowered – at least
initially. From the cell phone to the car to
the printing press to open source software,
all these inventions have helped billions
of people, but they did not distribute
themselves evenly, especially at first.
So the question cannot be reduced to –
will open data empower the empowered,
but to what degree, and where, and with
whom? I’ve seen plenty of evidence where
data has enabled small groups of people to
protect their communities or make more
transparent the impact (or lack there of)
of a government regulation. Open data
expands the number of people who can
use government information for their
own ends – this, I believe is a good thing
– but that does not mean we shouldn’t be
constantly looking for ways to ensure that
it does not reinforce structural inequity.
PROMOTING THE IDEA OF DATA
AS A PLATFORM
Some of the issues around usability I’ve
addressed above in the accessibility piece
– for portals that genuinely want users,
the axis of evolution is pointed in the
right direction with governments and
companies like Socrata trying to embed
more tools on the website to make the data
more usable.
I also agree with a point by Professor
Kitchin that, rather than creating a
virtuous circle, poorly thought out and
launched open data portals will create
negative “doomloops” in which poor
quality data begets little interest which
begets less data. However, the problem is
bigger than that.
One of the main reasons I have been an
advocate of open data is a desire to help
citizens, nonprofits, and companies gain
access to information that could help
them with their missions. I also wanted
to help change the way governments deal
with their data, so that they can share it
internally more effectively. I often cite a
public servant I know who had a summer
intern spend three weeks surfing the
national statistical agency website to find
data they knew existed but could not find
because of terrible design and search. A
poor open data site is not just a sign that
the public can’t access or effectively use
government data; it usually suggests that
the government’s employees can’t access or
effectively use their own data. This is often
deeply frustrating to many public servants.
Thus, the most important outcome created
by the open data movement may be that
government organizations, save for those
in the intelligence community, realize that
they are not comfortable with using data
to drive decisions. Getting governments
to think about data as a platform (yes,
I’m a fan of government as a platform for
external use, but above all for internal use)
is, in my mind, one way we can enable
public servants to gain better access to
information. Adoption of this principle
will also, in many cases, obviate the need
for costly solutions from huge vendors
(like SAP and Oracle), whose $100 million
dollar implementations often silo off data,
rarely produce the results promised and
are so obnoxiously expensive it boggles
the mind.
The key to all this is that open data cannot
be something you slap on top of a big IT
stack. I try to explain this in my blog post
“It’s the Icing Not the Cake” about how
Washington, DC was able to effectively
launch an open data program so quickly
(which was, apparently, so effective at
bringing transparency to procurement
data the subsequent mayor rolled it back).
The point is that governments need to start
thinking in terms of platforms if – over
One of the main
reasons I have been
an advocate of open
data is a desire to help
citizens, nonprofits,
and companies gain
access to information
that could help them
with their missions.
the long term – open data is going to work.
And it needs to start thinking of itself as
the primary consumer of the data that is
being served on that platform.
My main point is this: let’s not play at the
edges and merely define this challenge
as one of usability. It is a much bigger
problem than that. If we get it wrong,
then the big government vendors and the
inertia of bureaucracy win. If we get it
right, we could potentially save taxpayers
millions—while enabling a more nimble,
effective, and responsive government.
I try hard to be critical advocate of open
data – one who engages the risks and
challenges posed by open data. I’m not
perfect and balancing these two goals –
advocacy and a critical view – is not easy,
but I hope it is how we in the open data
movement see our role.
OPEN INNOVATION
•
WINTER 2014
9

6. OPEN DATA IN FOCUS
OPEN DATA IN FOCUS
In fact, today, the United Kingdom
hosts one of the most mature open data
programs in the world. From this vantage
point, UK’s thought leaders are talking
more and more about the tremendous
economic and social potential of open
data, especially in machine-readable,
standardized formats.
In late October 2013, Open Data Institute
(ODI) founders Sir Tim Berners-Lee
and Sir Nigel Shadbolt announced the
creation of a global network of “nodes”
where programs aligned with ODI’s
principles of openness and economic
innovation would be established. As
part of the announcement, they were
quoted as saying, “The best way that open
becomes the new default is demand:
from businesses and organizations, both
public and private, from individuals and
corporations.”
By Erika Smith
Socrata VP of Worldwide Markets
10
OPEN INNOVATION
•
WINTER 2014
Socrata CEO Kevin Merritt presented
on a panel at the Open Data Summit
with the title, “Open for Business – The
Commercial Impact of Open Data.”
Merritt commented on the growing
interest in economic growth through
open data, saying, “In the UK, and all of
Europe, people are asking, ‘Open data, so
With the growing list of businesses relying on
government open data, the speed and frequency
with which governments deliver that data has
become more crucial.
Open Data
in Europe
Leaps Beyond
Transparency
The open data movement is thriving in many parts
of Europe, including well-established programs
in the Netherlands, the United Kingdom, and
Scandinavian countries like Sweden and Norway.
But, in response to recent questions like, “What
have we achieved and what can we do with all
of this data?” the conversation has turned from
open data for transparency’s sake to open data to
strengthen the economy.
want to collaborate with them to promote
the creation of new products and services
based on open data.”
UK AIMS FOR ECONOMIC
GROWTH
In the fall of 2010, when the UK launched
a searchable database of business plans
for government departments, Prime
Minister David Cameron was quoted as
saying, “We want to be the most open
and transparent government in the
world.” Since then, getting non-sensitive,
non-personal data online has become
something British citizens expect.
Berners-Lee and Shadbolt assume that
open data will be put to use supporting
new businesses. The ODI, a not-for-profit
organization that is just over one year old,
has so far helped set up more than a dozen
open data-based startup companies in
the UK.
Socrata became an official partner of
ODI in the fall of 2013. ODI lead Gavin
Starks says, “We love Socrata’s view of
open data as fuel for new businesses. We
what?’ And, the ODI and other incubators
of new businesses help demonstrate that
innovation, new jobs, and better services
are possible when non-private data is easy
to access.”
FASTER MOVEMENT OF DATA TO
SUPPORT INNOVATION
With the growing list of businesses relying
on government open data, the speed
and frequency with which governments
deliver that data has become more
crucial. For example, Spend Network is a
startup supported by ODI that “uses open
spending data to create new insights for
Government and its suppliers.” Spend
Network’s services are only as useful and
accurate as the data it gathers.
delivery within Europe, Socrata recently
partnered with Microsoft to make the
Socrata Open Data Portal available on the
Microsoft Azure platform. Azure has a
number of data centers on the continent.
Socrata Senior Site Reliability Engineer
Paul Paradise noted, “As a growing
company, we scale internationally much
better by partnering with an established,
trusted, international data-hosting
platform like Microsoft Azure.”
MORE BUSINESS IN THE FUTURE
While transparency is still a central
goal for the open data movement, even
traditionally transparency-focused
organizations, like the Open Government
Partnership (OGP), acknowledge the
economic power of open data. Before
its Open Government Partnership
Summit in London in October, the OGP
announced that 37 countries had signed
commitments to, among many efforts,
“radically open up government data to
boost entrepreneurship, growth and
accountability.”
“Open data is fuel for innovation. The
world will never be the same, now that
we can so easily take public data and use
it to work to make people’s lives better,”
says Merritt. “Businesses will inevitably
put the public data shared to work for the
economy. Socrata’s job is to make sure that
it is easily accessed and in useful formats.”
Anticipating the greater demands on
datacenters across Europe and, in an effort
to improve the speed of its existing data
OPEN INNOVATION
•
WINTER 2014
11

7. OPEN DATA IN FOCUS
OPEN DATA IN FOCUS
Making the TwoWay Street of Open
Data a Reality
By Chris Whong
Socrata Data Solutions Architect
O
pen data has by and
large been a one-way
conversation. Governments
produce public data and
make it freely available, while citizens,
journalists, researchers, and hackers
consume it in whatever ways suit them.
But, having more eyes on the data once it
is released may be able to provide value
back to the government, turning users
of the data into a source of new data and
quality control. This is the experiment in
two-way open data that New York City is
pioneering with OpenStreetMap.
OpenStreetMap (OSM) is the “Wikipedia
of Maps,” where anyone can contribute
changes. (Yes, if there’s a footpath or bike
trail near your house that doesn’t show up
on mainstream web maps, you can literally
“draw” it into OSM, name it, and connect
it to existing roads.) Like Wikipedia,
changes to the map are subject to quality
control by the rest of the community,
12
OPEN INNOVATION
•
WINTER 2014
and can be just as easily undone. Users
may choose to update the map for many
reasons, from just knowing more about
conditions on the ground than anyone
else, to improving the map for a specific
project such as an app.
What if a user needed some building
outlines that OSM didn’t have yet? That
user could manually trace over the satellite
imagery, pointing and clicking lots of
custom polygons into existence. But what
if they needed a whole town? What if
they needed New York City? They do, and
NYC has an open dataset for that. The
city’s detailed GIS database of building
outlines and point data is freely available
for download at data.cityofnewyork.us.
While it will still take human effort to
import and verify data for over a million
buildings, creating them manually would
be an unfathomable and time-consuming
process. Leaders in the mapping tech
community have partnered with NYC’s
Department of Information Technology
and Telecommunications (provider of
the footprints data) and are launching
a community project to systematically
import the city’s treasure trove of
building data into OSM. The project
was announced in late September.
LEGIONS OF GEOSPATIAL
ANALYSTS
So what’s in it for the city? The potential
for updates that only a system like OSM
can provide. If OSM users see something
wrong, they can fix it. Maybe a building
footprint is misaligned, or maybe the
building doesn’t exist anymore. The city
receives a daily update of changes to
the building data, and can review those
changes. If they are legitimate, DoITT
can use these changes to guide updates to
their own master database, making it more
accurate and up to date. It’s as if the city
has legions of geospatial analysts qualitychecking their data and sending updates!
Alex Barth, Data Lead at the web mapping
company MapBox and OSM Advocate, has
been a key organizer of the NYC – OSM
collaboration, and has been working on
the idea since early 2012. The data was
already publicly available back then, but
carried a license that was incompatible
with OSM. NYC’s Open Data Law, passed
in March 2012, cleared up the licensing
issue and provided the way forward.
To Barth, the project is not simply about
buildings, but is an experiment and
learning experience about the impact of
community-driven projects like OSM. “It’s
a data improvement effort that has positive
side effects and really lets us grow the
community.” The longer-term vision goes
beyond OSM or even geodata, and hopes
to redefine open data publishing: “This is
about an open data commons, a single space
in which government and citizens interact.”
The first gathering of volunteer mappers
to work through the monumental task of
importing the city’s data met in October.
Liz Barry, another leader in the NYC-OSM
collaboration, hosted the meeting at the
offices of the Public Lab in Brooklyn, and 22
community members showed up to help. The
data was broken down into election districts,
and the team set out validating footprints
against aerial imagery, checking geometries,
and correcting overlapping polygons.
Existing attribute data in OSM could also
be merged with better polygons from the
city data. Barry said the workflow is still
being vetted, and is not quite ready for fullscale deployment. The idea is that once the
workflow is perfected, updates won’t require a
physical meetup. Volunteer OSM users will be
able to import a chunk of the city’s building
footprints whenever and wherever they can.
The real fun will begin when large amounts
of the data have been successfully imported
and the city can report back about the volume
and utility of OSM-contributed changes. In
many cases, there may be more information
about a building in OSM than the city
maintains on its own, meaning the “twoway street” of open data may not flow evenly
in both directions. The OSM community
has found a partner in DoITT, and this
experiment will serve as an early model of
the power of citizens and activists to improve
government data.
OpenStreepMap allows users to view multiple regions,
from a birds-eye view of Europe to a zoomed in view of the
streets of Manhattan.
OPEN INNOVATION
•
WINTER 2014
13

8. OPEN DATA IN FOCUS
OPEN DATA IN FOCUS
Open Data
Had Better Be
Data-Driven
By Thomas Levine
“Dada” Artist
We’ve been opening government data for some
time now. Without realizing it, we’ve amassed
some rather rich data about how people publish
and consume open data. With this data about
the use of open data, it’s possible to use databacked benchmarks, projections, and decisions in
our open data strategies, and this can make our
approaches to open data more systematic, logical,
and obtainable.
Curious about what the data behind open data can
teach us? Read on.
DATA, DATA, EVERYWHERE
We have data about open data, but it
always starts out in formats that are not
convenient for this sort of study of open
data. So, we first need to turn this raw data
into a dataset. In my mind, a dataset is a
collection of things, with some consistent
properties describing each of the things.
(We often represent datasets as tables.)
We’re going to treat each dataset as a thing
inside our collection of many datasets.
14
OPEN INNOVATION
•
WINTER 2014
A visualization of the
datasets available on
data.cityofnewyork.us by
category, showing both
raw datasets, as well as
derivative charts, maps, and
filtered views.
Every dataset has basic, easy-to-extract
properties like the number of records
it contains, and the size it takes up on a
hard drive. We can come up with more
complicated information too, like the
number of missing values and the date of
the oldest record.
In addition, when a dataset is published
online, it creates metadata – about the
dataset and its use. Metadata provides
details like when it was first published,
who uploaded it, and how many times
people have downloaded it since then.
By collecting some of these simple
properties and metadata from each
dataset, it’s possible to create a dataset
about the publication and use of other
datasets – what I call a super-dataset.
DATASETS AS DATA POINTS
A super-dataset compiles information
about each dataset – when it was
published, what kind of information it
contains, etc. – into a single row, creating a
record (i.e. a data point) about that dataset.
With this setup, you can perform all
kinds of functions and analyses about the
publication and use of datasets: you can
look at how many datasets are on different
catalogs, how data is queried and reported,
what licenses datasets have, and how many
there are of a certain category. Here are
some examples of analyses I have done.
Number of Datasets: I compared the
number of datasets on some various
government data portals that run Socrata’s
open data platform software. I found that
New York City, Chicago, and the state of
Oregon offer the most datasets.
Licensing: I also used my super-dataset of
properties and metadata to look into this
question: What licenses do people apply
to their open data? I discovered that many
portals favor public domain or some form
of open license, but most list no license at all.
Groupings of Datasets: By looking
at similarities in the titles, schemas,
and other metadata of datasets,
I determined what sorts of data
government organizations were putting on
their open data portals and how different
datasets were related to each other.
data platform displays metadata about
each dataset, such as community rating,
number of visits, number of downloads, etc.
about how people publish and use open
data. It paints the bigger picture of what’s
going on, what’s working, and what isn’t.
When you use metadata to populate
records in a super-dataset, that metadata
becomes data that we can see, analyze, and
learn from.
Imagine what we can do with this databacked understanding of our open data!
We can find out what has been done before
and what has worked, allowing open data
publishers to plan their releases more
strategically. We can measure release
strategies against solid, quantitative
statistics to make sure that they are
helping us achieve our goals. We can
even use these findings to build products
that help people interact with open data.
These are just some of the possibilities that
emerge when we think of data catalogs as a
datasets of datasets.
WHY THIS MATTERS
Even though the word “metadata” contains
the word “data,” people don’t typically
think of metadata as something to analyze.
There are numerous case studies, by
organizations such as Code for America,
CKAN, Open Data Institute, about how
to open up government data, but these are
based strongly on personal experiences,
not precise, quantitative statistics. There
is comparatively little work that uses data
to produce guidelines or best practices for
the opening of data.
Metadata is often invisible. If we could
see it, it might just look like background
information about web pages and their
contents. For example, Socrata’s open
Open data experts often talk about making
use of open data and building products
from it, but a super-dataset accomplishes
something different. It reveals information
METADATA AS DATA
OPEN INNOVATION
•
WINTER 2014
15

9. Fixing the
Federal IT
Procurement
Process
By Kevin Merritt
Socrata Founder and CEO
16
OPEN INNOVATION
•
WINTER 2014
S
ometimes it takes a disaster
for people to recognize the
need for change. The botched
rollout of HealthCare.gov—
something I would actually describe as a
systemic catastrophe—provides a historic
opportunity to overhaul the federal
government technology procurement
system, which is so clearly broken.
DIAGNOSING THE PROBLEM
Government technology projects have long
been characterized by significant delays,
mammoth cost overruns, and software
products that routinely underperform.
But, before the HealthCare.gov fiasco,
relatively few people were aware of the
magnitude of the problems plaguing
the federal IT purchasing system. Now
that the story has made front-page news,
everyone wants to know what went wrong.
Just as any good software engineer looks
to uncover the root causes of bugs in a
computer program, it is important to
meticulously diagnose the underlying
causes that continue to produce IT
disasters at the federal level.
SELLING TO THE FEDERAL
GOVERNMENT IS A BYZANTINE
PROCESS
Federal IT procurement is overly complex
and opaque. The process is full of arcane
rules and oppressive costs that discourage
many technology providers from entering
the market. The Federal Information
Security Management Act (FISMA) is
a prime example. This legislation was
enacted at a time when applications were
almost always hosted on premise and the
“waterfall” methodology (not “agile”) was
the predominant approach to software
development. Times have changed, and we
have learned a lot since then.
Yet, because FISMA is still in place,
vendors looking to bid on federal
technology projects must comply
with outmoded information security
requirements, like using tape backups to
replicate files offsite. This is completely
incongruent with modern IT best
practices. More importantly, the odds of
data loss in a single environment that uses
tape backups is statistically much higher
than in a geo-redundant environment
where data is replicated across multiple
servers.
OPEN INNOVATION
•
WINTER 2014
17

10. We admire
startups because
their core
competency is
innovation. They
establish traction
and momentum
because they
invent novel
solutions to old
problems, develop
new technologies,
or come up with
creative new ways
of doing business.
18
OPEN INNOVATION
•
WINTER 2014
THE GSA JUGGERNAUT AND
OTHER BURDENS
To bid on federal government projects,
businesses must first be listed on the
General Services Administration (GSA)
schedule. Maintaining this listing is
practically a full-time job, requiring
continuous updates, renewals, and, of
course, piles of paperwork. To keep up,
many smaller businesses are compelled
to hire consultants just to clear all of the
administrative hurdles.
In a very real sense, GSA requirements
operate like an extra tax on businesses,
siphoning profits from sales to federal
government agencies. In addition to this
“tax,” GSA vendors must offer their lowest
commercial price to the government, even
if the market value for their products is
significantly higher. This squeezes profits
further, disproportionately affecting the
ability of smaller businesses to compete.
As if this were not burdensome enough,
any vendor that wants to sell to the federal
government must be able to decipher the
Federal Acquisition Regulation (FAR),
the principal set of rules that govern the
federal procurement process. The FAR
is every bit as complex and unwieldy as
the federal tax code, containing rules as
picayune as the width of the margins for
request for proposal (RFP) responses.
And these rules change constantly. Few
emerging technology firms have the
wherewithal or the patience to wade into
that kind of logistical labyrinth.
INNOVATORS NEED NOT APPLY
The net result is that many nimble,
innovative companies simply forego doing
business with the federal government,
concluding that it is just too costly. Over
the years, this has created an environment
that nurtures mediocrity. Today, there
are a whole class of enterprise companies
that specialize in navigating the bloated
and broken procurement system. Most
of these businesses make their money by
selling consulting hours, hiring dozens
of subcontractors to do the work, and
overseeing the development of expensive,
customized technology solutions that
often do not perform as advertised.
We admire startups because their core
competency is innovation. They establish
traction and momentum because they
invent novel solutions to old problems,
develop new technologies, or come up with
creative new ways of doing business. The
irony in the federal IT space, however, is
that these behemoth systems integrators
(SIs) are winning business not because
they have innovated. Rather, it is just the
opposite. Their sole core competency
has nothing to do with technology or
innovation at all; their only real skill is
that they have perfected the art of selling
to the government!
First and foremost, there needs to be full
transparency into federal contracts. All of
the details for each contract—including
the amounts paid and the names of the
people who authorized each payment—
need to be online in machine readable,
fully sortable, and searchable formats.
CARVING A PATH FORWARD
The federal government also needs to
make it simple for the public to follow
the flow of money and influence between
federal government agencies requesting
vendor support and the contractors
bidding on projects. As a country, we
have made great strides in recent years
in increasing visibility into campaign
contributions. To weed out corruption
and improve efficiency, senior government
leaders need to bring that same level of
effort and scrutiny to the procurement
process.
Government leaders need to introduce
a serious-minded reform agenda if any
real progress is to be made in revamping
the federal IT procurement process. This
means directly applying the open data
ideals that the Obama administration has
generally espoused to the specifics of the
IT purchasing process.
Government
leaders need
to introduce a
serious-minded
reform agenda if
any real progress
is to be made
in revamping
the federal IT
procurement
process.
Further, policy makers should engage
providers of cloud and SaaS-based
technology solutions to discuss ways
to modernize the federal government’s
information security requirements. In
the wake of the failure of HealthCare.gov,
there is an exciting opportunity to educate
the government officials who certify these
security mandates about modern software
best practices.
WHO IS READY TO TAKE UP THE
MANTLE OF PROCUREMENT
REFORM?
The once-stodgy subject of federal IT
procurement has gone viral. In turn,
this awareness has produced broad
consensus around the need for meaningful
change—a precious commodity that
should not be wasted. Right here, right
now, I am putting forth a clarion call to
civic leaders on both sides of the aisle:
the time is ripe to take up the mantle of
real procurement reform. By unleashing
the ingenuity of software companies
that have previously been pushed to the
sidelines, you can help launch a new era of
technology innovation that improves the
lives of Americans for generations
to come.
OPEN INNOVATION
•
WINTER 2014
19

11. SOCRATA
2013 HIGHLIGHTS
Awards
Bloomberg Philanthropies
Mayors Challenge (Chicago)
2013 was a big year for the open data movement. It was filled with
hackathons, datapaloozas, and hack nights. Leaders and innovators in the
movement won awards and recognition. We here at Socrata were thrilled
to be a part of it all. Here is a look back at some of the top U.S. events
we attended, awards our customers won, and favorite things we heard
along the way.
“I was very impressed to see the level of
civic innovation and ownership from the
Code Michigan hackathon participants.
Many coders I spoke to had never been to
a hackathon before and were very proud
to share how their products were going to
improve their city and many others.”
Ian Kalin
Director of Open Data for Socrata
at Code Michigan
“For those of us who
work in government,
this feels like our own
national holiday.”
“Socrata on site is really
beneficial because having
somebody like Clint (a Socrata
engineer) is going to bring a
technical element to it to help
promote all the different tools
that are on the portal, available
through Socrata.”
Shannon Spanhake
City of San Francisco
Deputy Innovation Officer
on the National Day of
Civic Hacking
Tim Dupuis
Interim Director of the IT
Department and the Registrar
of Voters in Alameda County
at Alameda County Hackathon
Events
ATX Civic Hackathon III
(Austin, TX)
Data Liberation Award
(NYC Dept of Health)
Code Michigan
2013 Digital Counties Survey
Award (Montgomery County,
Snohomish County, King County)
State Program Innovation Award
(Oregon)
2013 Web 2.0 Award (Raleigh, NC)
2013 Best of the Web Winner
(Alameda County)
2013 Digital Government
Achievement Award Winner
(Alameda County)
20
OPEN INNOVATION
•
WINTER 2014
Vegas Hack
San Francisco Housing Data Jam
Open Science (Mountain View)
EcoHackSF (San Francisco)
Urban Data Challenge
Hackathon (San Francisco)
Data P2PU Course Sprint
(Mountain View)
Code for Seattle
2013 Achievement of Excellence
in Procurement Award
(Alameda County)
Hack 4 Change (Seattle)
2013 Digital Cities Winners
(Boston, Seattle, Austin, Chicago,
Baltimore, Raleigh)
NC Data Jam (Raleigh)
Alameda County Apps Challenge
NYC Data Week
SpoCode (Spokane)
Apps4Halifax
API Craft Conference (Detroit)
API Strategies and Practices
Conference (San Francisco)
“It is so amazing to see in a
city that has such a diverse
set of problems that we can
unite to create a diverse set
of solutions.”
“This is an amazing display of
the power of civic hacking and
what we can all do together.”
AWARDS
EVENTS
Code Across America (San Diego)
2013 CSAC Merit Award
(Alameda County)
Beth Blauer
Director of GovStat for Socrata
at Hack for Change Baltimore
Hannah Young
Program Coordinator of the National
CfA Brigade on the National Day of
Civic Hacking
Code Across America
(Philadelphia)
Code for America Summit
(San Francisco)
Code for L.A.
NC Datapaolooza (Raleigh)
Hawaii Digital Government
Summit
SXSW Eco Hackathon 2013
(Austin)
Atlanta Govathon
Code for Oakland
International Open Data Day
(Washington, D.C., New York City,
Seattle, Atlanta)
“A ‘datapalooza’ connects experts,
innovators, and entrepreneurs to
relevant data drawn from every level
of government.”
Jason Hare
Open Data Program Manager
at NC Datapalooza
OPEN INNOVATION
•
WINTER 2014
21

12. DESIGNING
THE FUTURE:
CODE FOR
AMERICA
MAKES AN
IMPACT
By Alida Moore
Socrata Content Strategist
Before 2011, parents in Boston, MA struggled to figure out
which schools their children were eligible to attend. The
confusing process, involving a 28-page manual and a lottery
system, was a source of contention and, at times, violent
conflict among parents across the city. A turning point came
when Code for America (CfA), a non-profit organization
dedicated to using data and technology to improve lives,
turned their team on the problem.
22
OPEN INNOVATION
•
WINTER 2014
OPEN INNOVATION
•
WINTER 2014
23

13. Ben Berkowitz, CEO and
Co-Founder of SeeClickFix,
presents at the Code for
America Summit 2011.
Top: Andrew McLaughlin,
self-proclaimed “nerd,”
Sunlight Foundation board
member, and CEO of Digg
and Instapaper, presents
at the Code for America
Summit 2011. Bottom: Laura
Meixell, 2013 Code for
America Fellow in
San Francisco.
L
auren Reid, Senior Public
Affairs Manager at CfA,
explains how they built a
solution. “The CfA fellows in
Boston developed an app where parents
could enter a few simple data points, such
as address and age of child, and find out
quickly which schools were available as
options for their child,” Reid says. The app
was hugely successful. “The City of Boston
told us that this app, developed in just
three short months, would have taken the
city more than two years and two million
dollars to create -- had it gone through the
standard procurement process. Together
with the City, we’re resolving a decadesold problem using modern technology and
open data, and changing the conversation
between parents and the schools system
from one of contention to one that is
positive and productive.”
24
OPEN INNOVATION
•
WINTER 2014
Code for America was founded in 2009
by Jennifer Pahlka, the current Deputy
Chief Technology Officer for the United
States. The organization seeks to find
out what can happen when smart, savvy
researchers, developers, and designers
are deployed to cities across the nation to
help local governments unlock their data
to create solutions and deliver services to
their citizens.
THE FORMULA
The Code for America formula is simple.
Developers and designers commit a year of
their careers to helping a city government
in need of problem solving. In exchange
for a modest stipend, these participants,
called fellows, live in their assigned
cities, and use their skills to help move
government forward to meet the needs
of 21st century citizens.
CFA PROGRAMS
FELLOWS IN ACTION
Code for America has developed four
programs to help further their mission.
Matthew Hampel, 2012 CfA Fellow
(Detroit), spoke to Socrata about his
experience as a fellow. One of his projects
for the city of Detroit was to create a
web and mobile app that would update
commuters about bus schedules. Hampel
told us about the app, called TextMyBus.
The Fellowship: Code for America’s
flagship program, in which developers
and designers are matched with local
governments to transform data into usable
forms for public improvement.
The Brigade: Civic-minded volunteers
come together to form brigades, bringing
grassroots efforts to data use and
transparency.
The Accelerator: Provides financial and
logistical support to civic tech startups,
from a $25K grant to mentorship and
networking opportunities.
The Peer Network: A learning network for
government innovators who want to work
with other local governments to harness
the power of open data in their cities.
“Originally, bus data was tracked
manually through an ancient interface
and it wasn’t available to public,” Hampel
explained. “So the city provided us with
access to their servers and we exposed the
data. With that, we built a text messaging
app that helps people figure out when their
buses are arriving.” TextMyBus has proven
quite popular. As of December 2013, the
app has served over 1.1 million users.
CfA fellows have contributed 62 apps
across America so far, from helping
citizens navigate the public school system
to receiving text alerts when services, like
food stamps, are about to expire.
We are on
the cutting
edge of Gov
2.0 and civic
hacking and
geeking.
OPEN INNOVATION
•
WINTER 2014
25

14. Foley agrees. “Your civic technical
ecosystem should overlap with
entrepreneurship. As part of that
ecosystem, our brigade influences the civic
IT department by promoting open data.
We are on the cutting edge of Gov 2.0 and
civic hacking and geeking,” Foley says.
You can expect to see more brigades pop
up in 2014, allowing for crossover and
BRIGADES IN ACTION
Some of Code for America’s most
passionate, committed volunteers live in
Raleigh, NC. Jason Hibbets and Chad
Foley are two of the Raleigh brigade’s four
co-captains. Each man volunteers hours of
time each month, outside of their day jobs,
to the brigade.
Top: Volunteers gather at
a brigade meeting in San
Francisco. Bottom left: Former
City of Seattle Chief Technology
Officer Bill Schrier presents at
the Code for America Summit
2011. Bottom Right: CfA will send
more fellow to more cities than
ever before in 2014.
26
OPEN INNOVATION
•
WINTER 2014
Data gives you
the power to
make informed
decisions,
and informed
decisions are
better decisions.
partnership. As Foley explains, “Your city
boundaries shouldn’t limit innovation.”
If, for example, you have an app that
catalogues the greenways in your city,
it should continue beyond city limits,
says Foley.
Code for America
by Numbers
2011
19
fellows deployed to Boston, Philadelphia, and Seattle
2012
26
fellows sent to eight cities: Austin, Chicago, Detroit, Honolulu,
Macon, New Orleans, Philadelphia, and Santa Cruz
2013
27
fellows embedded within nine communities: Kansas City, Kan. +
Kansas City, Mo.; Las Vegas; Louisville, Ken.; New York; Oakland,
Calif.; San Francisco; San Mateo County, Calif.; South Bend, Ind.;
Summit County, Ohio
2014
“Being part of the Raleigh brigade allows
us to have an impact on local government
and our community,” says Hibbets. He
believes that government transparency
in data is paramount to innovation.
“Open data is the foundation of civic
entrepreneurship,” Hibbets explains.
“Open data belongs to the people. Put
data in the right hands and apps that can
improve the daily life of citizens can
be created.”
A LOOK FORWARD
In 2014, Code for America plans to add 31
new fellows to the fellowship program.
For people like Matt Hampel, who is
passionate about harnessing open data for
civic improvement, Code for America has
been a vehicle for positive change. “Data
helps you design the future you want to
see,” Hampel explains. “Data gives you the
power to make informed decisions, and
informed decisions are better decisions.”
31
fellows will join the Code for America program. Cities to be
announced.
62
apps created by CfA fellows over the years
OPEN INNOVATION
•
WINTER 2014
27

15. Beth Blauer is a
leading expert
in implementing
“stat” programs in
government and is
Director of GovStat
at Socrata.
DATA-DRIVEN
GOVERNMENT
IN ACTION
By Beth Blauer
In the previous issue of Open Innovation, I
talked about the importance of fact-based
decision-making and how gut decisions
can be expensive and dangerous. For
governments to play a leading role in the
data revolution, performance measurement
and successful delivery are mandatory. This
truth is something I am passionate about.
My passion resonates from my experience
managing StateStat and the Delivery Unit in
Maryland and is affirmed in conversations I
have had with government leaders since.
F
ortunately, the open data
movement has resulted in
tools for data-driven decisionmaking, performance, and
delivery. Socrata’s product, GovStat,
helps local governments become more
transparent, engage citizens, and measure
progress against initiatives and goals. It
also allows stakeholders to collaborate
throughout the entire process on one
common platform.
In Beyond Transparency: Open Data and
the Future of Civic Innovation1, I stressed
the idea that it’s not a matter of if datadriven government can create the best
solutions to society’s problems; it’s a matter
of how soon governments will embrace the
idea and reap the benefits. Using a tool like
GovStat enables governments to collect
and update data across departments,
build beautiful data visualizations, and
create both internal and citizen-facing
dashboards to track progress.
I would like to highlight three government
organizations that have chosen to use
GovStat as their primary performance
measurement solution. I will share their
specific challenges in moving toward data
1
28
OPEN INNOVATION
•
WINTER 2014
Code for America Press (2013)
transparency, why they chose GovStat as
their solution, and how each organization
plans to use GovStat to increase efficiency
and accountability to their constituents.
TELLING THE STORIES
THAT MATTER:
COOK COUNTY, IL
In 2011, Cook County began a program
to create a strategic plan around the use
of open data for decision-making and
communications. Andrew Schwarm, Chief
Performance Officer of Cook County, was
the project head tasked with finding a
performance measurement tool.
“The county was publishing our quarterly
reports in a PDF on our website, which
was far from best practice,” Schwarm says.
Using a tool like
GovStat enables
governments
to collect and
update data across
departments, build
beautiful data
visualizations, and
create both internal
and citizen-facing
dashboards to track
progress.
OPEN INNOVATION
•
WINTER 2014
29

16. DYNAMIC REPORTING
AND COLLABORATION:
KANSAS CITY, MO
Kansas City, MO has been a leader in
government transparency and datadriven government. Since taking office,
Mayor James has convened regular
KCStat meetings with his senior team and
holds them accountable to the goals and
strategies that are most important for the
citizens of Kansas City. The technology
behind this program began as their
homebuilt performance measurement
platform, KCStat. A year later, the City
added Socrata’s open data portal.
An early adopter of the GovStat platform,
Cook County decided on a two-phase
implementation process. First, the County
replicated the PDF data and created
reports for each department. “GovStat
allowed us to take the data already
gathered and put it on a more flexible,
user-friendly, open, and transparent
platform,” Schwarm says.
Once the data was updated and made
available to the entire organization,
Cook County entered phase two of
implementation: using data to drive
decisions. “[Data-driven decision-making]
is now part of our culture and the way we
do business,” explains Schwarm. Further,
30
OPEN INNOVATION
•
WINTER 2014
accurate data allows Cook County to tell
accurate stories. One goal the county
tracks is lowering sick leave for county
staff; Cook County hopes to reduce
employee sick leave to 5.2 hours per month
before December 2013. On October 25,
2013, Cook County published a GovStat
report on the progress toward this goal,
declaring it “on track.”
Schwarm is excited to track other county
initiatives, including public safety and
healthcare goals. “We plan to roll out one
goal per month and continue to report on
its progress,” Schwarm says. He continues,
“The use of good, timely, accurate data,
especially for a government facing tough
fiscal situations, allows us to make
decisions and prioritize at a high level.”
Of all GovStat’s features, the KCStat team
is particularly excited about the tool’s
drag-and-drop reporting capabilities.
As the program grew, the team found they
were limited in their reporting abilities. So
when Senior Performance Analysts Kate
Bender and Julie Steenson began looking
for a performance management solution to
Citizens of Cook County can view progress on goals, like reducing
the amount of sick leave taken by government employees, using
GovStat’s citizen dashboard.
“We wanted to transform our external
performance reporting from a static
PDF to a dynamic open data web portal.
GovStat jumped out as the product that
made the most sense for us.”
enhance KCStat, they knew GovStat would
meet those needs.
The use of good,
timely, accurate
data, especially
for a government
facing tough fiscal
situations, allows us
to make decisions
and prioritize at a
high level.
“We already were on board with the
concept of telling our story through data
and improving government performance,”
Bender says. “The only thing missing was
the ability to create dynamic reports and
data visualizations,” she continues.
Kansas City hopes to improve how they
communicate their results to the larger
community. The Kansas City council has
always been dedicated to civic engagement.
Each monthly council meeting is filmed,
televised, and shared online, which
attracts serious stakeholders. Still, the
city wanted a way to communicate with
every citizen, quickly and easily. They
found the GovStat public-facing, or
“citizen,” dashboard most useful. “The
GovStat dashboard is a way to engage
every stakeholder,” says Steenson. “It’s a
storytelling device that makes the data
more accessible.”
Kansas City launched its citizen dashboard
in early October and looks forward to
using the tool to make progress toward
the city’s initiatives. “One big step forward
was having the city council adopt a
set of strategic priorities, which form
the backbone of our dashboard,” says
Steenson. “The next step was assigning
measurements to those priorities,” she
continues. “The council made a public
statement about their priorities and
then adopted specific measures to track
This chart, updated daily, shows citizens in Kansas City the percentage
of customers who have been satisfied with the city’s response to water
main break service requests.
OPEN INNOVATION
•
WINTER 2014
31

17. Executives in the County Manager’s Office
viewed the approval of the tax – known
locally as Measure A – as an opportunity
to take the county’s transparent, datadriven approach to budgeting to the
next level. “The Measure A sales tax will
generate about $65 million in revenue
each year over the next 10 years,” says
Reyna Farrales, Deputy County Manager
for San Mateo County. “We have a duty
to show how the services funded by the
tax contribute to specific, measurable
goals and how those results ultimately fit
together with the priorities in our Shared
Vision 2025.”
GovStat’s mapping
capabilities help
establish patterns.
The map above shows
all water leak, water
meter, and hydrant
repair requests open in
Kansas City.
progress. These metrics for tracking
progress will be the advantage of the
GovStat dashboard as we build it out.”
OPEN INNOVATION
The San Mateo County government
is known for its deep commitment to
transparency. The County’s Shared
Vision 2025, a comprehensive community
planning process designed to get direct
input from citizens, is a sterling example of
this open, collaborative governing style. So
when county residents approved a halfcent sales tax increase in 2012, the Board
of Supervisors promised that residents
would be able to see how their tax dollars
were being spent.
After implementing GovStat, the County
Manager’s Office used the system to help
departments define goals and metrics
for their respective Measure A funding
proposals. In September 2013, San
Mateo County became the first county
government in the U.S. to deploy a publicfacing GovStat site. The launch of SMC
Performance coincided with the Board
of Supervisors’ approval of 22 projects
totaling more than $50 million.
Kansas City plans to roll out a new goal
every month over the next six months,
into 2014. Residents can follow each goal’s
progress on the citizen dashboard. In the
meantime, Bender is excited to see how the
tool helps improve efficiency. “Dynamic
reporting saves so much time,” she says.
“It’s great to work in a system designed
around government use.”
In September 2013,
San Mateo County
became the first
county government
in the U.S. to deploy
a public-facing
GovStat site.
32
TAKING PERFORMANCE
MANAGEMENT TO THE NEXT
LEVEL: SAN MATEO COUNTY, CA
When voters approved the sales tax
increase, county leaders began their
search for a technology solution to keep
the community informed about progress
in the coming years. “We were aware of
the success of the StateStat program in
Maryland and were really impressed with
the performance dashboard they were
using,” says Farrales. “We discovered that
Maryland, along with a number of other
cities and states, were all using the Socrata
platform. Our newly appointed Chief
Innovation Officer (CIO) Jon Walton came
from San Francisco and had experience
with Socrata, so he and his staff were able
to help us ramp up our Socrata-powered
open data portal and move into GovStat
right away.”
•
WINTER 2014
SMC Performance features a dashboard
with a series of interactive tiles, each tile
acting as a hub for the goals associated
with a specific Measure A initiative. By
clicking on the tile, users can explore
performance measures, review raw data,
read the actual proposal and, in some
cases, look at charts, graphs, and maps
created with the data. “One of my favorite
things about GovStat is how easy it is to
create maps and other visualizations of
the data we collect,” says Shanna Collins,
a Budget Analyst in San Mateo County’s
Office of Budget and Performance. “These
visualizations help us identify where the
greatest needs are throughout the county,
so we can make budget choices that are
based on objective data.”
In the coming months, San Mateo County
will use GovStat to track the performance
of the original programs funded by
Measure A to monitor progress in key
areas. In addition, the County is planning
to roll out a new dashboard to follow the
nine community impact goals that make
up its Shared Vision 2025 in early 2014.
“GovStat has been at the heart of our move
toward true data-driven management,”
says Farrales. “And it gives us a platform
for involving the community and our
employees in decision-making, which is
central to our mission.”
GOALS IN ACTION
Each of these organizations offers a
compelling story of what is possible
by putting the principles of datadriven decision-making, performance
measurement, and delivery into action.
Greater adoption of these practices is not
a trend; it’s a fundamental shift in the
way governments around the world are
embracing their mission.
This pie chart shows the number of emergency response vehicles by
category for San Mateo.
OPEN INNOVATION
•
WINTER 2014
33

18. CODELESCENCE:
ENGINEERING
COMES OF AGE
By Patrick Hasseries
Staff Writer
Socrata is maturing, and it is most evident
through the evolution of its engineering—past,
present, and future.
“What’s really exciting about Socrata right now
is that we’re transitioning from a startup – in
the true sense of the word – to a company,”
says Steve Greenberg, one of Socrata’s newest
Engineering Program Managers.
W
hen Socrata was
originally founded,
it began with a small
team of scrappy
software engineers. Team structure
was simple, and early team members
quickly developed an office culture that
paralleled the products they were building:
open, engaging, and collaborative. They
encouraged free discussion, shared a love
for challenges, and held events – planned
or spontaneous – that built camaraderie.
Since the earliest days of the company,
dozens of engineers have joined
Socrata, from part-time high school
interns to veteran programmers with
decades of education and experience.
The engineering team has grown into
an engineering department, and the
original infrastructure must evolve to
accommodate more members and
bigger projects.
DIVIDING TEAMS WITHOUT
CAUSING DIVISION
When companies expand, they must be
careful not to let growth impede progress
or dull internal culture. So how will
Socrata’s engineering team restructure
itself without sacrificing its liveliness?
“We’re emulating Spotify’s ‘Tribes, Guilds,
Squads’ model,” says Jerome Gagner,
Socrata Director of Engineering. “Through
it we’ll keep the culture that we have as
well as develop engineers’ careers and
individual disciplines.”
Based on Spotify’s model for structuring
internal teams, Socrata’s engineers are
reorganizing into three types of groups.
All engineers dedicated to a specific
service or product form large teams
called “Tribes.” All engineers who share a
34
OPEN INNOVATION
•
WINTER 2013
Challenging
Queries
Data is Socrata’s
business, and it handles
datasets of all sizes—
from those with only
a dozen records to
others with records
ranging in the millions.
A search request on
a small dataset takes
only a few seconds, but
advanced searches on
large datasets naturally
take much more time
to process. Socrata’s
engineers have recently
enjoyed the challenge
of developing methods
to make even advanced
searches in large
datasets complete in less
than four seconds.
Socrata’s engineering team
members have attended
dozens of hackathons
around the world, serving
as instructors, presenters,
and judges.
discipline (front-end development, backend development, user interface design,
etc.) collaborate through inter-tribal
groups called “Guilds.” Within each tribe
are also “Squads,” small teams dedicated
to specific disciplines within the tribe.
Squads act like miniature startups in their
own right, maintaining their own cultures
and core values.
A CULTURE OF CODE
AND CHARACTER
Anyone who has visited Socrata can attest
to the dedication and personality that
each employee brings to the company,
particularly the engineers.
Anyone who has visited Socrata can attest to the
dedication and personality that each employee
brings to the company, particularly the engineers.
“One thing I frequently talk about with
friends outside of work is that Socrata’s
engineering team is filled with a bunch
of unique, funny people. We have a lot
of characters here, but at the end of the
day, everyone is deeply committed to our
customers and to the company’s mission,”
says Greenberg.
Socrata’s engineers are inventive and
rarely bored. When they aren’t building or
running code, they run marathons, make
hot sauce, craft micro-brews, and use 3D
printers to produce models of Iron Man’s
helmet. In between tasks, they also pursue
friendly feuds, from office Nerf-gun battles
to internet pranks.
Tradition is another important part of
Socrata’s culture, with the engineers being
among its most avid followers. Since the
company’s founding, engineers have eaten
lunch every Friday at the same teriyaki
place in Seattle’s Pioneer Square. They
enjoy an office happy hour on Friday
afternoons, bringing in beer or whiskey
OPEN INNOVATION
•
WINTER 2014
35

19. to share. On their anniversary of employment, team
members also share one pound of chocolate for every
year they’ve worked at Socrata.
While these perks and traditions are part of what
makes Socrata great, ask the engineers what they
love most about their job and they will likely reply
“learning, coding, and problem-solving.”
“We face a lot of hard engineering challenges, and
those challenges are very exciting to work on,” says
Anthony Nowell, a Socrata Software Developer. “I
have no doubt that I’m surrounded by ridiculously
bright people who can overcome those challenges, and
I’m really encouraged knowing that we’re helping each
other to grow technically.”
EXPONENTIAL GROWTH
Pioneering tech companies like Google, Spotify, and
Facebook have demonstrated that well-defined culture
and open organizational models are valuable growth
tools. They help attract new team members, develop
employee competency, and foster loyalty, which
in turn inspires employees to produce innovative
projects that put the company on the leading edge
of entrepreneurship.
Socrata adds members to its team on a near-weekly
basis, many of them engineers. Part of this growth
success comes from the attractiveness of Socrata’s
culture, which the company has nurtured since its
founding. Moving forward, Socrata will continue
to emphasize organizational models that maintain
that culture.
“It’s amazing seeing Socrata grow as big as it has,” says
Chris Metcalf, Director of Developer Platform and sixyear Socrata employee. “I don’t have kids but, to me,
this is like seeing my child go off to college.”
Meet
Socrata’s
Engineers
When we say “geek out,” we mean
it. Socrata’s engineering team hosts
a bright, quirky cast of characters
who are known for much more than
creating great software.
CHRIS “CHARMS” ARMSTRONG
JEROME GAGNER
LILIA GUTNIK
Achievements unlocked: Helped
develop a number of startups
Achievements unlocked: Taught
himself to read and speak Arabic;
deployed numerous large-scale
applications in languages such as Java,
PHP, .Net, Ruby, etc.
Achievements unlocked: Talked about
her prank adventures on The Moth;
developed GovStat from the ground up
Geeks out on: Baking, beer, current
events, dev-ops, live music, tabletop
games, travel
Geeks out on: Data visualization,
Keanu Reeves, stand-up comedy, tennis
Most likely to: Become a pastry chef
Geeks out on: Camping, flight
simulators, SCALA, server-side and
front-end technologies
Most likely to: Sleep with a Keanu
Reeves body pillow
GIACOMO FERRARI
STEVE GREENBERG
KARIN HELLMAN
Achievements unlocked: 3D printed
Iron Man’s helmet and arc reactor; cocreated Socrata’s charting library
Achievements unlocked: Recently had
his first child, helped design Microsoft’s
error reporting system
Achievements unlocked: “I’ve made
it through a year in the U.S. without
getting fat.”
Geeks out on: 3D printing, electronics,
web development
Geeks out on: Cycling, family,
remodeling his house
Geeks out on: Climbing, colors,
painting, Photoshop
Most likely to: Invent a working flux
capacitor, blow himself up in a lab
36
OPEN INNOVATION
•
WINTER 2014
OPEN INNOVATION
•
WINTER 2014
37

20. Meet Socrata’s Engineers
JOHN KEW
AYN LESLIE-COOK
ANTHONY NOWELL
PAUL PARADISE
JEFF SCHERPELZ
Achievements unlocked: Worked on
a number of evolutionary computation
projects; developed code used on
millions of servers around the world
Achievements unlocked: “My kids
don’t seem like they’ll turn out to be
criminals.”
Achievements unlocked: Recently had
his first child, built a small application to
assist his mother-in-law with her daycare
business
Achievements unlocked: Helped
shape Socrata from the very beginning
Achievements unlocked: Led Socrata
through several versions of its front
end (including conversion from FLEX to
Javascript); designed and built his own
house
Geeks out on: Family, illustrating
microorganisms, microbiology
Geeks out on: Family, statistical
modelling, swimming, theatre
Geeks out on: Computer hardware,
computer networking, triathlon
Geeks out on: Coding, family, web
frameworks
Geeks out on: Cars, food, knitting
JASON KROLL
CHRIS METCALF
BRIAN OLDFIELD
DAN RATHBONE
CLINT TSENG
Achievements unlocked: Built a
system to identify the best type of
education for over 15,000 careers and
find schools offering such education
Achievements unlocked: Helped
Socrata grow from a startup into a
company
Achievements unlocked: Accepted to
grad school
Achievements unlocked: Travelled
and worked all over the world
Geeks out on: Bitcoin mining, website
design
Geeks out on: Database technologies,
hiking, running, triathlon
Achievements unlocked: Taught
math and English in India; worked on
a project deployed to 80+ countries
and the international space station
Geeks out on: Computer science,
economics, music, running, statistics,
tennis
38
OPEN INNOVATION
•
WINTER 2014
Geeks out on: APIs, developer tools,
home improvement, micro-brewing,
photography
Most likely to: Wander off to explore
and then be found living in a hut in the
middle of nowhere
Geeks out on: Design, guitar, music,
movies, sailing
Most likely to: Freak that this was
written in Comic Sans
OPEN INNOVATION
•
WINTER 2014
39

21. info@socrata.com
(206) 340-8008
83 S. King St., Ste. 107
Seattle, WA 98104
www.socrata.com
twitter.com/socrata
facebook.com/socrata
Subscribe to future issues of Open Innovation by going to www.socrata.com/magazine