1. Publishing Data to the Socrata
Open Data Platform
Jeff Johnson
Manager, SFGIS Program and Open Data Services
City and County of San Francisco
9 April 2013

2. SFGIS and Open Data
 Jeff Johnson
 Manager, SFGIS Program and Open Data Services, Department of
Technology, City and County of San Francisco
 With the City since 1993
 Intern at Public Works for GIS sub-committee of EISPC
 Worked on the initial Basemap project
 With DT since 2002
 Using FME regularly since 2010

3. SFGIS serves as central resource
for spatial data for the City
 SFGIS collects spatial data from:
• Internal city departments (DPW, et al.)
• State agencies (e.g., Hazard Zones)
• Federal agencies (e.g., Census 2010)
• Commercial sources (Imagery, Demographic and Business Data)
 SFGIS distributes spatial data to departments and public via:
• Database access to SFGIS System of Record
• Open Data: DataSF.org  Data.SFGov.Org (Socrata)
• ArcGIS Online for Organizations (sfgov.maps.arcgis.com)
 SFGIS has expertise in Extract, Transform and Load (ETL) workflow:
• FME Workbench
• FME Server

4. History of Data for the Public
 Shapefiles available for download in 2002
through SFGIS ftp site
 DataSF launched in 2009 – internally hosted site
 Catalog of data
 Pointers to internal ftp site with zipped shapefiles
 Data.sfgov.org launched in February 2012
 Still pointers to ftp site, but …
 Socrata Open Data Platform

5. What is Socrata?
 Seattle-base cloud software company
 Open Data hosted sites
 Focused on access to government data
 Many cities now using to offer datasets to public
 Chicago, New York, Seattle, San Francisco
 Tools for querying, visualizing and embedding
hosted datasets
 API for developers – SODA 2.0 API

6. Open Data in San Francisco
 Open Data policy adopted in 2010
 Open Data legislation in the works
 Departments required/encouraged to post non-
confidential datasets
 How to make it easy for departments to do so?
 Who in departments will do this?
 What is most common component of City data?

7. Wouldn’t it be great if …
 Yes, you can use the SODA API to append,
replace, update and insert datasets on Socrata
programmatically – python, java, javascript,
ruby, et al. -- but …
 How about a writer in FME that writes directly to
your open data site on Socrata? No programming
required?

8. Solution – FME Socrata Writer
 How to make it easy?
 Use FME to easily create ETL procedures that
connect to your data sources and then write to
Socrata in one action, then automate
 Who will use?
 GIS analysts and power users already are familiar
with geospatial concepts and manage departmental
data
 What is most common component?
 Location

9. Demonstration
 Service Requests from 311 Call Service Center
 Potholes, Graffiti, Trees ... Need fixing

10. Workspace

11. Dataset on Data.Sfgov.org

12. Visualization on Data.Sfgov.org

13. Embedded in Webpage

14. Thank You!
 Questions?
 For more information:
 Jeff.Johnson@sfgov.org
 City and County of San Francisco
 DataSF: http://data.sfgov.org
 SFGIS: http://www.sfgov.org/gis
 SODA developers: http://dev.socrata.com/