Given at MongoDC on June 27, 2011.

1. Open Government
Data & MongoDB
Luigi Montanez
luigi@sunlightfoundation.com

3. Question? @LuigiMontanez

4. Open Data + Open Source
= Open Government
Question? @LuigiMontanez

5. MongoDB enables
open data
Question? @LuigiMontanez

6. Opening Up Data
✴ Gather data from disparate sources
✴ Data dumps (SQL, Fixed-width columns)
✴ Web scraping
✴ Text/PDF parsing
✴ Serving RESTful JSON APIs
Question? @LuigiMontanez

7. JSON
✴ Tree structure, not tabular
✴ Still relational
✴ JSON for data, XML for documents
✴ Closely resembles native data structures
✴ No manual parsing needed
Question? @LuigiMontanez

8. Three Projects
✴ Poligraft
✴ Real Time Congress API
✴ Open State Project
Question? @LuigiMontanez

9. Three Projects
✴ Poligraft
✴ Real Time Congress API
✴ Open State Project
Question? @LuigiMontanez

10. App design
drives
schema design
Question? @LuigiMontanez

15. {
"title": "President Obama's climate 'Plan B' in hot water -
Darren Samuelsohn - POLITICO.com"
}
Text

17. {
"title": "President Obama's climate 'Plan B' in hot water -
Darren Samuelsohn - POLITICO.com",
"slug": "EOsc",
"source_url": "http://www.politico.com/news/stories/
0810/40534.html",
"content": ".................",
}
Text

20. {
"title": "President Obama's climate 'Plan B' in hot water -
Darren Samuelsohn - POLITICO.com",
"slug": "EOsc",
"source_url": "http://www.politico.com/news/stories/
0810/40534.html",
"content": ".................",
"entities": [...] Text
}

21. {
"title": "President Obama's climate 'Plan B' in hot water -
Darren Samuelsohn - POLITICO.com",
"slug": "EOsc",
"source_url": "http://www.politico.com/news/stories/
0810/40534.html",
"content": ".................",
"entities": [
{
Text
"name": "Barack Obama",
"type": "politician",
},
...
]
}

23. {
"title": "President Obama's climate 'Plan B' in hot water -
Darren Samuelsohn - POLITICO.com",
"slug": "EOsc",
"source_url": "http://www.politico.com/news/stories/
0810/40534.html",
"content": ".................",
"entities": [
{
Text
"name": "Barack Obama",
"type": "politician",
"breakdown": {"indiv": "33", "pac": "67"}
"top_industries": ["Lawyers/Lobbyists","Finance/Insurance/
Real Estate","Misc. Business"]
},
...
]
}

25. Natural Schemas
Question? @LuigiMontanez

26. Three Projects
✴ Poligraft
✴ Real Time Congress API
✴ Open State Project
Question? @LuigiMontanez

27. Real-Time Congress API
Credit: vgm8383 on Flickr

28. Android App: “Congress”

29. Politiwidgets

30. Requirements
✴ Aggregate lots of data
Biographical, Bills, Votes, Earmarks,
Video Clips, Floor Updates, Legislative
Documents, Committee Schedules,
Contributions, Interest Group Ratings
✴ Lightweight responses
Question? @LuigiMontanez

31. {legislator: {
in_office: true,
title: "Rep",
nickname: "",
district: "9",
bioguide_id: "L000551",
govtrack_id: "400237",
phone: "202-225-2661",
website: "http://lee.house.gov/index.html",
twitter_id: "",
last_name: "Lee",
name_suffix: "",
last_updated: "2010/04/13 00:00:14 +0000",
party: "D",
chamber: "house",
state: "CA",
youtube_url: "http://www.youtube.com/RepLee",
first_name: "Barbara",
gender: "F",
congress_office: "2444 Rayburn House Office Building",
earmarks: {
average_number: 20,
total_amount: 10000000,
average_amount: 22994535,
total_number: 28,
last_updated: "2010-03-18",
fiscal_year: 2010,
}
...
}

32. // limit selection to a subset of fields
db.people.find( { 'first_name' : 'john' },
{ 'last_name' : 1,
'address' : 1 } );
// use dot-notation to dig into an object
db.people.find( { 'state': 'CA' },
{ 'address.zip_code': 1 } );

33. ?sections=last_name,first_name,state,earmarks
{legislator: {
last_name: "Lee",
first_name: "Barbara",
state: "CA",
earmarks: {
average_number: 20,
total_amount: 10000000,
average_amount: 22994535,
total_number: 28,
last_updated: "2010-03-18",
fiscal_year: 2010,
}
}

34. ?sections=last_name,first_name,state,earmarks.total_amount,earmarks.total_number
{legislator: {
last_name: "Lee",
first_name: "Barbara",
state: "CA",
earmarks: {
total_amount: 10000000,
total_number: 28
}
}

35. Partial responses
make payloads
smaller
Question? @LuigiMontanez

36. Three Projects
✴ Poligraft
✴ Real Time Congress API
✴ Open State Project
Question? @LuigiMontanez

38. 50 States =
50 Formats
Question? @LuigiMontanez

39. Schemalessness
allows for granular
control
Question? @LuigiMontanez

40. Custom Fields
✴ Traditional RDBMS
✴ Update the schema for new fields, run a
migration, feel icky
✴ Create a custom_fields table
✴ MongoDB
✴ Just store it
Question? @LuigiMontanez

41. Speaking JSON
natively
Question? @LuigiMontanez

42. Python
Source Scraped JSON PostgreSQL
Transform

43. Source Scraped JSON MongoDB

44. Three Projects
✴ Poligraft
✴ Real Time Congress API
✴ Open State Project
Question? @LuigiMontanez

45. Developer Happiness

46. Thanks!
sunlightlabs.com
@LuigiMontanez
Question? @LuigiMontanez