Things you didn't know you can use in your Salesforce
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
1. NYC DataWeb
A platform for Integrating Public Data into NYC.gov
Joel Natividad
Click here for narrated version TCG
Thursday, June 9, 2011
SemTech 2011
2. About Me
• TCG Software
• Software Services arm of “The Chatterjee Group”
• Several Portfolio companies in Lifesciences, Telecom,
Aviation, Energy, Real Estate, & Info Technology
• Headquartered in NYC
• Delivery Centers in Bangalore, Kolkata & Mumbai
• Look after Knowledge Engineering Practice of TCG
6. Main Goals
• stimulate development of apps
that improve access to info
and govt transparency,
and;
• encourage innovation & the
creation of new IP with
commercial potential
10. CROWDSOURCING
• Wisdom of the Crowd
• Self-selecting, motivated developers
• Bang for the Buck
• Ignites Entrepreneurship
11. CROWDSOURCING
• Challenge:
Improve Recommendation Algorithm
by 10%
• Dataset:
STATISTICS
• 100 million ratings (training set) • just 6 days into contest,
• Half a million Users Cinematch bested by 1%
• 18 thousand movies • 20,000 Teams, 150 countries
• Entrants:
• Prize: • Bell Labs
One million US Dollars
• Opera Solutions
• Well-renowned universities
12. CROWDSOURCING
• Challenge:
Improve Recommendation Algorithm
by 10%
• Dataset:
STATISTICS
• 100 million ratings (training set) • just 6 days into contest,
• Half a million Users Cinematch bested by 1%
• 18 thousand movies • 20,000 Teams, 150 countries
• Entrants:
• Prize: • Bell Labs
One million US Dollars
• Opera Solutions
• Well-renowned universities
21. • First Federal CIO - Vivek Kundra
• Open Government Initiative
• Recovery.gov
• Data.gov
• USAspending.gov
• IT Dashboard
• Performance.gov
• Fedspace
• Citizen Services Dashboard
22. • First Federal CIO - Vivek Kundra
• Open Government Initiative
• Recovery.gov
• Data.gov
• USAspending.gov
• IT Dashboard
• Performance.gov
• Fedspace
• Citizen Services Dashboard
23. • First Federal CIO - Vivek Kundra
• Open Government Initiative
• Recovery.gov
• Data.gov
• USAspending.gov
• IT Dashboard
• Performance.gov
• Fedspace
• Citizen Services Dashboard
24. • First Federal CIO - Vivek Kundra
• Open Government Initiative
• Recovery.gov
• Data.gov
• USAspending.gov
• IT Dashboard
• Performance.gov
• Fedspace
• Citizen Services Dashboard
25. • First Federal CIO - Vivek Kundra
• Open Government Initiative
• Recovery.gov
}
• Data.gov Li fe
S u pp o r
t
• USAspending.gov
• IT Dashboard
• Performance.gov
• Fedspace
• Citizen Services Dashboard
26. • First Federal CIO - Vivek Kundra
• Open Government Initiative
•
sh ed
Recovery.gov
}
e t• sla o u Li fe
t S pp
B u dg i lli on
Data.gov
• m
ort
$ 34 o n USAspending.gov
fr o m •m i l l i
$8
IT Dashboard
• Performance.gov
• Fedspace
• Citizen Services Dashboard
52. and
Principles b io ni
ch
• Cost Effective (NOT $500 million dollars)
• Easy to Use (Developers/Publishers/Citizens)
• based on Open Standards
• Low Adoption Curve
• Help Accelerate Open Data Innovation
• Useable Data Now!
54. Useable Data Now
• “Beautiful” Website
• Useable by Developers/Publishers/Citizens
• based on Open Standards
• Low Adoption Curve
• Help Accelerate Open Data Innovation
• Useable Data Now!
55. What
NYCBigApps
Developers
were
Doing
Download &
Decipher
ETL Text
Processes
Siloed Data
• Spend inordinate amount of time interpreting data
• Massaged Data was then staged locally
• Developers kept reinventing the wheel
• Limited Data mashups
• Applications disconnected from NYCDatamine
46
57. How it Started
• Oct 12, 2010 - NYCBigApps 2.0 announced
• Nov 9, 2010 - NYCBigApps 2.0 kickoff meeting
• late Nov 2010 - spoke with Revelytix/Spry about
collaborating
• early Dec 2010 - started work on NYCDataWeb
• Jan 26, 2011 ~4:30p - submitted entry
60. What
We
Did
Domain
Ontology
Query &
Results
Cache Optimizer
Definitions
Re-Writer Planner
Siloed Data
Indexes Rules
Re-Writer Optimizer Mapping
Ontology
Indexes Planner Rules
Metadata
Ontology
51
61. “Beautiful” Website
Three dashboards were built
• NYC Agile Analytics (Spry)
• NYCreation (SMW+)
- visualized SPARQL query results
• NYCmantics (SMW+)
- NYC datamine explorer
95. Phase 2
Aug 2011 (Powered by NYCDataWeb)
• Hide Complexity • Open-source
(Simplicity = Adoption) collaboration with
vendors & other
• Incorporate the whole institutions
NYC datamine
• Incorporate the best of
• Make it easier for Socrata and data.gov
Publishers
• Improved Visualizations
• Make it easier for
Developers
• Make it easier for Citizens
96. Phase 2
Aug 2011 (Powered by NYCDataWeb)
• Hide Complexity • Open-source
(Simplicity = Adoption) collaboration with
vendors & other
• Incorporate the whole institutions
NYC datamine
• Incorporate the best of
• Make it easier for Socrata and data.gov
Publishers
• Improved Visualizations
• Make it easier for
Developers • Position NYCDataWeb as
the accelerated data
• Make it easier for Citizens mashup platform
97. Phase 3
Nov 2011 (NYCBigApps 2011)
• DataWeb Deployment Framework SMW bundle
• More Data Sources (Federator - Spinner)
• Linked Open Data
• Make it easier STILL for Publishers, Developers
and Citizens
• Enable Widespread adoption of NYCDataWeb
(NYCDataWeb bootcamp)
98. The
Broader
Vision
Domain
Ontology
Query &
Results
RDF
Ontology
NYC
Information
Web
Partners
RDF RDF
RDF
RDF RDF
Web
Pages
Other
Agency
Data
Sensorss Triplestores 85
99. Phase 4
Post NYC BigApps 2011
• Multiple solutions powered by NYCDataWeb
• <Your city/community/company here> DataWeb
• Help foster a viable ecosystem of Linked Data
• ... keep standing on the shoulders of giants
106. We need your help & feedback
A Platform for Integrating Public Data into NYC.gov
Find out more at
http://knoodl.com/ui/groups/NYC_Homepage
108. CREDITS
• Lego Faceparty picture by RichardAM (http://www.richard-am.net/)
• Lego Inauguration Pictures from various Flickr Users (sluggobear, Atwater, Dan
Hontz)
• Lego Luke looses his Hand by Flickr user wwwayazdotcom
• Tim Berners-Lee highlight from TED (http://www.ted.com/talks/
tim_berners_lee_on_the_next_web.html)
• Hans Rosling highlight from TED (http://www.ted.com/talks/
hans_rosling_shows_the_best_stats_you_ve_ever_seen.html)
• FlowerPowerpont2.pptx provided by Anna Rosling Rönnlund of gapminder
• “Star Wars Gangsta Rap” highlight, SizzlechestXXX
(http://www.youtube.com/watch?v=Ij4w7ChpuaM)
• Various screenshots provided by Revelytix, Spry Inc. and TCG Software
Services