1. CKAN Tutorial
@ odw2013
Presenter: 李承錱 Cheng-Jen Lee (Sol)
Email: cjlee AT iis.sinica.edu.tw
This work is licensed under a
Creative Commons Attribution-ShareAlike 3.0 Taiwan License.

2. 2013/11/9
Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Installation
● Harvesters
● Issues

3. 2013/11/9
Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Installation
● Harvesters
● Issues

4. 2013/11/9
About CKAN
● The Comprehensive Knowledge Archive
Network
● A powerful data management system
● Publishing
● Sharing
● Finding
● Using Data

5. 2013/11/9
About CKAN
61 instances

6. 2013/11/9
CKAN 2

7. 2013/11/9
Feature Tour

8. 2013/11/9
Demo Site
demo.ckan.org

9. 2013/11/9
Feature Tour (1)
Publish & Find Datasets
Add Dataset Basic
Information

10. 2013/11/9
Feature Tour (1)
Publish & Find Datasets
Add Data
Under the Dataset

11. 2013/11/9
Feature Tour (1)
Publish & Find Datasets
Add Metadata
About the Dataset

12. 2013/11/9
Feature Tour (1)
Publish & Find Datasets
Filter By
Keywords

13. 2013/11/9
Feature Tour (1)
Publish & Find Datasets
Filter By
Geographical Features

14. 2013/11/9
Feature Tour (2)
Store & Manage Data
Data Explorer:
recline_preview (csv, xls)
json_preview
pdf_preview
ckanext-spatial

15. 2013/11/9
Feature Tour (2)
Store & Manage Data
Graphing data

16. 2013/11/9
Feature Tour (3)
Engage with Users & Others
Share

17. 2013/11/9
Feature Tour (3)
Engage with Users & Others
Organization

18. 2013/11/9
Feature Tour (3)
Engage with Users & Others
Manage Users of
an Organization

19. 2013/11/9
Feature Tour (3)
Engage with Users & Others
Manage Role of Members
Admin: edit datasets & members
Editor: edit datasets
Viewer: view (private) datasets
Note: Public datasets are visible to
everyone

20. 2013/11/9
Feature Tour (3)
Engage with Users & Others
Harvest and Federation

21. 2013/11/9
Feature Tour (3)
Engage with Users & Others
History

22. 2013/11/9
Feature Tour (4)
Customise & Extend
● RESTful JSON APIs
– The Action API
– The DataStore API
– The FileStore API...
● Extensions (over 60)
– ckanext-harvest
– ckanext-spatial
Open source is good!
● Themable
● Integrates with other CMS (ex. Drupal)

23. CKAN and 5 ★ Open Data
2013/11/9

24. CKAN and 5 ★ Open Data
● ★ Make your stuff available on the Web (whatever
format) under an open license
2013/11/9

25. CKAN and 5 ★ Open Data
● ★★ Make it available as structured data (e.g.,
Excel instead of image scan of a table)
★★★ Use non-proprietary formats (e.g., CSV
instead of Excel)
– Accept any data format
– Beautiful data demonstration
2013/11/9

26. CKAN and 5 ★ Open Data
● ★★★★ Use URIs to denote things, so that people
can point at your stuff
– Permanent link for each dataset
– Get Dataset URI through API
● ★★★★★ Link your data to other data to provide
context
– Linked data and RDF for metadata
2013/11/9

27. CKAN and 5 ★ Open Data
● RDF for metadata
– DCAT and Dublin Core
– curl -L -H "Accept:application/rdf+xml"
http://thedatahub.org/dataset/gold-prices
2013/11/9

28. 2013/11/9
Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Installation
● Harvesters
● Issues

29. 2013/11/9
United Kingdom
DATA.GOV.UK

30. 2013/11/9
United States
DATA.GOV

31. 2013/11/9
Brazil
DADOS.GOV.BR

32. 2013/11/9
European Union
PUBLICDATA.EU

33. 2013/11/9
Geospatial Data Explorer:
Lat/Long field

34. 2013/11/9
Geospatial Data Explorer:
GeoJSON

35. 2013/11/9
Geospatial Data Explorer:
WMS

36. 2013/11/9
Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Installation
● Harvesters
● Issues

37. 2013/11/9
System Architecture

38. What You Should Know
● Python, Pylons
● ckan plugins toolkit
● SQLAlchemy, SQL
● HTML, JavaScript
● Babel
● Web Server (Nginx, uWSGI...)
2013/11/9

39. 2013/11/9
Install from Source
● Virtual environment
● Checkout the source (via GIT)
– https://github.com/okfn/ckan
● Create a CKAN config file
● Setup Jetty & Solr
● Initialize Database (user, db)
● Link to who.ini
● Create a Sysadmin User
● Deployment (nginx + uWSGI)
● Install other extensions...

40. 2013/11/9
Installation Notes
● https://ckan-docs-tw.readthedocs.org/

41. 2013/11/9
Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Installation
● Harvesters
● Issues

42. 2013/11/9
Harvesters
● ckanext-harvest
– Remote harvesting extension
– https://github.com/okfn/ckanext-harvest
● Source Type
– CKAN (built-in)
– CSW
– WAF
– Custom (csv/xls/website… etc)

43. 2013/11/9
Harvested from TGOS
CSW service

44. 2013/11/9
Harvesters
http://Mydomain.com/harvest

45. 2013/11/9
Harvesters
Add a new harvest source

46. 2013/11/9
Harvesters
Create a harvest job

47. 2013/11/9
Harvesters
Overview of harvested datasets

48. 2013/11/9
Harvesters
Background Process
● Manually
– (pyenv) $ paster --plugin=ckanext-harvest
harvester gather_consumer -c
/etc/ckan/default/production.ini
– (pyenv) $ paster --plugin=ckanext-harvest
harvester fetch_consumer -c
/etc/ckan/default/production.ini
– (pyenv) $ paster --plugin=ckanext-harvest
harvester run -c
/etc/ckan/default/production.ini

49. 2013/11/9
Harvesters
Background Process
● Automatically
– Supervisor (for gather & fetch consumer)
– Cron (for run)

50. 2013/11/9
Harvesters
Custom harvester
● Implement the harvester interface
to perform harvesting operations
● Three stages
● gather: get the identification
● fetch: fetch the contents
● import: create ckan package(dataset)

51. 2013/11/9
Harvesters
The harvesting interface
from base import HarvesterBase
class SRDAHarvester(HarvesterBase):
def _set_config(self,config_str):
def info(self):
...
def gather_stage(self, harvest_job):
def fetch_stage(self, harvest_object):
def import_stage(self, harvest_object):
See the extension site for details
An example (SRDA): http://goo.gl/ZMnND7

52. 2013/11/9
Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Installation
● Harvesters
● Issues

53. 2013/11/9
Issues
● CJK Support
– CJK Search
– Some broken translations
– Frequent updates
● Extensions compatibilities
● Tons of tweaks needed
● Performance Issue
● Complicated Architecture

54. ● Official Documents:
– http://docs.ckan.org/en/latest/
● Installation Notes (in Chinese):
– https://ckan-docs-tw.readthedocs.org/
● CKAN Development Discussions:
– http://lists.okfn.org/mailman/listinfo/ckan-dev
● CKAN Taiwan Interest Group:
– https://groups.google.com/forum/#!forum/ckan-taiwan-interest-group
2013/11/9
Resources

55. Thanks for your attention!
2013/11/9
Any Q?
Email: u10313335 AT citi.sinica.edu.tw
http://about.me/sollee