1. CKAN 2 IntroductionPresenter: Cheng-Jen Lee (Sol)Email: cjlee AT iis.sinica.edu.twThis work is licensed under aCreative Commons Attribution-ShareAlike 3.0 Taiwan License.

2. Agenda About CKAN Feature Tour Publish & Find Datasets Store & Manage Data Engage with Users & Others Customise & Extend CKAN and 5 Open Data Showcase Issues Extended Topic 1: Installation Extended Topic 2: Harvesters2014/5/22 2 3. Agenda About CKAN Feature Tour Publish & Find Datasets Store & Manage Data Engage with Users & Others Customise & Extend CKAN and 5 Open Data Showcase Issues Extended Topic 1: Installation Extended Topic 2: Harvesters2014/5/22 3 4. About CKAN The Comprehensive Knowledge ArchiveNetwork A powerful data management system Publishing Sharing Finding Using Data2014/5/22 4 5. About CKAN83 instancesaround the worldin May 20142014/5/22 5 6. CKAN 22014/5/22 6 7. Feature Tour2014/5/22 7 8. Demo Sitedemo.ckan.org2014/5/22 8 9. Feature Tour (1)Publish & Find DatasetsAdd Dataset BasicInformation2014/5/22 9 10. Feature Tour (1)Publish & Find DatasetsAdd DataUnder the Dataset2014/5/22 10 11. Feature Tour (1)Publish & Find DatasetsAdd MetadataAbout the Dataset2014/5/22 11 12. Feature Tour (1)Publish & Find DatasetsFilter ByKeywords2014/5/22 12 13. Feature Tour (1)Publish & Find DatasetsFilter ByGeographical Features2014/5/22 13 14. Feature Tour (2)Store & Manage DataData Explorer:recline_preview (csv, xls)json_previewpdf_previewckanext-spatial2014/5/22 14 15. Feature Tour (2)Store & Manage DataGraphing data2014/5/22 15 16. Feature Tour (3)Engage with Users & OthersShare2014/5/22 16 17. Feature Tour (3)Engage with Users & OthersOrganization2014/5/22 17 18. Feature Tour (3)Engage with Users & OthersManage Users ofan Organization2014/5/22 18 19. Feature Tour (3)Engage with Users & OthersManage Role of MembersAdmin: edit datasets & membersEditor: edit datasetsViewer: view (private) datasetsNote: Public datasets are visible toeveryone2014/5/22 19 20. Feature Tour (3)Engage with Users & OthersHarvest and Federation2014/5/22 20 21. Feature Tour (3)Engage with Users & OthersHistory2014/5/22 21 22. Feature Tour (4)Customise & Extend RESTful JSON APIs The Action API The DataStore API The FileStore API... Extensions (over 60) ckanext-harvest ckanext-spatialOpen source is good! Themable Integrates with other CMS (ex. Drupal)2014/5/22 22 23. CKAN and 5 Open Data2014/5/22 23 24. CKAN and 5 Open Data Make your stuff available on the Web (whateverformat) under an open license2014/5/22 24 25. CKAN and 5 Open Data Make it available as structured data (e.g.,Excel instead of image scan of a table) Use non-proprietary formats (e.g., CSVinstead of Excel) Accept any data format Beautiful data demonstration Datastore : Indexing for structure data2014/5/22 25 26. CKAN and 5 Open Data Use URIs to denote things, so that peoplecan point at your stuff Permanent link for each dataset Get Dataset URI through API Link your data to other data to providecontext Linked data and RDF for metadata2014/5/22 26 27. CKAN and 5 Open Data RDF for metadata DCAT and Dublin Core curl -L -H "Accept:application/rdf+xml"http://thedatahub.org/dataset/gold-prices2014/5/22 27 28. Agenda About CKAN Feature Tour Publish & Find Datasets Store & Manage Data Engage with Users & Others Customise & Extend CKAN and 5 Open Data Showcase Issues Extended Topic 1: Installation Extended Topic 2: Harvesters2014/5/22 28 29. United KingdomDATA.GOV.UK2014/5/22 29 30. United StatesDATA.GOV2014/5/22 30 31. BrazilDADOS.GOV.BR2014/5/22 31 32. European UnionPUBLICDATA.EU2014/5/22 32 33. Geospatial Data Explorer:Lat/Long field2014/5/22 33 34. Geospatial Data Explorer:GeoJSON2014/5/22 34 35. Geospatial Data Explorer:WMS2014/5/22 35 36. Agenda About CKAN Feature Tour Publish & Find Datasets Store & Manage Data Engage with Users & Others Customise & Extend CKAN and 5 Open Data Showcase Issues Extended Topic 1: Installation Extended Topic 2: Harvesters2014/5/22 36 37. Issues CJK Support CJK Search Some broken translations Frequent updates Extensions compatibilities Tons of tweaks needed Performance Issue Complicated Architecture2014/5/22 37 38. System Architecture2014/5/22 38 39. Issues (Cont'd) What You Should Know Python & Pylons ckan plugins toolkit SQLAlchemy & SQL HTML, JavaScript Babel Web Server (Nginx, uWSGI...)2014/5/22 39 40. Resources Official Documents: http://docs.ckan.org/en/latest/ Installation Notes (in Chinese): https://ckan-docs-tw.readthedocs.org/ CKAN Development Discussions: http://lists.okfn.org/mailman/listinfo/ckan-dev CKAN Taiwan Interest Group: https://groups.google.com/forum/#!forum/ckan-taiwan-interest-group2014/5/22 40 41. Thanks for your attention!Any Q?Email: u10313335 AT citi.sinica.edu.twhttp://about.me/sollee2014/5/22 41 42. Agenda About CKAN Feature Tour Publish & Find Datasets Store & Manage Data Engage with Users & Others Customise & Extend CKAN and 5 Open Data Showcase Issues Extended Topic 1: Installation Extended Topic 2: Harvesters2014/5/22 42 43. Install from Source Virtual environment Checkout the source (via GIT) https://github.com/okfn/ckan Create a CKAN config file Setup Jetty & Solr Initialize Database (user, db) Link to who.ini Create a Sysadmin User Deployment (nginx + uWSGI) Install other extensions...2014/5/22 43 44. Installation Notes https://ckan-docs-tw.readthedocs.org/2014/5/22 44 45. Agenda About CKAN Feature Tour Publish & Find Datasets Store & Manage Data Engage with Users & Others Customise & Extend CKAN and 5 Open Data Showcase Issues Extended Topic 1: Installation Extended Topic 2: Harvesters2014/5/22 45 46. Harvesters ckanext-harvest Remote harvesting extension https://github.com/okfn/ckanext-harvest Source Type CKAN (built-in) CSW WAF Custom (csv/xls/website etc)2014/5/22 46 47. Harvested from TGOSCSW service2014/5/22 47 48. Harvestershttp://Mydomain.com/harvest2014/5/22 48 49. HarvestersAdd a new harvest source2014/5/22 49 50. HarvestersCreate a harvest job2014/5/22 50 51. HarvestersOverview of harvested datasets2014/5/22 51 52. HarvestersBackground Process Manually (pyenv) $ paster --plugin=ckanext-harvestharvester gather_consumer -c/etc/ckan/default/production.ini (pyenv) $ paster --plugin=ckanext-harvestharvester fetch_consumer -c/etc/ckan/default/production.ini (pyenv) $ paster --plugin=ckanext-harvestharvester run -c/etc/ckan/default/production.ini2014/5/22 52 53. HarvestersBackground Process Automatically Supervisor (for gather & fetch consumer) Cron (for run)2014/5/22 53 54. HarvestersCustom harvester Implement the harvester interfaceto perform harvesting operations Three stages gather: get the identification fetch: fetch the contents import: create ckan package(dataset)2014/5/22 54 55. HarvestersThe harvesting interfacefrom base import HarvesterBaseclass SRDAHarvester(HarvesterBase):def _set_config(self,config_str):def info(self):...def gather_stage(self, harvest_job):def fetch_stage(self, harvest_object):def import_stage(self, harvest_object):See the extension site for detailsAn example (SRDA): http://goo.gl/ZMnND72014/5/22 55