ckan tutorial odw2013 131109

Download Ckan tutorial odw2013 131109

If you can't read please download the document

Upload: chengjen-lee

Post on 18-May-2015

751 views

Category:

Technology


6 download

TRANSCRIPT

  • 1. CKAN Tutorial@ odw2013Presenter: Cheng-Jen Lee (Sol)Email: cjlee AT iis.sinica.edu.twThis work is licensed under aCreative Commons Attribution-ShareAlike 3.0 Taiwan License.

2. 2013/11/9Agenda About CKAN Feature Tour Publish & Find Datasets Store & Manage Data Engage with Users & Others Customise & Extend CKAN and 5 Open Data Showcase Installation Harvesters Issues 3. 2013/11/9Agenda About CKAN Feature Tour Publish & Find Datasets Store & Manage Data Engage with Users & Others Customise & Extend CKAN and 5 Open Data Showcase Installation Harvesters Issues 4. 2013/11/9About CKAN The Comprehensive Knowledge ArchiveNetwork A powerful data management system Publishing Sharing Finding Using Data 5. 2013/11/9About CKAN61 instances 6. 2013/11/9CKAN 2 7. 2013/11/9Feature Tour 8. 2013/11/9Demo Sitedemo.ckan.org 9. 2013/11/9Feature Tour (1)Publish & Find DatasetsAdd Dataset BasicInformation 10. 2013/11/9Feature Tour (1)Publish & Find DatasetsAdd DataUnder the Dataset 11. 2013/11/9Feature Tour (1)Publish & Find DatasetsAdd MetadataAbout the Dataset 12. 2013/11/9Feature Tour (1)Publish & Find DatasetsFilter ByKeywords 13. 2013/11/9Feature Tour (1)Publish & Find DatasetsFilter ByGeographical Features 14. 2013/11/9Feature Tour (2)Store & Manage DataData Explorer:recline_preview (csv, xls)json_previewpdf_previewckanext-spatial 15. 2013/11/9Feature Tour (2)Store & Manage DataGraphing data 16. 2013/11/9Feature Tour (3)Engage with Users & OthersShare 17. 2013/11/9Feature Tour (3)Engage with Users & OthersOrganization 18. 2013/11/9Feature Tour (3)Engage with Users & OthersManage Users ofan Organization 19. 2013/11/9Feature Tour (3)Engage with Users & OthersManage Role of MembersAdmin: edit datasets & membersEditor: edit datasetsViewer: view (private) datasetsNote: Public datasets are visible toeveryone 20. 2013/11/9Feature Tour (3)Engage with Users & OthersHarvest and Federation 21. 2013/11/9Feature Tour (3)Engage with Users & OthersHistory 22. 2013/11/9Feature Tour (4)Customise & Extend RESTful JSON APIs The Action API The DataStore API The FileStore API... Extensions (over 60) ckanext-harvest ckanext-spatialOpen source is good! Themable Integrates with other CMS (ex. Drupal) 23. CKAN and 5 Open Data2013/11/9 24. CKAN and 5 Open Data Make your stuff available on the Web (whateverformat) under an open license2013/11/9 25. CKAN and 5 Open Data Make it available as structured data (e.g.,Excel instead of image scan of a table) Use non-proprietary formats (e.g., CSVinstead of Excel) Accept any data format Beautiful data demonstration2013/11/9 26. CKAN and 5 Open Data Use URIs to denote things, so that peoplecan point at your stuff Permanent link for each dataset Get Dataset URI through API Link your data to other data to providecontext Linked data and RDF for metadata2013/11/9 27. CKAN and 5 Open Data RDF for metadata DCAT and Dublin Core curl -L -H "Accept:application/rdf+xml"http://thedatahub.org/dataset/gold-prices2013/11/9 28. 2013/11/9Agenda About CKAN Feature Tour Publish & Find Datasets Store & Manage Data Engage with Users & Others Customise & Extend CKAN and 5 Open Data Showcase Installation Harvesters Issues 29. 2013/11/9United KingdomDATA.GOV.UK 30. 2013/11/9United StatesDATA.GOV 31. 2013/11/9BrazilDADOS.GOV.BR 32. 2013/11/9European UnionPUBLICDATA.EU 33. 2013/11/9Geospatial Data Explorer:Lat/Long field 34. 2013/11/9Geospatial Data Explorer:GeoJSON 35. 2013/11/9Geospatial Data Explorer:WMS 36. 2013/11/9Agenda About CKAN Feature Tour Publish & Find Datasets Store & Manage Data Engage with Users & Others Customise & Extend CKAN and 5 Open Data Showcase Installation Harvesters Issues 37. 2013/11/9System Architecture 38. What You Should Know Python, Pylons ckan plugins toolkit SQLAlchemy, SQL HTML, JavaScript Babel Web Server (Nginx, uWSGI...)2013/11/9 39. 2013/11/9Install from Source Virtual environment Checkout the source (via GIT) https://github.com/okfn/ckan Create a CKAN config file Setup Jetty & Solr Initialize Database (user, db) Link to who.ini Create a Sysadmin User Deployment (nginx + uWSGI) Install other extensions... 40. 2013/11/9Installation Notes https://ckan-docs-tw.readthedocs.org/ 41. 2013/11/9Agenda About CKAN Feature Tour Publish & Find Datasets Store & Manage Data Engage with Users & Others Customise & Extend CKAN and 5 Open Data Showcase Installation Harvesters Issues 42. 2013/11/9Harvesters ckanext-harvest Remote harvesting extension https://github.com/okfn/ckanext-harvest Source Type CKAN (built-in) CSW WAF Custom (csv/xls/website etc) 43. 2013/11/9Harvested from TGOSCSW service 44. 2013/11/9Harvestershttp://Mydomain.com/harvest 45. 2013/11/9HarvestersAdd a new harvest source 46. 2013/11/9HarvestersCreate a harvest job 47. 2013/11/9HarvestersOverview of harvested datasets 48. 2013/11/9HarvestersBackground Process Manually (pyenv) $ paster --plugin=ckanext-harvestharvester gather_consumer -c/etc/ckan/default/production.ini (pyenv) $ paster --plugin=ckanext-harvestharvester fetch_consumer -c/etc/ckan/default/production.ini (pyenv) $ paster --plugin=ckanext-harvestharvester run -c/etc/ckan/default/production.ini 49. 2013/11/9HarvestersBackground Process Automatically Supervisor (for gather & fetch consumer) Cron (for run) 50. 2013/11/9HarvestersCustom harvester Implement the harvester interfaceto perform harvesting operations Three stages gather: get the identification fetch: fetch the contents import: create ckan package(dataset) 51. 2013/11/9HarvestersThe harvesting interfacefrom base import HarvesterBaseclass SRDAHarvester(HarvesterBase):def _set_config(self,config_str):def info(self):...def gather_stage(self, harvest_job):def fetch_stage(self, harvest_object):def import_stage(self, harvest_object):See the extension site for detailsAn example (SRDA): http://goo.gl/ZMnND7 52. 2013/11/9Agenda About CKAN Feature Tour Publish & Find Datasets Store & Manage Data Engage with Users & Others Customise & Extend CKAN and 5 Open Data Showcase Installation Harvesters Issues 53. 2013/11/9Issues CJK Support CJK Search Some broken translations Frequent updates Extensions compatibilities Tons of tweaks needed Performance Issue Complicated Architecture 54. Official Documents: http://docs.ckan.org/en/latest/ Installation Notes (in Chinese): https://ckan-docs-tw.readthedocs.org/ CKAN Development Discussions: http://lists.okfn.org/mailman/listinfo/ckan-dev CKAN Taiwan Interest Group: https://groups.google.com/forum/#!forum/ckan-taiwan-interest-group2013/11/9Resources 55. Thanks for your attention!2013/11/9Any Q?Email: u10313335 AT citi.sinica.edu.twhttp://about.me/sollee