Wikis and Semanticsin NYC and Open Government

Joel Natividad, co-founder@jqnatividad

SemTech West June 2012San Francisco, CA

Human-powered,Machine-accelerated,

Collective Knowledge Systems

CROWDKNOWING

“Smart” Cities brand of Ontodia

"Over  the  next  decade,  cities  will  continue  to  grow  larger  at  a  rapid  pace.  At  the  same  time,  new  technologies  will  unlock  massive  streams  of  data  about  cities  and  their  residents.  As  these  forces  collide,  they  will  turn  every  city  into  a  unique  civic  laboratory—a  place  where  technology  is  adapted  in  novel  ways  to  meet  local  needs."  

December  2010.    Institute  of  the  Future's  2020  Forecast  –  The  Future  of  Cities,  Information  and  Inclusion.  

Ci#es:Worldwide,  city  leaders  and  managers  need  cost-­‐effective  &  smart  solutionsPopula'on  Growth:

-­‐ 221  cities  globally  with  more  than  1  million  citizens-­‐  China  will  move  300  million  people  to  cities  by  2020-­‐  90%  of  these  cities  are  in  emerging  markets-­‐  In  2008,  more  people  lived  in  cities  (3.3  billion),  by  2030,  5  billion-­‐ Cities  are  more  efJicient  and  have  less  environmental  impact

Cost  of  City  Services:Aging  infrastructure,  resource  constraints  &  waste

-­‐ Washington  DC’s  water  system  has  elements  that  date  to  the  Civil  War-­‐ InefJiciency,  leaks  and  waste  rival  maintenance  and  expansion  costs-­‐ Legacy  infrastructure  in  megacities  like  NYC  that  are  too  cost-­‐prohibitive  to  replace

source:  Gartner  –  Is  Smart  Cities  the  Next  Big  Market?    March  2011

Open Data in NYC

Council Member Gale Brewer

Int. No. 29: Accessibility to Public Data Sets

“...requires  that  all  public  data  sets  maintained  by  City  agencies  shall  be  made  available  on  the  Internet  through  a  single  web  portal,  formatted  to  enable  viewing  by  web  browsers  and  mobile  devices  and  also  in  their  raw  or  unprocessed  form.”

Why NYC?• City Population:

8.4 M (NYC estimate)

• Metro NYC Population: 18.9 million (2010 Census)

• City Density: 10,630/km2 (2010 Census)

• Metro NYC Density: 1,085.7/km2 (2010 Census)

• 50 million visitors a year

• DoITT Annual Budget ~$325M

• Gross Metropolitan Product: USD $ 1.28 Trillion (Greyhill Advisors)

• De facto Capital of the World

• Fastest growing Tech Industry - “New Tech City” (Center for an Urban Future)

• Second only to Silicon Valley for most startups

• Emphasis on public-private partnerships

Large Organization Award

Grand Prize

0. Huge Open Data1. Extract Metadata

2. Derive ExtraMetadata (Semantics + Statistics + Algorithm + Crowd)

3. Do Federated Queries on both the Metadata AND the Data

Crowdknowing

Human-powered, Machine-accelerated, Collective Knowledge Systems

Crowdknowing

Curation, Comments, Microcontributions, Feedback,

Bug Reports,Likes, Shares, Profile, Votes,

Subscribes, Tagging, etc. etc. etc.

Ontology, Inferencing, Semantic Mapping, Query Federation, Statistics,

Pattern Recognition, Multivariate Analysis & Forecasting, Automated

linking, Feeds, Notificationsetc. etc. etc.

a Semantic Data Dictionary

Semantic Steroids

• Searchable• Faceted Search• Drilldown• Interlinked• Semantic Browsing• Queryable• Query Results Formats

~3.5M facts~950 datasets/views

• Derived using “Semantics, Statistics, Algorithm & the Crowd”

• “Supercharacterize” each datasetnot just the schema, but by sampling the underlying data as well

• Score each dataset - Pediacities Rank

• Virtuous Feedback Loop micro-conversations/contributions around the Data

ExtraMetadata?

Top Level ExtraMetadata

• Number of Rows

• Pediacities Rank

• Freshness Score• Sparseness Score• Social Score• Views Score• Download Score• Rating Score

Detail ExtraMetadata

• Top Values

• Descriptive statistics

• Nulls/Non-nulls• Smallest Value• Largest Value• “Uniqueness”

• Simple Visualization

ExtraMetadata

Microconversations/contributions

• Overall Rating

• Comments (comment rating)

• Bug Reports (data quality)

• Likes/Shares

• Downloads

“Crowd”

NYC Open Data PoliHack Unconference - May 19, 2012

nyc.gov/datastandards

•  More  Datasources!•  Not  just  Metadata!  Data  too!•  Federated  Queries!  •  SPARQL  support•  Collaborative  Ontology  Modeling•  Feeds  /  Subscriptions  /  NotiQications•  Microcontributions•  GamiQication•  combine  NYCDataWeb  and  NYCFacets•  Support  both  Web  2.0  &  Web  3.0  APIs

in time for

4.0

As of September 2011

MusicBrainz

(zitgist)

P20

Turismo de

Zaragoza

yovisto

Yahoo! Geo

Planet

YAGO

World Fact-book

El ViajeroTourism

WordNet (W3C)

WordNet (VUA)

VIVO UF

VIVO Indiana

VIVO Cornell

VIAF

URIBurner

Sussex Reading

Lists

Plymouth Reading

Lists

UniRef

UniProt

UMBEL

UK Post-codes

legislationdata.gov.uk

Uberblic

UB Mann-heim

TWC LOGD

Twarql

transportdata.gov.

uk

Traffic Scotland

theses.fr

Thesau-rus W

totl.net

Tele-graphis

TCMGeneDIT

TaxonConcept

Open Library (Talis)

tags2con delicious

t4gminfo

Swedish Open

Cultural Heritage

Surge Radio

Sudoc

STW

RAMEAU SH

statisticsdata.gov.

uk

St. Andrews Resource

Lists

ECS South-ampton EPrints

SSW Thesaur

us

SmartLink

Slideshare2RDF

semanticweb.org

SemanticTweet

Semantic XBRL

SWDog Food

Source Code Ecosystem Linked Data

US SEC (rdfabout)

Sears

Scotland Geo-

graphy

ScotlandPupils &Exams

Scholaro-meter

WordNet (RKB

Explorer)

Wiki

UN/LOCODE

Ulm

ECS (RKB

Explorer)

Roma

RISKS

RESEX

RAE2001

Pisa

OS

OAI

NSF

New-castle

LAASKISTI

JISC

IRIT

IEEE

IBM

Eurécom

ERA

ePrints dotAC

DEPLOY

DBLP (RKB

Explorer)

Crime Reports

UK

Course-ware

CORDIS (RKB

Explorer)CiteSeer

Budapest

ACM

riese

Revyu

researchdata.gov.

ukRen. Energy Genera-

tors

referencedata.gov.

uk

Recht-spraak.

nl

RDFohloh

Last.FM (rdfize)

RDF Book

Mashup

Rådata nå!

PSH

Product Types

Ontology

ProductDB

PBAC

Poké-pédia

patentsdata.go

v.uk

OxPoints

Ord-nance Survey

Openly Local

Open Library

OpenCyc

Open Corpo-rates

OpenCalais

OpenEI

Open Election

Data Project

OpenData

Thesau-rus

Ontos News Portal

OGOLOD

JanusAMP

Ocean Drilling Codices

New York

Times

NVD

ntnusc

NTU Resource

Lists

Norwe-gian

MeSH

NDL subjects

ndlna

myExperi-ment

Italian Museums

medu-cator

MARC Codes List

Man-chester Reading

Lists

Lotico

Weather Stations

London Gazette

LOIUS

Linked Open Colors

lobidResources

lobidOrgani-sations

LEM

LinkedMDB

LinkedLCCN

LinkedGeoData

LinkedCT

LinkedUser

FeedbackLOV

Linked Open

Numbers

LODE

Eurostat (OntologyCentral)

Linked EDGAR

(OntologyCentral)

Linked Crunch-

base

lingvoj

Lichfield Spen-ding

LIBRIS

Lexvo

LCSH

DBLP (L3S)

Linked Sensor Data (Kno.e.sis)

Klapp-stuhl-club

Good-win

Family

National Radio-activity

JP

Jamendo (DBtune)

Italian public

schools

ISTAT Immi-gration

iServe

IdRef Sudoc

NSZL Catalog

Hellenic PD

Hellenic FBD

PiedmontAccomo-dations

GovTrack

GovWILD

GoogleArt

wrapper

gnoss

GESIS

GeoWordNet

GeoSpecies

GeoNames

GeoLinkedData

GEMET

GTAA

STITCH

SIDER

Project Guten-berg

MediCare

Euro-stat

(FUB)

EURES

DrugBank

Disea-some

DBLP (FU

Berlin)

DailyMed

CORDIS(FUB)

Freebase

flickr wrappr

Fishes of Texas

Finnish Munici-palities

ChEMBL

FanHubz

EventMedia

EUTC Produc-

tions

Eurostat

Europeana

EUNIS

EU Insti-

tutions

ESD stan-dards

EARTh

Enipedia

Popula-tion (En-AKTing)

NHS(En-

AKTing) Mortality(En-

AKTing)

Energy (En-

AKTing)

Crime(En-

AKTing)

CO2 Emission

(En-AKTing)

EEA

SISVU

education.data.g

ov.uk

ECS South-ampton

ECCO-TCP

GND

Didactalia

DDC Deutsche Bio-

graphie

datadcs

MusicBrainz

(DBTune)

Magna-tune

John Peel

(DBTune)

Classical (DB

Tune)

AudioScrobbler (DBTune)

Last.FM artists

(DBTune)

DBTropes

Portu-guese

DBpedia

dbpedia lite

Greek DBpedia

DBpedia

data-open-ac-uk

SMCJournals

Pokedex

Airports

NASA (Data Incu-bator)

MusicBrainz(Data

Incubator)

Moseley Folk

Metoffice Weather Forecasts

Discogs (Data

Incubator)

Climbing

data.gov.uk intervals

Data Gov.ie

databnf.fr

Cornetto

reegle

Chronic-ling

America

Chem2Bio2RDF

Calames

businessdata.gov.

uk

Bricklink

Brazilian Poli-

ticians

BNB

UniSTS

UniPathway

UniParc

Taxonomy

UniProt(Bio2RDF)

SGD

Reactome

PubMedPub

Chem

PRO-SITE

ProDom

Pfam

PDB

OMIMMGI

KEGG Reaction

KEGG Pathway

KEGG Glycan

KEGG Enzyme

KEGG Drug

KEGG Com-pound

InterPro

HomoloGene

HGNC

Gene Ontology

GeneID

Affy-metrix

bible ontology

BibBase

FTS

BBC Wildlife Finder

BBC Program

mes BBC Music

Alpine Ski

Austria

LOCAH

Amster-dam

Museum

AGROVOC

AEMET

US Census (rdfabout)

Media

Geographic

Publications

Government

Cross-domain

Life sciences

User-generated content

As of September 2011

MusicBrainz

(zitgist)

P20

Turismo de

Zaragoza

yovisto

Yahoo! Geo

Planet

YAGO

World Fact-book

El ViajeroTourism

WordNet (W3C)

WordNet (VUA)

VIVO UF

VIVO Indiana

VIVO Cornell

VIAF

URIBurner

Sussex Reading

Lists

Plymouth Reading

Lists

UniRef

UniProt

UMBEL

UK Post-codes

legislationdata.gov.uk

Uberblic

UB Mann-heim

TWC LOGD

Twarql

transportdata.gov.

uk

Traffic Scotland

theses.fr

Thesau-rus W

totl.net

Tele-graphis

TCMGeneDIT

TaxonConcept

Open Library (Talis)

tags2con delicious

t4gminfo

Swedish Open

Cultural Heritage

Surge Radio

Sudoc

STW

RAMEAU SH

statisticsdata.gov.

uk

St. Andrews Resource

Lists

ECS South-ampton EPrints

SSW Thesaur

us

SmartLink

Slideshare2RDF

semanticweb.org

SemanticTweet

Semantic XBRL

SWDog Food

Source Code Ecosystem Linked Data

US SEC (rdfabout)

Sears

Scotland Geo-

graphy

ScotlandPupils &Exams

Scholaro-meter

WordNet (RKB

Explorer)

Wiki

UN/LOCODE

Ulm

ECS (RKB

Explorer)

Roma

RISKS

RESEX

RAE2001

Pisa

OS

OAI

NSF

New-castle

LAASKISTI

JISC

IRIT

IEEE

IBM

Eurécom

ERA

ePrints dotAC

DEPLOY

DBLP (RKB

Explorer)

Crime Reports

UK

Course-ware

CORDIS (RKB

Explorer)CiteSeer

Budapest

ACM

riese

Revyu

researchdata.gov.

ukRen. Energy Genera-

tors

referencedata.gov.

uk

Recht-spraak.

nl

RDFohloh

Last.FM (rdfize)

RDF Book

Mashup

Rådata nå!

PSH

Product Types

Ontology

ProductDB

PBAC

Poké-pédia

patentsdata.go

v.uk

OxPoints

Ord-nance Survey

Openly Local

Open Library

OpenCyc

Open Corpo-rates

OpenCalais

OpenEI

Open Election

Data Project

OpenData

Thesau-rus

Ontos News Portal

OGOLOD

JanusAMP

Ocean Drilling Codices

New York

Times

NVD

ntnusc

NTU Resource

Lists

Norwe-gian

MeSH

NDL subjects

ndlna

myExperi-ment

Italian Museums

medu-cator

MARC Codes List

Man-chester Reading

Lists

Lotico

Weather Stations

London Gazette

LOIUS

Linked Open Colors

lobidResources

lobidOrgani-sations

LEM

LinkedMDB

LinkedLCCN

LinkedGeoData

LinkedCT

LinkedUser

FeedbackLOV

Linked Open

Numbers

LODE

Eurostat (OntologyCentral)

Linked EDGAR

(OntologyCentral)

Linked Crunch-

base

lingvoj

Lichfield Spen-ding

LIBRIS

Lexvo

LCSH

DBLP (L3S)

Linked Sensor Data (Kno.e.sis)

Klapp-stuhl-club

Good-win

Family

National Radio-activity

JP

Jamendo (DBtune)

Italian public

schools

ISTAT Immi-gration

iServe

IdRef Sudoc

NSZL Catalog

Hellenic PD

Hellenic FBD

PiedmontAccomo-dations

GovTrack

GovWILD

GoogleArt

wrapper

gnoss

GESIS

GeoWordNet

GeoSpecies

GeoNames

GeoLinkedData

GEMET

GTAA

STITCH

SIDER

Project Guten-berg

MediCare

Euro-stat

(FUB)

EURES

DrugBank

Disea-some

DBLP (FU

Berlin)

DailyMed

CORDIS(FUB)

Freebase

flickr wrappr

Fishes of Texas

Finnish Munici-palities

ChEMBL

FanHubz

EventMedia

EUTC Produc-

tions

Eurostat

Europeana

EUNIS

EU Insti-

tutions

ESD stan-dards

EARTh

Enipedia

Popula-tion (En-AKTing)

NHS(En-

AKTing) Mortality(En-

AKTing)

Energy (En-

AKTing)

Crime(En-

AKTing)

CO2 Emission

(En-AKTing)

EEA

SISVU

education.data.g

ov.uk

ECS South-ampton

ECCO-TCP

GND

Didactalia

DDC Deutsche Bio-

graphie

datadcs

MusicBrainz

(DBTune)

Magna-tune

John Peel

(DBTune)

Classical (DB

Tune)

AudioScrobbler (DBTune)

Last.FM artists

(DBTune)

DBTropes

Portu-guese

DBpedia

dbpedia lite

Greek DBpedia

DBpedia

data-open-ac-uk

SMCJournals

Pokedex

Airports

NASA (Data Incu-bator)

MusicBrainz(Data

Incubator)

Moseley Folk

Metoffice Weather Forecasts

Discogs (Data

Incubator)

Climbing

data.gov.uk intervals

Data Gov.ie

databnf.fr

Cornetto

reegle

Chronic-ling

America

Chem2Bio2RDF

Calames

businessdata.gov.

uk

Bricklink

Brazilian Poli-

ticians

BNB

UniSTS

UniPathway

UniParc

Taxonomy

UniProt(Bio2RDF)

SGD

Reactome

PubMedPub

Chem

PRO-SITE

ProDom

Pfam

PDB

OMIMMGI

KEGG Reaction

KEGG Pathway

KEGG Glycan

KEGG Enzyme

KEGG Drug

KEGG Com-pound

InterPro

HomoloGene

HGNC

Gene Ontology

GeneID

Affy-metrix

bible ontology

BibBase

FTS

BBC Wildlife Finder

BBC Program

mes BBC Music

Alpine Ski

Austria

LOCAH

Amster-dam

Museum

AGROVOC

AEMET

US Census (rdfabout)

Media

Geographic

Publications

Government

Cross-domain

Life sciences

User-generated content

.NYC

.NYC - the First Linked Open Data City

We need your help & feedback

A Smart Data Exchange for All Data NYC

Find out more athttp://nyc.pediacities.com/facets

@jqnatividad @samimirzabaig @pediacities @ontodia

CREDITS• Flickr User Weston Price, Paleo-Caveman-Omnivore-

LowCarb-Meat-Diet-Info (http://www.flickr.com/photos/paleo-atkins-meat-diet-info/with/6718805047/)

• Flickr User Gao Yi (http://www.flickr.com/photos/gaoyi/178514677/)

• Senator Arlen Specter being confronted at a Town Hall meeting after passage of Healthcare Reform Act (Bradley C Bower-AP)

• Several pictures taken from NYC.gov/NYCEDC properties, Tumblr and Flickr accounts