mining social and semantic network data on the web · 2011-07-11 · mining social and semantic...

Mining Social and Semantic Network Data on the Web

Markus Schatten, PhD

University of ZagrebFaculty of Organization and Informatics

May 4, 2011

Introduction

Web 2.0, Semantic Web, Web 3.0

Network science

Data collectionSemistructured dataXPathRegular expressionsAjaxAPIs

Data analysisNetwork matricesFolksonomy modelLogic programming

Case study - CROSBI

2 of 54

Outline

Introduction

Network science

Data collection

Data analysis

Case study - CROSBI

3 of 54

Questions

� Why mining social and semantic network data?� How to collect these data?� What are the common obstacles and how to solve them?� How to analyze these data?

4 of 54

Outline

Introduction

Network science

Data collection

Data analysis

Case study - CROSBI

5 of 54

Why should one bother with these buzzwords?

6 of 54

Why mining (social) web data?

Millions of people,

on thousands of sites, across hundreds ofdiverse cultures, leave “trails” about various aspects of their lives...

“Social systems are systems of communication”

(Niklas Luhman)

By analyzing the “trails” (the results of social systems structuralcoupling) we can analyze the social system itself:politics, culture,media, business, technology, science ...

7 of 54

Millions of people, on thousands of sites,

across hundreds ofdiverse cultures, leave “trails” about various aspects of their lives...

(Niklas Luhman)

7 of 54

Millions of people, on thousands of sites, across hundreds ofdiverse cultures,

leave “trails” about various aspects of their lives...

(Niklas Luhman)

7 of 54

Millions of people, on thousands of sites, across hundreds ofdiverse cultures, leave “trails”

about various aspects of their lives...

(Niklas Luhman)

7 of 54

Millions of people, on thousands of sites, across hundreds ofdiverse cultures, leave “trails” about various aspects of their lives...

(Niklas Luhman)

7 of 54

(Niklas Luhman)

7 of 54

(Niklas Luhman)

By analyzing the “trails” (the results of social systems structuralcoupling) we can analyze the social system itself:

politics, culture,media, business, technology, science ...

7 of 54

(Niklas Luhman)

7 of 54

Outline

Introduction

Network science

Data collection

Data analysis

Case study - CROSBI

8 of 54

The “new” science of networks

source: The New York Times (April 3rd, 1933., p. 17).

source: An Attraction Network in a Fourth Grade Class (Moreno, Who shall survive?, 1934).

9 of 54

Political and financial networks

Mark Lombardi (1980s & 1990s)

10 of 54

Terrorist networks

11 of 54

Board of directors membership networks

source: http://theyrule.net

12 of 54

On-line social networks

13 of 54

Internet

source: Bill Cheswick http://www.cheswick.com/ches/map/gallery/index.html14 of 54

Air traffic networks

source: Northwest Airlines WorldTraveler Magazine

15 of 54

Railway networks

source: TRTA, March 2003 - Tokyo rail map

16 of 54

Semantic networks

source: http://wordnet.princeton.edu/man/wnlicens.7WN

17 of 54

Gene networks

source: http://www.zaik.uni-koeln.de/bioinformatik/regulatorynets.html.en

18 of 54

Food chain networks

source: http://marinebio.org/Oceans/Biotic-Structure.asp19 of 54

On-line social and semantic networks

Where can we find networks?� social networking sites

� forums, buletinboards, discussion groups, mailing lists� blogs, microblogs, vlogs� wikis, semantic wikis� social tagging� RSS feeds, RDF repositories, metadata collections...A social network can be found on any application where userscommunicate.A semantic network can be found in any specification whereconcepts are meaningfully connected.

20 of 54

Where can we find networks?� social networking sites� forums, buletinboards, discussion groups, mailing lists

� blogs, microblogs, vlogs� wikis, semantic wikis� social tagging� RSS feeds, RDF repositories, metadata collections...A social network can be found on any application where userscommunicate.A semantic network can be found in any specification whereconcepts are meaningfully connected.

20 of 54

Where can we find networks?� social networking sites� forums, buletinboards, discussion groups, mailing lists� blogs, microblogs, vlogs

� wikis, semantic wikis� social tagging� RSS feeds, RDF repositories, metadata collections...A social network can be found on any application where userscommunicate.A semantic network can be found in any specification whereconcepts are meaningfully connected.

20 of 54

Where can we find networks?� social networking sites� forums, buletinboards, discussion groups, mailing lists� blogs, microblogs, vlogs� wikis, semantic wikis

� social tagging� RSS feeds, RDF repositories, metadata collections...A social network can be found on any application where userscommunicate.A semantic network can be found in any specification whereconcepts are meaningfully connected.

20 of 54

Where can we find networks?� social networking sites� forums, buletinboards, discussion groups, mailing lists� blogs, microblogs, vlogs� wikis, semantic wikis� social tagging

� RSS feeds, RDF repositories, metadata collections...A social network can be found on any application where userscommunicate.A semantic network can be found in any specification whereconcepts are meaningfully connected.

20 of 54

Where can we find networks?� social networking sites� forums, buletinboards, discussion groups, mailing lists� blogs, microblogs, vlogs� wikis, semantic wikis� social tagging� RSS feeds, RDF repositories, metadata collections

...A social network can be found on any application where userscommunicate.A semantic network can be found in any specification whereconcepts are meaningfully connected.

20 of 54

Where can we find networks?� social networking sites� forums, buletinboards, discussion groups, mailing lists� blogs, microblogs, vlogs� wikis, semantic wikis� social tagging� RSS feeds, RDF repositories, metadata collections...

A social network can be found on any application where userscommunicate.A semantic network can be found in any specification whereconcepts are meaningfully connected.

20 of 54

Where can we find networks?� social networking sites� forums, buletinboards, discussion groups, mailing lists� blogs, microblogs, vlogs� wikis, semantic wikis� social tagging� RSS feeds, RDF repositories, metadata collections...A social network can be found on any application where userscommunicate.

A semantic network can be found in any specification whereconcepts are meaningfully connected.

20 of 54

Where can we find networks?� social networking sites� forums, buletinboards, discussion groups, mailing lists� blogs, microblogs, vlogs� wikis, semantic wikis� social tagging� RSS feeds, RDF repositories, metadata collections...A social network can be found on any application where userscommunicate.A semantic network can be found in any specification whereconcepts are meaningfully connected.

20 of 54

Outline

Introduction

Network science

Data collectionSemistructured dataXPathRegular expressionsAjaxAPIs

Data analysis

Case study - CROSBI21 of 54

Common obstacles in data collection

� Semi-structured data (HTML,XHTML, XML, RDF, OWL ...)

� The need for pattern matching

� Client-side content generation

22 of 54

Semistructured data and XPath

� query language for semistructured data� part of the W3C standard� uses path expressions to navigate through documents� has a standard library with over 100 useful functions

Example:

/html/body/div[@id=’background’]/div/div[@id=’mainBar’]/div/p[last()-1]/strong

23 of 54

XPather (Firefox add-on)

24 of 54

Regular expressions

� Useful for extracting various formatted data and searching forkeywords

Example (date matching)27-9-2001 [0-9]{1,2}[0-9]{1,2}[1-2][0-9]{3}

25 of 54

Client side generated content� Some sites use Ajax to generate data on page load or user

interaction (e.g. mouse click, mouse over etc.)

� In combination with obfuscated JavaScript code this is hard toscrape!

� Current solutions:� Browser scripting (can be slow and cumbersome) - e.g. with

selenium� Headless browser - e.g. mechanize, spidermonkey, HtmlUnit

26 of 54

interaction (e.g. mouse click, mouse over etc.)� In combination with obfuscated JavaScript code this is hard to

scrape!� Current solutions:

� Browser scripting (can be slow and cumbersome) - e.g. withselenium

� Headless browser - e.g. mechanize, spidermonkey, HtmlUnit

26 of 54

Open APIs

� Most popular social web sites have an open API that allow youto connect your application to their site (Facebook, Twitter,Wikipedia, ...)

� Still such APIs often have drawbacks for network datacollection:� Limited access rights (e.g. Facebook)� Limited queries per time interval (e.g. Twitter) etc.

27 of 54

Open APIs

� Still such APIs often have drawbacks for network datacollection:

� Limited access rights (e.g. Facebook)� Limited queries per time interval (e.g. Twitter) etc.

27 of 54

Open APIs

� Still such APIs often have drawbacks for network datacollection:� Limited access rights (e.g. Facebook)

� Limited queries per time interval (e.g. Twitter) etc.

27 of 54

Open APIs

� Still such APIs often have drawbacks for network datacollection:� Limited access rights (e.g. Facebook)� Limited queries per time interval (e.g. Twitter) etc.

27 of 54

Outline

Introduction

Network science

Data collection

Data analysisNetwork matricesFolksonomy modelLogic programming

Case study - CROSBI

28 of 54

Networks are most often analyzed using graph theory:

DefinitionA graph G is the pair (N , E) whereby N represents the set of verticles ornodes, and E ⊆ N ×N the set of edges connecting pairs from N .

DefinitionA directed graph or digraph G is the pair (N ,A), whereby N representsthe set of nodes, and A ⊆ N ×N the set of ordered pairs of elementsfrom N that represent the set of graph arcs.

DefinitionA valued or weighted graph GV is the triple (N ,A,V) whereby Nrepresents the set of nodes or verticles, A ⊆ N ×N the set of pairs ofelements from N that represent the set of graph arcs, and V : N → R afunction that attaches values or weights to nodes.

29 of 54

DefinitionA bipartite graph G = (N , E) is a graph for which set of nodes thereexists an partition N = {X , Y}, such that every arc has one end in X, aand the other in Y .

DefinitionA multigraph G = (N , E) is a graph in which E is a multiset, e.g. therecan exist multiple edges between two nodes.

30 of 54

DefinitionA bipartite graph G = (N , E) is a graph for which set of nodes thereexists an partition N = {X , Y}, such that every arc has one end in X, aand the other in Y .

DefinitionA multigraph G = (N , E) is a graph in which E is a multiset, e.g. therecan exist multiple edges between two nodes.

30 of 54

Matrix representation

DefinitionLet G be a graph defined with the set of nodes {n1, n2, ..., nm} and edges{e1, e2, ..., el}. For every i , j (1 6 i 6 m and 1 6 j 6 m) we define

{1, if there is an edge between nodes ni and nj

0, otherwise

Matrix A = [aij ] is then the adjacency matrix of graph G. The matrix isymmetric since if there is an edge between nodes ni and nj then clearlythere is also an edge between nj and ni . Thus A = [aij ] = [aji ].

31 of 54

Simple graph

32 of 54

Digraph

33 of 54

Digraph

34 of 54

Weighted graph

35 of 54

Bipartite graph

36 of 54

Multigraph

37 of 54

Matrix algebra and interpretation

Matrices can be transposed, multiplied ...

Examples:P - process flow matrix (p × p, p - number of processes)W - co-worker matrix (w × w , w - number of workers)O - process responsibility matrix (p × z)WW - co-workers of co-workersPT - inversed process flowOTO - matrix of workers who are responsible for the sameprocessesPO - matrix of workers who are responsible for the forthcomingprocessesPTO - matrix of workers who are responsible for the previousprocesses...

38 of 54

Matrices can be transposed, multiplied ...Examples:P - process flow matrix (p × p, p - number of processes)

W - co-worker matrix (w × w , w - number of workers)O - process responsibility matrix (p × z)WW - co-workers of co-workersPT - inversed process flowOTO - matrix of workers who are responsible for the sameprocessesPO - matrix of workers who are responsible for the forthcomingprocessesPTO - matrix of workers who are responsible for the previousprocesses...

38 of 54

Matrices can be transposed, multiplied ...Examples:P - process flow matrix (p × p, p - number of processes)W - co-worker matrix (w × w , w - number of workers)

O - process responsibility matrix (p × z)WW - co-workers of co-workersPT - inversed process flowOTO - matrix of workers who are responsible for the sameprocessesPO - matrix of workers who are responsible for the forthcomingprocessesPTO - matrix of workers who are responsible for the previousprocesses...

38 of 54

Matrices can be transposed, multiplied ...Examples:P - process flow matrix (p × p, p - number of processes)W - co-worker matrix (w × w , w - number of workers)O - process responsibility matrix (p × z)

WW - co-workers of co-workersPT - inversed process flowOTO - matrix of workers who are responsible for the sameprocessesPO - matrix of workers who are responsible for the forthcomingprocessesPTO - matrix of workers who are responsible for the previousprocesses...

38 of 54

Matrices can be transposed, multiplied ...Examples:P - process flow matrix (p × p, p - number of processes)W - co-worker matrix (w × w , w - number of workers)O - process responsibility matrix (p × z)WW - co-workers of co-workers

PT - inversed process flowOTO - matrix of workers who are responsible for the sameprocessesPO - matrix of workers who are responsible for the forthcomingprocessesPTO - matrix of workers who are responsible for the previousprocesses...

38 of 54

Matrices can be transposed, multiplied ...Examples:P - process flow matrix (p × p, p - number of processes)W - co-worker matrix (w × w , w - number of workers)O - process responsibility matrix (p × z)WW - co-workers of co-workersPT - inversed process flow

OTO - matrix of workers who are responsible for the sameprocessesPO - matrix of workers who are responsible for the forthcomingprocessesPTO - matrix of workers who are responsible for the previousprocesses...

38 of 54

Matrices can be transposed, multiplied ...Examples:P - process flow matrix (p × p, p - number of processes)W - co-worker matrix (w × w , w - number of workers)O - process responsibility matrix (p × z)WW - co-workers of co-workersPT - inversed process flowOTO - matrix of workers who are responsible for the sameprocesses

PO - matrix of workers who are responsible for the forthcomingprocessesPTO - matrix of workers who are responsible for the previousprocesses...

38 of 54

Matrices can be transposed, multiplied ...Examples:P - process flow matrix (p × p, p - number of processes)W - co-worker matrix (w × w , w - number of workers)O - process responsibility matrix (p × z)WW - co-workers of co-workersPT - inversed process flowOTO - matrix of workers who are responsible for the sameprocessesPO - matrix of workers who are responsible for the forthcomingprocesses

PTO - matrix of workers who are responsible for the previousprocesses...

38 of 54

Matrices can be transposed, multiplied ...Examples:P - process flow matrix (p × p, p - number of processes)W - co-worker matrix (w × w , w - number of workers)O - process responsibility matrix (p × z)WW - co-workers of co-workersPT - inversed process flowOTO - matrix of workers who are responsible for the sameprocessesPO - matrix of workers who are responsible for the forthcomingprocessesPTO - matrix of workers who are responsible for the previousprocesses

38 of 54

Matrices can be transposed, multiplied ...Examples:P - process flow matrix (p × p, p - number of processes)W - co-worker matrix (w × w , w - number of workers)O - process responsibility matrix (p × z)WW - co-workers of co-workersPT - inversed process flowOTO - matrix of workers who are responsible for the sameprocessesPO - matrix of workers who are responsible for the forthcomingprocessesPTO - matrix of workers who are responsible for the previousprocesses...

38 of 54

ACI model (Mika 2007)

� A - actors

� C - concepts� I - instances

� T ⊆ A× C × I - folksonomy

A tripartite hypergraph which can be reduced into three bipartitegraphs:� AC - network of actors and concepts� AI - network of actors and instances� CI - network of concepts and instances

39 of 54

� A - actors� C - concepts

� I - instances

39 of 54

� A - actors� C - concepts� I - instances

39 of 54

A tripartite hypergraph which can be reduced into three bipartitegraphs:

� AC - network of actors and concepts� AI - network of actors and instances� CI - network of concepts and instances

39 of 54

A tripartite hypergraph which can be reduced into three bipartitegraphs:� AC - network of actors and concepts

� AI - network of actors and instances� CI - network of concepts and instances

39 of 54

A tripartite hypergraph which can be reduced into three bipartitegraphs:� AC - network of actors and concepts� AI - network of actors and instances

� CI - network of concepts and instances

39 of 54

Delicious - analysis

40 of 54

Problems with matrix representation

� Only the “1”-s contain information

41 of 54

Problems with matrix representation

� Only the “1”-s contain information41 of 54

Sparse matrix

42 of 54

Problems with sparse matrix

� If A is connected to B, then B is connected to A (undirectedgraphs)

� How to find friends of friends?� Is there a path from node A to node B? Which path is that?� How to compute the previously outlined graphs?

43 of 54

� How to find friends of friends?

� Is there a path from node A to node B? Which path is that?� How to compute the previously outlined graphs?

43 of 54

� How to find friends of friends?� Is there a path from node A to node B? Which path is that?

� How to compute the previously outlined graphs?

43 of 54

Logic programming as a solution

� Deductive languages like Prolog, FLORA-2

� Advantages:� Declarative problem solving (“on a higher level”)� Very appropriate for expressing mathematical structures

� Disadvantages:� Combinatoric explosion (use heuristics and predicate tabling)

44 of 54

� Deductive languages like Prolog, FLORA-2� Advantages:

� Declarative problem solving (“on a higher level”)� Very appropriate for expressing mathematical structures

44 of 54

� Deductive languages like Prolog, FLORA-2� Advantages:

� Declarative problem solving (“on a higher level”)� Very appropriate for expressing mathematical structures

44 of 54

Examples

If A is connected to B, then B is connected to A

link( A, B ) :- link( B, A ).

45 of 54

Examples

Friend of a friend

ff( A, B ) :- link( A, C ), link( C, B ).

46 of 54

Examples

Is there a path from node A to node B?

path( A, B ) :- link( A, B ).path( A, B ) :- link( A, C ), path( C, B ).

47 of 54

Examples

Which path is that?

whichpath( X, Y, [ X, Y ] ) :- link( X, Y ).whichpath( X, Y, [ X, Z | T ] ) :-

link( X, Z ),whichpath( Z, Y, [ Z | T ] ).

48 of 54

Outline

Introduction

Network science

Data collection

Data analysis

Case study - CROSBI

49 of 54

Project outline

� Croatian Scientific Bibliography (http://bib.irb.hr)

� Scientists enter data about their research activities by themselves which makes it a social application

� Research hypotheses:1. analyze two social systems (the Croatian scientific community and

the Croatian public) and see how the most important researchconcepts (as viewed by the scientific community) are interpretedby the public (does the public understand most important conceptsfrom Croatian science in the last decade?).

2. analyze how various scientific fields are conceptually and sociallyinterrelated (do scientists do interdisciplinary research together ifthey have conceptually connected fields?)

50 of 54

Project outline

� Croatian Scientific Bibliography (http://bib.irb.hr)� Scientists enter data about their research activities by them

selves which makes it a social application

� Research hypotheses:1. analyze two social systems (the Croatian scientific community and

the Croatian public) and see how the most important researchconcepts (as viewed by the scientific community) are interpretedby the public (does the public understand most important conceptsfrom Croatian science in the last decade?).

50 of 54

Project outline

selves which makes it a social application� Research hypotheses:

1. analyze two social systems (the Croatian scientific community andthe Croatian public) and see how the most important researchconcepts (as viewed by the scientific community) are interpretedby the public (does the public understand most important conceptsfrom Croatian science in the last decade?).

50 of 54

Project outline

50 of 54

Project outline

50 of 54

Methodology

� Social systems are represented by their “trails”: CROSBI as theCroatian scientific community, Croatian Wikipedia as theCroatian public

� One system can interpret concepts from another if the conceptexists within its conceptual network (e.g. keyword or wiki page)

51 of 54

Data collection

� CROSBI data has been collected using Scrapy in November2010 (a total of 285,234 entries has been collected)

� Wikipedia data has been collected using the Wikipedia API +Scrapy for parsing

52 of 54

Results

53 of 54

Bibliography

1. Adamic, L.: Why networks are interesting to study, University of Michigan, School of Information,https://open.umich.edu/education/si/si508/fall2008

2. Barratm M., Barthlemy, M., Vespignani, A.: Dynamical Processes on Complex Networks, Cambridge UniversityPress, 2008.

3. Carley, K.M.: Dynamic Network Analysis, Summary of the NRC workshop on social network modeling andanalysis, Committee on Human Factors, National Research Council, 133145, Eds: Breiger, R., Carley, K.M., andPattison, P., 2003.

4. Divjak, B., Lovrencic, A.: Diskretna matematika s teorijom grafova, TIVA & FOI, 2005

5. Krackhardt, David, and Carley, Kathleen M.: A PCANS Model of Structure in Organization, Proceedings of the1998 International Symposium on Command and Control Research and Technology, Evidence Based Research,Vienna, VA, June 1998

6. Mika, Peter: Ontologies are us: A unified model of social networks and semantics, Web Semantics: Science,Services and Agents on the World Wide Web 5(1), volume 5, 515, March 2007

7. Newman, M., Barabsi, A.-L., Watts, D. j.: The Structure and Dynamics of Networks, Princeton University Press,2006.

8. Various web sites (images, graphs)

54 of 54

mining social and semantic network data on the web · 2011-07-11 · mining social and semantic...

Documents

enhancing semantic mining

@twitter mining #microblogs using #semantic technologies

semantic media mining seminar - kickoff

towards semantic web mining

web usage mining with semantic analysis

schatten van afmetingen

data-mining the semantic web @tcd

semantic data mining: an ontology based approach

text mining in poolparty semantic suite

workshop semantic web mining

shadowrun - deutschland in den schatten

hadassa - im schatten roms - 9783865918901

text mining and the semantic web

semantic data...

2nd workshop semantic web mining

data mining and semantic web

mining social and semantic network data on the...

: web usage mining with semantic analysis

fuzzy ontology mining and semantic information granulation

a semantic relationship mining method among disorders