contested collective intelligence: resilience, complexity & sensemaking
DESCRIPTION
PARC: Apr 1, 2011 Contested Collective Intelligence: Resilience, Complexity & Sensemaking Simon Buckingham Shum & Anna De Liddo Knowledge Media Institute, Open Learning Network Project Open University UK http://people.kmi.open.ac.uk/sbs http://people.kmi.open.ac.uk/anna ABSTRACT To thrive, organizational entities (learning communities; teams of analysts; formal companies) must make sense of a complex, changing environment. Our interest is in how sociotechnical “collective intelligence” infrastructures may augment this capacity. We are seeking conceptual lenses that illuminate this challenge, and draw ideas from resilience thinking, sensemaking, and complexity science. We propose that these motivate the concept of Contested Collective Intelligence (CCI), and give examples of how the Cohere platform is being designed in response to these requirements. This is a social/semantic web annotation and knowledge mapping environment, with tools for monitoring networks of ideas and generating novel analytics. We also report experimental integration with the Xerox Incremental Parser, in order to evaluate human+machine annotation of knowledge-level claims expressed through rhetorical moves in documents. Simon Buckingham Shum is a Senior Lecturer and Associate Director (Technology) at the UK Open University’s Knowledge Media Institute (KMi), where he leads the Hypermedia Discourse Group. Following a PhD at U. York in HCI/Hypertext/Design Rationale (sponsored by Xerox EuroPARC) he has developed a human-centered computing perspective to the challenge of computer-supported sensemaking, reflected in the books Visualizing Argumentation and Knowledge Cartography. He co-founded the Compendium Institute and LearningEmergence.net. http://people.kmi.open.ac.uk/sbs Anna De Liddo is a Research Associate in KMi, where she works with Simon on the Open Learning Network project (olnet.org), focusing on the design and development of a Collective Intelligence infrastructure for the Open Education Resources movement. She gained her PhD at Polytechnic of Bari, investigating ICT for Participatory Planning and Deliberation, after which she held a postdoctoral position in KMi evaluating human-centred argument mapping for Climate Change. http://people.kmi.open.ac.uk/annaTRANSCRIPT
Contested Collective Intelligence Resilience, Complexity & Sensemaking
1
Simon Buckingham Shum & Anna De Liddo Knowledge Media Institute, Open Learning Network Project Open University UK http://people.kmi.open.ac.uk/sbs http://people.kmi.open.ac.uk/anna
PARC, Apr 1st 2011
Acknowledgements
Open Learning Network project (2009-12): olnet.org funded by the William & Flora Hewlett Foundation
2
OLnet Collective Intelligence workstream: http://olnet.org/collective-intelligence
Developing conceptual foundations and infrastructure (people+proceses+tools) for Contested Collective Intelligence on the open social web.
Example: Open Education Evidence Hub http://ci.olnet.org (alpha)
Your team, organization, school, professional network, community...
3
Your team, organization, school, professional network, community...
4
Your team, organization, school, professional network, community...
5
How do we augment this system’s capacity to sense, respond to, and shape its environment?
6
§ Through the lens of complex adaptive systems, resilience and network science...
§ Through the lens of sensemaking and HCI...
§ Hypermedia Discourse: social-semantic web + models of discourse
How do we augment this system’s capacity to sense, respond to, and shape its environment?
7
§ Through the lens of complex adaptive systems, resilience and network science... § many interacting agents (human
and software) § many weak signals that can
build up unexpectedly § diversity and redundancy § feedback loops § visual analytics to reveal
emergent patterns and network properties
§ ability to withstand change and shock to the system
Resilience
§ Walker, et al. (2004) define resilience as
“the capacity of a system to absorb disturbance and reorganize while undergoing change, so as to still
retain essentially the same function, structure, identity, and feedbacks”
8
Resilience Platforms
9
http://www.futureofed.org/driver/Platforms-for-Resilience.aspx
Resilience Platforms
10
http://www.futureofed.org/driver/Platforms-for-Resilience.aspx
Creating flexibility and innovation amid system failures “Platforms for resilience - enabling responsive flexibility, distributed collaboration, and transparency - will allow institutions to meet such challenges through innovation, adaptation, and openness.”
Resilience in knowledge-intensive ecosystems
11
When knowledge and understanding are key variables in the system, resilience depends on the capacity for learning
e.g. awareness of discrepant evidence, critical practice, reflection and dialogue when confronted by challenges or shocks to the system.
How does this help?
Working hypothesis:
Confronted by overwhelming complexity... (e.g. incomplete, ambiguous data, complex adaptive systems, diverse perspectives, technical/social/political dimensions, time pressure…)
…Personal and Collective Cognition
break down in particular ways…
We need Theories, Tools and Practices in order to create CI for tackling such dilemmas
(and we need ways to teach these, both to our children, and the current workforce)
12
Augmenting human intellect (ack. Engelbart)
13
Phenomenon Role for CI infrastructure? Dangers of entrained thinking from experts who fail to recognise a novel phenomenon
• Pay particular attention to exceptions • Computer-supported argumentation • Make the system open to diverse
perspectives ontologically, and in usability
Complex systems only seem to make sense retrospectively: narrative is an appropriately complex form of knowledge sharing and reflection for such domains
• Stories and coherent pathways are important
• Reflection and overlaying of interpretation(s) is critical
Patterns are emergent • Generate gestalt views from the data evidenced in the platform, not from preconceptions
Much of the relevant knowledge is tacit, shared through discourse, not formal codifications
• Scaffold the formation of significant inter-personal, learning relationships
Many small signals can build over time into a significant force/change
• Enable individuals to highlight important events and connections à aggregate
• Recommend connections based on different kinds of significant relationship
Sources include: Weick (1995); Kurtz & Snowden (2003); Browning, L. and Boudès, T. (2005); Hagel et al (2010)
Designing CI to embody resilience principles
14
Resilience principle Role for CI infrastructure? build in the potential for diversity • manage diversity of worldviews, and the
tensions this sets up make tight feedback loops • shared awareness of dis/agreement
amongst peers promote building of trust/social capital
particularly for learning and sensemaking
• using social media to build learning relationships: trust, affirmation, challenge
enable experimentation • effective dissemination of findings in relation to key issues and what is already known
use a decentralised, modular architecture
• both technically (enabling innovation, interoperability and mashups) but also in how we represent interpretations (ideas as networks, not big chunks of text)
a stable state – however temporary – in epistemic terms is a plausible narrative
• model key coherence relations; explore narrative indexing
How do we augment this system’s capacity to sense, respond to, and shape its environment?
15
§ Through the lens of sensemaking and HCI... § many plausible narratives: what
was, is, or might be going on?... § many representational artifacts
being shared and annotated § attention to the quality of
conversation: how well are agents listening to each other and what kinds of contributions do they make?
§ informal interaction mixed with stronger public claims
§ many connections being made, both formal and fuzzy
• critical thinking • argumentation • rhetorical moves • assumptions • analogical thinking
• causality • juxtapositions • “kinda related...”
Sensemaking: the search for plausible, narrative connections
§ In their review of sensemaking, Klein, et al. conclude:
§ “By sensemaking, modern researchers seem to mean something different from creativity, comprehension, curiosity, mental modeling, explanation, or situational awareness, although all these factors or phenomena can be involved in or related to sensemaking. Sensemaking is a motivated, continuous effort to understand connections (which can be among people, places, and events) in order to anticipate their trajectories and act effectively. […] A frame functions as a hypothesis about the connections among data.”
16
Sensemaking
Weick proposes that: § “Sensemaking is about such things as
placement of items into frameworks, comprehending, redressing surprise, constructing meaning, interacting in pursuit of mutual understanding, and patterning.” (Weick, [23], p.6)
17
Sensemaking
Weick: § “The point we want to make here is that
sensemaking is about plausibility, coherence, and reasonableness. Sensemaking is about accounts that are socially acceptable and credible” ([23] p.61)
18
contested collective intelligence...
conversations are critical to sensemaking
there is no master worldview
we need CI infrastructures to pool awareness of how people are reading small signals, and amplify important
connections
19
Where our tools fit… Given a wealth of documents…
20
Where our tools fit… Given a wealth of documents, and tools to detect and render potentially significant patterns…
21
Where our tools fit… Given a wealth of documents, and tools to detect and render potentially significant patterns…
22
Where our tools fit: making meaningful connections between information elements…
23
Where our tools fit: making meaningful connections between interpretations
24
interpretation
interpretation
interpretation
interpretation
Where our tools fit: making meaningful connections between interpretations
25
interpretation
interpretation interpretation
interpretation
interpretation
(a hunch – no grounding
evidence yet)
interpretation
Where our tools fit: making meaningful connections between information elements
26
predicts causes
interpretation
interpretation interpretation
interpretation
interpretation
(a hunch – no grounding
evidence yet)
interpretation
Is pre-requisite for
Where our tools fit: making meaningful connections between information elements
27
prevents
predicts causes
interpretation
interpretation interpretation
interpretation
interpretation
(a hunch – no grounding
evidence yet) Is inconsistent with
interpretation
challenges
Is pre-requisite for
Where our tools fit: building the story that makes sense of the evidence… i.e. plausible arguments
28
Question
Answer
Supporting Argument… Challenging
Argument…
challenges supports
responds to
Assumption
motivates
Where our tools fit: building the story that makes sense of the evidence… i.e. plausible arguments
29
Question
Answer
Supporting Argument… Challenging
Argument…
challenges supports
responds to
Hunch
motivates
Where our tools fit: building the story that makes sense of the evidence… i.e. plausible arguments
30
Question
Answer
Supporting Argument… Challenging
Argument…
challenges supports
responds to
Data
motivates
31
http://cohere.open.ac.uk
Convergence of… web annotation social bookmarking concept mapping structured debate
a prototype infrastructure for collective intelligence/social learning
Structured deliberation and debate in which Questions, Evidence and Connections are first class entities (linkable, addressable, embeddable, contestable…)
32
33
Structured deliberation and debate in which Questions, Evidence and Connections are first class entities (linkable, addressable, embeddable, contestable…)
— web annotation of OER (Firefox extension)
User/community-defined visual language
35
36
Structured deliberation and debate in which Questions, Evidence and Connections are first class entities (linkable, addressable, embeddable, contestable…)
Social Network
Social Discourse Network
Concept Network
By looking at the post type table it is possible to evaluate learner’s performance connecting the discourse outcomes with the specific learning goal.
Cohere analytics
Legend:
Positive link type
Negative link type
Neutral link type
Cohere analytics
Comparing usage of connections
Comparison of one’s own ideas to others
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L.(2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
Does the learner compare his/her own ideas to that of peers, and if so, in what ways?
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L.(2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
Does the learner act as a broker, connecting the ideas of his/her peers, and if so, in what ways?
Link broker: connecting other people’s ideas
seeing the connections people make as they annotate the web using Cohere
Visualizing all the connections that a set of analysts have made
— but unfiltered, this may not be very helpful
Visualizing multiple learners’ interpretations of global
warming sources
Connections have been filtered by a set of semantic
relationships grouped as Consistency
— semantic filtering of connections
De Liddo, A. and Buckingham Shum, S. (2010). Cohere: A prototype for contested collective intelligence. In: ACM Computer Supported Cooperative Work (CSCW 2010) - Workshop: Collective Intelligence In Organizations, February 6-10, 2010, Savannah, Georgia, USA. http://oro.open.ac.uk/19554
— web annotation for sensemaking
http
://w
ww
.flic
kr.c
om/p
hoto
s/ba
rtelo
meu
s/41
8470
5426
/
OLnet is searching out the
evidence for effec4ve OER, and building an Evidence Hub — a living map by, of and for
the OER movement — and those we need
to impact
!
Moving from document annotation to connection-making for sensemaking
Discourse analysis with Xerox Incremental Parser
BACKGROUND KNOWLEDGE:
Recent studies indicate …
… the previously proposed …
… is universally accepted ...
NOVELTY:
... new insights provide direct evidence ...
... we suggest a new ... approach ...
... results define a novel role ...
OPEN QUESTION: … little is known … … role … has been elusive Current data is insufficient …
GENERALIZING: ... emerging as a promising approach Our understanding ... has grown exponentially ... ... growing recognition of the importance ...
CONRASTING IDEAS: … unorthodox view resolves … paradoxes … In contrast with previous hypotheses ... ... inconsistent with past findings ...
SIGNIFICANCE: studies ... have provided important advances Knowledge ... is crucial for ... understanding valuable information ... from studies
SURPRISE: We have recently observed ... surprisingly We have identified ... unusual The recent discovery ... suggests intriguing roles
SUMMARIZING: The goal of this study ... Here, we show ... Altogether, our results ... indicate
Detection of salient sentences based on rhetorical markers:
Ágnes Sándor & OLnet Project: http://olnet.org/node/512
XIP annotation to Cohere
XIP annotations in Cohere’s Firefox Ideas sidebar
XIP annotations in Cohere’s Firefox Connections sidebar
XIP annotations visualized in Cohere (ack: prefuse)
XIP/Cohere integration: conclusions from analysis of the corpus (ack: Ágnes Sándor, XRCE)
§ Machine annotation can effectively draw attention to
key issues and contrasting ideas, in a cost effective and timely manner
§ Human annotation adds higher-level cognitive activities such as abstracting, contextualizing and summarizing.
An appropriate combination of both machine and human annotation can augment and enhance both human and machine analysis.
OpenEd Evidence Hub: ci.olnet.org an alpha release
54
OpenEd Evidence Hub: ci.olnet.org an alpha release
55
OpenEd Evidence Hub: ci.olnet.org an alpha release
56
57
olnet.org
ci.olnet.org