Evaluating Semantic Search Query Approaches with Expert and Casual

Users

Khadija Elbedweihy, Stuart N. Wrigley and Fabio CiravegnaOAK Research Group,

Department of Computer Science, University of Sheffield, UK

• Motivation• Research Question• Evaluation Design• Evaluation Setup• Findings• Conclusions

Outline

Motivation – Semantic Search

• Wikipedia states that Semantic Search “seeks to improve search accuracy by understanding searcher intent and the contextual meaning of terms as they appear in the searchable dataspace, whether on the Web or within a closed system, to generate more relevant results”.

• Covers broad category of applications in Semantic Web:– Search engines (e.g., Swoogle, FalconS, Sindice)– Closed-domain query interfaces (e.g., AquaLog, Querix)– Open-domain query interfaces (e.g., PowerAqua)

Motivation - Evaluations

• Evaluation of software is critical.

• Large-scale evaluations foster research and development.

• Semantic search evaluations (SemSearch, TREC ELC, QALD) focused on assessing retrieval performance.

Assessing usability of tools and user satisfaction is important in Semantic Search.

Research Question

How do different types of users perceive the usability of different query approaches?

• Method- Assess usability and user satisfaction of:

* Free-NL, Controlled-NL, Form-based, Graph-based

- from the perspective of * expert users and casual users

Query Approaches

Free-NLNatural language queries

What is the capital of Alabama? Submit

capital Alabama Submit

Controlled-NLSpecific vocabulary

Which state has Submitrivercapitallakemountainaany

Form-based Visualize the search space

Graph-based Visualize the search space

Evaluation Design: Dataset

• Mooney Natural Language Learning Data- simple and well-known domain (geography)- used by other studies within the search community- questions already available (877 NL questions)

• Geography Dataset:– Concepts: State, City, Lake, Mountain, Capital, River, etc– Properties: population of state, length of river, etc– Relations linking concepts: State ‘hasCity’ City

Evaluation Design: Data Collected

• Objective data:1) Input time 2) Number of attempts3) Success rate

• Subjective data, collected using: 1) Questionnaires (e.g., System Usability Scale ‘SUS’)2) Ranking of the tools (w.r.t: system, query approach,

results content, results presentation)3) Observations

Evaluation Setup

• 20 subjects – 10 casual users, 10 expert users– 12 females, 8 males

• Within-subjects: allows direct comparison.

• Randomising tool order: normalize learning or tiredness effects.

• Randomising question order: normalize learning effects.

Results

• Evaluated tools:

– Free-NL: NLP-Reduce

– Controlled-NL: Ginseng

– Form-based: K-Search

– Graph- based: • Semantic-Crystal (Graph-based 1)• Affective Graphs (Graph-based 2)

QUERY APPROACH

Results for expert users

• Expert users prefer graph- and form- based approach.• View-based allow more complex queries than NL-based.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Graph-based1Graph-based2Form-basedControlled-NLFree-NL

Que

ry L

angu

age

Ran

k

Best

Worst

Results for casual users

• Casual users prefer form-based query approach.• Required less input time than graph-based approach.

Que

ry L

angu

age

Ran

k

Best

Worst0

102030405060708090

100

Inpu

t Tim

e (S

ec)

00.10.20.30.40.50.60.70.80.9

1

Graph-based1Graph-based2Form-basedControlled-NLFree-NL

ONTOLOGY VISUALIZATION

Results for expert users• Visualizing the entire ontology supports query formulation

– Semantic Crystal: shows the entire ontology.– Affective Graphs: shows selected concepts & relations.

Results for casual users

• Not showing ontology more complex for casual users: – Semantic Crystal receiving higher scores.– Affective Graphs perceived as complex and difficult to use

• 50% of the users found it to increase complexity and difficulty

CONTROLLED-NL APPROACH

Results for expert users

• Controlled-NL very restrictive for expert users (least-liked) • Highest query input time

00.10.20.30.40.50.60.70.80.9

1

Graph-based1Graph-based2Form-basedControlled-NLFree-NL

Que

ry L

angu

age

Ran

k

Best

Worst0

20

40

60

80

100

120

Inpu

t Tim

e (S

ec)

Results for casual users

• Controlled-NL provided most support for casual users.

• Users’ positive feedback for controlled-NL:– allow only correct queries (50%)– suggestions and guidance to formulate queries (40%)

Example: Although Ginseng is limited to specific vocabulary, I knew that I will get answers once I can do the query because it only allows the correct ones and thus I didn't keep trying a lot of queries that I wasn't sure about.

RESULTS INDEPENDENT OF USER TYPE

Free-NL approach

+ simplest and most natural- suffer from habitability problem.

• Feedback: “I have to guess the right words”– Example: `run through’ with `river’ but not `traverse’.

• NLP-Reduce:– lowest success rate: 20%– highest number of attempts: 4.2

Negation

• Tell me which rivers do not traverse the state with the capital Nashville?

Expert Users Casual Users0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Graph-based1Graph-based2Form-basedControlled-NLFree-NL

Ans

wer

Fou

nd R

ate

Negation

Tell me which states does the river Mississippi does not traverse.

• “Closed world assumption (CWA): presumption that what is not currently known to be true is false”.

<Mississippi, traverse, Louisiana>

• “Open world assumption (OWA): assumption that the truth-value of a statement is independent of whether or not it is known by any single observer or agent to be true”.<Mississippi, not_traverse, Alabama>

Formal Query

• Formal Query (e.g., SPARQL)

Formal Query

• Benefit of showing formal query depends on user type.

• Formal query perceived by:

– Casual users: not understandable and confusing

– Expert users: increased confidence

Also, performing direct changes to the formal queryincreased the expressiveness of the query language.

Results presentation

• Results presentation and format affected usability and user satisfaction.– Unless users are very familiar with the data, presenting URIs

alone is not very helpful.

– Example: A query for rivers returns one of the answers:http://www.mooney.net/geo#tennesse2

Results Content

• Results should be augmented with associated information to provide a `richer’ user experience.

• Users feedback:– Maybe a `mouse over' function to show more

information.– Perhaps related information with the results.– Results very limited, would be good to have more

context.

CONCLUSIONS

Research Question & Approach

How do different types of users perceive the usability of different query approaches?

- Assess usability and user satisfaction of:* Free-NL, Controlled-NL, Form-based, Graph-

based

- from the perspective of * expert users and casual users

Conclusions

Expert Users• Graph-based most preferred - Intuitive - Support complex queries

• Controlled-NL least preferred - Very restrictive. - Limited expressiveness

• Prefer flexibility of free-NL• Formal query provides confidence - Ability to change query increases

expressiveness.

Casual Users• Form-based mid-point - Allow more complex queries than

NL. - Easier than graph-based - Faster than graph-based• Controlled-NL most supportive - Only valid queries: Confidence - Vocabulary suggestions: guidance

• Formal Query not understandable and confusing.

• Users want search results to be augmented with more information to have a better understanding of the answers.

Recommendations

Cater to both expert and casual users:

• Hybridized query approach: Combine a view-based approach (visualize search space) with a NL-input feature (balance difficulty and speed) while including optional suggestions for the NL input (provide guidance).

• Results Content: Augment results with ‘extra’ and ‘related’ information.– extra information: for ‘State’: capital, area, population.– related information: for ‘State’: rivers, lakes, mountains.

Limitations & Future work

• Limitation: Small size of the dataset.

• Assess learnability of different query approaches.

• Assess how interaction with the search tools affect the information seeking process: usefulness.

– Use questions with an overall goal and compare users' knowledge before and after the search task.

Questions

Questions?