relevance ranking of learning objects
DESCRIPTION
Presentation at EC-TEL 2007 (Crete). The need for relevance rank is presented and basic metrics are presented and evaluated.TRANSCRIPT
![Page 1: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/1.jpg)
Relevance Ranking Metrics for Learning Objects
Xavier Ochoa, ESPOL, Ecuador
Erik Duval, KULeuven, Belgium
![Page 2: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/2.jpg)
Agenda
• Why Relevance Ranking?
• What is Relevance?
• Relevance Ranking Metrics
• Do they work?
• What is next?
![Page 3: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/3.jpg)
Economy of Abundance
![Page 4: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/4.jpg)
Economy of Abundance
Put your LMS here!
![Page 5: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/5.jpg)
Why Relevance Ranking?
Abundance
=
Difficult to find most relevant
![Page 6: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/6.jpg)
Why Relevance Ranking?
Abundance
=
Difficult to find most relevant
Solution:Do it the Google way
![Page 7: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/7.jpg)
Why Relevance Ranking?
Abundance
=
Difficult to find most relevant
Relevance RankingMEANINGFUL & SCALABLE
![Page 8: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/8.jpg)
How we do it now?
• Manual rating:–Meaningful but not Scalable
• Text based Ranking Algorithms:–Scalable but not Meaningful
![Page 9: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/9.jpg)
What is Relevance?
![Page 10: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/10.jpg)
Relevance Ranking Metrics
Relevance Ranking
Relevance Ranking
MetricB
MetricA
MetricC
Context
![Page 11: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/11.jpg)
Relevance Ranking Metrics
Relevance Ranking
Relevance Ranking
MetricB
MetricA
MetricC
Context
Meaningful
![Page 12: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/12.jpg)
Relevance Ranking Metrics
Relevance Ranking
Relevance Ranking
MetricB
MetricA
MetricC
Context
Scalable
![Page 13: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/13.jpg)
Topi
cal R
elev
ance
![Page 14: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/14.jpg)
Personal Relevance
![Page 15: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/15.jpg)
Situational Relevance
![Page 16: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/16.jpg)
Do they work?
• Exploratory Study– 10 users (8 Prof. and 2 R.A.)– MIT OCW learning objects (34,640 LO)– Search and Select Objects to create 10
lessons (10 different topics on CS)– Web application was used
![Page 17: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/17.jpg)
![Page 18: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/18.jpg)
![Page 19: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/19.jpg)
![Page 20: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/20.jpg)
Ranking the Rankings
• Baseline Rank: TF-IDF similarity algorithm
• Re-rank according to Basic Metrics
• All rankings compared with manual relevance ranking
• Kendall Tau was used to measure similaritiy between lists
![Page 21: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/21.jpg)
Results
![Page 22: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/22.jpg)
Basic Topical Relevance MetricTrees
![Page 23: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/23.jpg)
Basic Personal Relevance Metric
Net.
![Page 24: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/24.jpg)
Basic Situational Relevance Metric
XML
![Page 25: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/25.jpg)
Linear Combination
![Page 26: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/26.jpg)
Linear Combination
![Page 27: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/27.jpg)
Results
• Even basic metrics improve the ranking
• Linear Combination should be learnt
• Limited (and Synthetic) Study!
![Page 28: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/28.jpg)
What is next?
• Implementation in ARIADNE NEXT
• Capture user behavior and learn from it for 3 months
• Natural (Real) evaluation of the metrics
• Develop BETTER metrics
![Page 29: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/29.jpg)
Conclusions
• Meaningful and Scalable metrics are possible
• Cheap to implement • Provide significant improvement• While not optimal, they could be
use as a base-line for further research
![Page 30: Relevance Ranking of Learning Objects](https://reader033.vdocument.in/reader033/viewer/2022052823/5551443bb4c905bd1c8b4ea7/html5/thumbnails/30.jpg)
Thank You! Dank U! Gracias!• Questions, Comments, Critics…
are all welcome!!
Xavier [email protected]
http://www.cti.espol.edu.ec/xavier
Erik [email protected] http://www.cs.kuleuven.ac.be/~erikd