3 rd a3eh workshop at 12 th international conference on artificial intelligence, amsterdam, 2005 tu/...
TRANSCRIPT
![Page 1: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/1.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Evaluation of Interoperability of Adaptive Hypermedia Systems:
testing the MOT to WHURLE conversion in a classroom setting
Alexandra Cristea1, Craig Stewart2, Tim Brailsford2 and Paul Cristea3
1 - Information System Department, Faculty of Mathematics and Computing Science, Technical
University Eindhoven2 - School of Computer Science and IT, University of
Nottingham3 - Digital Signal Processing Laboratory,
“Politehnica” University of Bucharest
![Page 2: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/2.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Outline
• Introduction• MOT• WHURLE• MOT2WHURLE conversion• Student Evaluation
– Hypotheses– Results
• Conclusion
![Page 3: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/3.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Introduction
• Creation of adaptive hypermedia content:– Extremely time intensive– Complex– Platform lockdown– Lack of standard tools
![Page 4: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/4.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Introduction
• Ideal creation of adaptive hypermedia content:– Automated (this will aid with the time
and complexity factors)– Interoperable (conversion, common
language)– Standardised
![Page 5: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/5.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Introduction
• Our answer:– Develop a series of tools that will allow
for interoperability between different systems
– We use MOT as an authoring system to author materials for:• AHA! (an AEH)• WHURLE (an AEH)• Blackboard (a non-adaptive commercial
system, uses a pre-adapted methodology)
![Page 6: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/6.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Introduction
• Here we present an evaluation of the authoring process using MOT and WHURLE
• A class of 31 students were introduced to this new authoring paradigm
![Page 7: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/7.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
MOT
• My Online Teacher• Based on the LAOS framework• Is a generic AEH delivery system• Also a powerful and simple
authoring system– Web-form, therefore a non-technical
author can easily use it– Very flexible, as it is easy to extend
![Page 8: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/8.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
MOT
• Domain Maps
![Page 9: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/9.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
MOT
• Goal and Constraint Maps– AND/OR– Weights– Labels
![Page 10: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/10.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
WHURLE
• An adaptive XML learning environment
• Basic content building block: Chunk
• Structure applied using Lesson Plans
![Page 11: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/11.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
MOT2WHURLE
• Currently a command line environment
• Conceptually maps the MOT Goal & Constraints map structure to the WHURLE Lesson Plan– A MOT Concept = WHURLE chunks(s)
• It does this using the MOT weights and labels.
![Page 12: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/12.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
MOT2WHURLE
![Page 13: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/13.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
MOT2WHURLE
• These rules are used to determine which MOT attributes are to be aggregated into WHURLE chunks.– ’35’ = visual– ’75’ = verbal– ‘0’ = common
![Page 14: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/14.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
MOT2WHURLE
![Page 15: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/15.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Student Evaluation
• A class of 31 students in the 4th year of study for a technical Masters degree at the University of Bucharest, Romania
• All subjected to a week long intensive course on AH
• Ideal representation for non-AH expert authors.
• After initial lectures, they were given the following task:
![Page 16: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/16.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Student Evaluation1. Create 2-3 MOT Domain Concept Maps, with
approximately 5-10 concepts on the http://e-learning.dsp.pub.ro/mot/ MOT server
2. The attributes of each concept were: title; keywords; introduction; text; conclusion and figure. With limits placed on the type and amount of content in each one (this was done so as that each group would not spend their limited time creating a vast corpus of information).
3. Create a single MOT Lesson (Goal & Constraints Map) using their Concepts maps.
4. Alter the lesson so that the weights and labels of each concept agreed with those described in the second Table previously.
5. Run the ‘mot2whurle’ conversion program and copy the files to WHURLE.
![Page 17: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/17.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Student Evaluation6. Check that the WHURLE XML files are well-formed.7. Run and login to WHURLE to check that the lesson
matches their design and make any necessary changes.
8. Finally at the end of the week, each student was asked to complete a series of questionnaires: three generic SUS (System Usability Scale) questionnaires, one for each system (MOT, mot2whurle and WHURLE) and a single specific questionnaire designed to determine their level of knowledge about each system, as well as to gather non-statistical information.
![Page 18: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/18.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Evaluation hypotheses• The hypotheses that we wished to examine were:
1. The systems (MOT, mot2whurle, WHURLE) are simple and intuitive to use, with a minimum amount of explanation.
2. The students understood the theoretical background (Adaptive Hypermedia, LAOS, Adaptive Strategies) of these systems.
3. The students understood the connection between LAOS and MOT.
4. The students used MOT purely for authoring adaptive hypermedia, and perceived it as such.
5. The students used WHURLE solely for delivering adaptive hypermedia, and perceived it as such.
6. Students consider automatic conversion between one-to-many or many-to-many adaptive hypermedia systems useful.
![Page 19: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/19.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Evaluation Results
• SUS scores:– MOT
0
0,5
1
1,5
2
2,5
3
3,5
4
4,5
5
use frequently
complex
easy
need support
well integrated
inconsistency
learn quickly
cumbersome
confident to use
need to learn a lot to use
75%
![Page 20: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/20.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Evaluation Results
• WHURLE
0
0,5
1
1,5
2
2,5
3
3,5
4
4,5
5
use frequently
complex
easy
need support
well integrated
inconsistency
learn quickly
cumbersome
confident to use
need to learn a lot to use
66.6%
![Page 21: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/21.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Evaluation Results
• MOT2WHURLE
0
0,5
1
1,5
2
2,5
3
3,5
4
4,5
5
use frequently
complex
easy
need support
well integrated
inconsistency
learn quickly
cumbersome
confident to use
need to learn a lot to use
60.7%
![Page 22: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/22.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Evaluation Results
• Hypothesis 1:– Overall SUS scores:
• MOT: 75% ±15%• WHURLE: 66.6% ±19.1%• MOT2WHURLE: 60.7% ±19%
– NB: SUS scores are comparative
![Page 23: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/23.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Evaluation Results• Hypotheses 2-5 extracted from
questionnaires.• Hypothesis 2:
[The students understood the theoretical background (Adaptive Hypermedia, LAOS, Adaptive Strategies) of these systems.]
– 70% ±24% • Hypothesis 3:
[The students understood the theoretical background (Adaptive Hypermedia, LAOS, Adaptive Strategies) of these systems.]
– 70% ±21.7%• Indicates that students on average
understood the theoretical background after an intensive introduction.
• However the wide SD indicates a great range of abilities.
![Page 24: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/24.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Evaluation Results
• Hypothesis 4:[The students used MOT purely for authoring adaptive hypermedia, and perceived it as such.]
– 25 students out of 29 (86%) selecting MOT to be an adaptive hypermedia authoring system
– Indicates that students understand the benefits of using MOT as an authoring system.
![Page 25: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/25.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Evaluation Results
• Hypothesis 5:[The students used WHURLE solely for delivering adaptive hypermedia, and perceived it as such.]
– 21 students out of 29 (72%) selecting WHURLE as an adaptive hypermedia delivery system
– Indicates that students understand the benefits of using WHURLE as a delivery system.
![Page 26: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/26.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Evaluation Results
• Hypothesis 6:[Students consider automatic conversion between one-to-many or many-to-many adaptive hypermedia systems useful.]
– 4.57 (out of 5) approval rating.– Indicates that students understand
the benefits of the paradigm shift that we are proposing.I.E. away from a ‘one-to-one’ paradigm.
![Page 27: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/27.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Conclusion
• There has been one previous attempt to convert content between systems: AHA! & Interbook.
• This is the first (that we are aware of) to empirically test the actual authoring and conversion process between two systems.
![Page 28: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/28.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Conclusion
• The responses from this study have validated our efforts to:– Create a powerful & flexible authoring
environment– Create an interoperable methodology– Use a single authoring environment to
deliver content to other AEH delivery systems
![Page 29: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/29.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Conclusion
• Major disadvantages:– The majority of which concerned the
lack of ‘polish’ for the MOT2WHURLE conversion program.
– This does not invalidate our methodology but does indicate that additional work needs to be done.
![Page 30: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/30.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Conclusion
• It appears that a “write once, use often” authoring methodology is desired by authors.
• MOT is a simple to use, yet flexible authoring system.
• Conceptual mapping between MOT and other authoring systems demonstrates the feasibility of such and approach
![Page 31: 3 rd A3EH workshop at 12 th International Conference on Artificial Intelligence, Amsterdam, 2005 TU/ e eindhoven university of technology Evaluation of](https://reader035.vdocument.in/reader035/viewer/2022070306/55160ddc550346d46f8b60e3/html5/thumbnails/31.jpg)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/e
eindhoven university of technology
Questions … ?