specification of ecaad methodology v3 - next-tell · 2013. 5. 28. · students [ learning through...

© NEXT-TELL consortium: all rights reserved

Deliverable D2.5

Specification of ECAAD Methodology V3

Identifier: NEXT-TELL-D2.5-MTO-ECAAD_Tools_V3_V06.doc

Deliverable number: D2.5

Author(s) and company: Peter Reimann, MTO Michael Kickmeier-Rust, TUG Gerhilde Meissl-Egghart, TALK Wilfrid Utz, BOC

Internal reviewer: Roland Unterberger, JRS

Work package / task: WP02

Document status: Final

Confidentiality: Public

Version 2013-04-03

D2.5 Specification of ECAAD Methodology V3

© NEXT-TELL consortium: all rights reserved page ii

History

Version Date Reason of change

1 2013-01-02 document created

2 2013-01-30 First set of contributions

3 2013-02-25 First consolidated version

4 2013-03-25 Ready for internal review

5 2013-03-30 Final after internal review

6 2013-04-03 Some layout optimisations - final version submitted to EC

Impressum

Full project title: Next Generation Teaching, Education and Learning for Life

Grant Agreement No: 285114

Workpackage Leader: Peter Reimann, MTO

Project Co-ordinator: Harald Mayer, JRS

Scientific Project Leader: Peter Reimann

Acknowledgement: The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 258114.

Disclaimer: This document does not represent the opinion of the European Community, and the European Community is not responsible for any use that might be made of its content.

This document contains material, which is the copyright of certain NEXT-TELL consortium parties, and may not be reproduced or copied without permission. All NEXT-TELL consortium parties have agreed to full publication of this document. The commercial use of any information contained in this document may require a license from the proprietor of that information.

Neither the NEXT-TELL consortium as a whole, nor a certain party of the NEXT-TELL consortium warrant that the information contained in this document is capable of use, nor that use of the information is free from risk, and does not accept any liability for loss or damage suffered by any person using this information.


© NEXT-TELL consortium: all rights reserved page iii

Table of Contents

1 Executive Summary .................................................................................................................................... 1

2 Introduction ................................................................................................................................................ 2

2.1 Purpose of this Document ......................................................................................................................... 2

2.2 Scope of this Document ............................................................................................................................. 2

2.3 Status of this Document ............................................................................................................................. 2

2.4 Related Documents .................................................................................................................................... 2

3 General rationale of the V3 specification .................................................................................................... 3

4 ECAAD Planner V3 ...................................................................................................................................... 4

4.1 Observations on ECAAD V2 Application/Demonstration ........................................................................... 4

4.2 Lessons learned and Year 3 development work ........................................................................................ 4

5 Methods and Tools for STEM learning and assessment V3 ......................................................................... 9

6 Methods and Tools for TESL learning and assessment V3 ......................................................................... 12

6.1 OpenSim ................................................................................................................................................... 12 6.1.1 Real-time feedback .................................................................................................................................................................. 12 6.1.2 Tool Integration ....................................................................................................................................................................... 13

6.2 Moving the “Chatterdale scenario” towards a unified method for virtual and physical learning environments ........................................................................................................................................... 13

6.3 Integrating TESL with 21C learning: Facilitating team work .................................................................... 16 6.3.1 The classroom practice ............................................................................................................................................................ 16 6.3.2 NEXT-TELL technology support ................................................................................................................................................ 17

7 Conclusions ............................................................................................................................................... 19

8 References ................................................................................................................................................ 20

9 Glossary .................................................................................................................................................... 21


© NEXT-TELL consortium: all rights reserved page 1

1 Executive Summary

The ECAAD methodology (ECAAD stands for Evidence-centered Activity and Assessment Design) comprises the planning of ICT-enhanced learning activity sequences, and the design of ICT-supported formative assessment methods that can be integrated into the learning activity sequences. Thus, ECAAD provides three elements for the other NEXT-TELL layers: (a) it provides a ‘script’ for the learning activities and the (mostly embedded) assessment processes that are to be realized in the Learning Environment, (b) it provides information on what aspects of students’ learning activities need to be tracked and which aspects of students’ work need to be stored in the e-portfolio, and (c) it provides information on how to transform data on students’ activities and products into information about students’ competencies that can be rendered in the Open Learner Model. The Open Learner Model is the main interface in NEXT-TELL for teachers, students, and parents.

This technical specification document describes mainly how we move in version 3 technical development from a concept of usability for the teacher (which was the main requirement for version 2) to a more general notion of usability: usable in classrooms. The ECAAD level tools revision 3 will be designed to take into account today’s classroom ICT, and today’s teaching and learning practices around classroom ICT. This means in particular that we are aiming for accommodating the fact that the relation of computer to student is often still far from 1:1, and that ICT focused activities occur spread out thinly over a school week. The use of ICT is by and large denser and more regular in students’ home, for instance, when doing homework.

The deliverable starts by identifying requirements and specifications for revision 3 of the ECAAD planner. A main goal for R3 is to provide feedback on an activity/lesson plan at design time to the user, for instance in terms of estimated cognitive density. A second goal is to develop a version of the planner that can be put into the hand of students, for instance to develop plans for their self-guided learning and plans for group work. As regards revision 3 of ECAAD level tools for STEM teaching, the focus will be on capturing information on students’ learning through tablets, and making learning analytics based on that available to the teacher. For teaching English as a Second Language (TESL), the focus is also on classroom integration: Immersive learning activities will be linked to activities in classrooms, i.e., in physical space, and in a second line of work face-to-face group work in classrooms will be blended together with on-line activities in a manner flexible enough to accommodate a wide variety of classroom infrastructure and pedagogies.



2 Introduction

2.1 Purpose of this Document

The document specifies version 3 of the ECAAD method and tool support in terms of modifications to and extensions of version 2 (see D2.4). ECAAD stands for Evidence-centred Activity and Assessment Design. The main activities to be supported comprise on the most general level domain analysis (what is the relevant knowledge in a particular domain, how is it learned and used, how do you know it when you see it), domain modeling (which aspects of students activities need to be recorded and how do these get transformed into statements about students proficiencies), and the delivery of assessment either in explicit form (e.g. through a quiz engine) or in embedded form (e.g., the analysis of part of a log file comprising a student's interaction with a simulation program). The ECAAD framework explicates the interrelationships among substantive arguments from the domain, assessment designs, and operational processes of assessment delivery and embedding. This document further specifies the method and the tool support for conducting ECAAD analysis and planning, its application in STEM and TESL learning, and further steps to integrating the modeler/planner with the learning environment provided in NEXT-TELL.

2.2 Scope of this Document

This document provides an update to specifications for ECAAD method and tools, not the method and tools itself (those will be covered in D2.6)

2.3 Status of this Document

This is the final version of D2.5

2.4 Related Documents

Before reading this document it is recommended to be familiar with the following documents:

D2.1

D2.3

D2.4



3 General rationale of the V3 specification

Infante and Nussbaum (Infante, 2010) make an important distinction between three kinds of barriers to the integration of ICT into classrooms: First order barriers are those that a teacher cannot control, such as the quality of the network infrastructure; second order barriers are those related to the teacher, such as skills and competencies, motivation, values and attitudes. These first two kinds of barriers have been extensively studied. But there is a third order barrier that Infante and Nussbaum identify, which is much less researched and understood: concrete classroom practices, and how they interact with integration of innovative ICT tools: “This category relates to the direct needs of teachers at the time of teaching their class, to the social relationships that occur in the classrooms and to the environment that is propitious for learning” (p. 2).

The argument that in order to integrate ICT tools deeply into work practices their design needs to be informed by an understanding of these practices is by no means new (e.g., Béguin, 2003), but this heuristic has not been much applied for the case of school ICT. Otherwise, we probably would not see so much use of off-the-shelve “office” software, for instance. Attending to practice aspects for integration of ICT is particularly important in the case of schools and classrooms because of the importance of classroom management for the success of learning. The pedagogical literature stresses the importance of regulatory relations in the pedagogical relationship. Following Bernstein, Infante and Nussbaum (2010) summarize: “Regulative refers to teacher control about what is happening inside classroom; and to the rules that govern what both teacher and student understand to be valid to do in the classroom. These are the social rules that are required in order to achieve an appropriate behavior in the pedagogical relationship” (p. 7).

With V3 of ECAAD level tools, we are aiming more explicitly than before at designing for the use in the classroom, addressing first order (the classroom’s spatial and technical aspects), second order (teacher capacity) and third order (classroom management) barriers.



4 ECAAD Planner V3

4.1 Observations on ECAAD V2 Application/Demonstration

The observation on the application of the ECAAD planner V2 build on feedback received and documented in the course of school studies and demonstration to the NEXT-TELL advisory board meeting. The observations made and conclusions drawn are summarized below:

Applicability to current planning practices: ECAAD focus needs to focus on how current practices could be enhanced/supported continuously. The feedback gathered from the user studies suggests, that the overhead exposed by a detailed, methodological sound planning tool as ECAAD is too restrictive and needs to be open and flexible to “understand” existing practices from different backgrounds (cultural, country-specifics). This is also valid for pre-service teachers and the adaptation to the local planning approach implemented.

Project-focused planning: for teachers involved in project-work, the planner is provided on a too detailed level; project-work especially creative sessions cannot be structured and pre-planned on a level that ECAAD suggests. Simplification in the actual planning task of unstructured sequence elements and the degree of freedom in describing these elements need to be modified accordingly. An approach to integrate structured sequences and unstructured ad-hoc elements is foreseen for Y3.

Teacher/Student collaboration: a main item as already identified after year 1, the collaborative features of the planner need to be enhanced not only from a teacher-to-teacher perspective, but on a negotiation level between teachers and students (proposal system, negotiation means). The hesitation in sharing information among each other on teacher level as already described in D2.3 is therefore moved to a different level building on a proposal-based approach.

Technological viewpoint: integration and runtime-view on models needs to be strengthened to enable a comprehensive view on plans. This means that the added-value from the planning effort using ECAAD instead of regular planning instruments like spread sheets or the like should become obvious.

4.2 Lessons learned and Year 3 development work

For V3 of ECAAD the following focus points for development have been identified based upon the experiences and observations outlined above and a review on current state-of-practice in the TEL field and related domains:

ECAAD-Y3-F1 Continuous planning feedback: as mentioned above, NEXT-TELL does not intend to revolutionize teachers’ practices through a comprehensive planning method and toolset but aims at supporting and externalizing added-value in current practices by exposing a step-by-step procedure model and appealing visualisation/externalisation of expert knowledge. This is on one hand accomplished by the selected meta-modelling technology that allows for flexible adaptation and personalization of the planning procedure and approach to local needs, on the other hand continuous feedback algorithms/mechanisms are foreseen on meta-model level. In year 3 it is planned to enhance the feedback engine to support distinct a scoring algorithm that provides instant feedback during the planning task in the quality of the model, potentially from a cognitive densities viewpoint.

ECAAD-Y3-F2 Learner/Student persona: as already mentioned in D2.3, for Y3 it is intended to “open” the planning system to students/learners to enable a self-planning of learning activities and assessment approach following a proposal-based approach. Students should have the possibility to suggest plans to the teacher for negotiation and approval. For this purpose the level of formality of the plans developed is reduced to support an abstract a free form of input specification.

ECAAD-Y3-F3 Lesson Plans as Apps: as already communicated in Y2, the self-planning engagement of teachers is limited due to mainly time-restrictions, therefore for Y3 it is planned to pre-package full lessons and lesson fragments for application. The focus lies upon pre-structuring certain elements, also relating to 21

st century skill set that allow for an easy integration and combination in pre-existing



lesson designs. The intention of the packages is not to provide learning objects of content for a specific domain but to enhance generically the uptake of cross-curriculum skills such as meeting facilitation, communication and the like.

ECAAD-Y3-F4 Annotation mechanisms: with respect to simplification on UI level, powerful search mechanisms and reporting features are foreseen to make the existing plan repository accessible. Lesson plans are available by searching on currently full-text level, through annotation, plans and plan elements with key-value pairs, the search experience is enhanced and allow for a hybrid combination approach of existing fragments with newly developed elements. The same mechanism is also used on the feedback requirement defined in ECAAD-Y3-F1 for qualitative and quantitative annotation of plan elements and further calculation.

Each of the elements above (with the exception of F4, that is already described as a building block of F1) is further specified in the following, providing conceptual/architecture considerations, mock-ups and functional elicitation of requirements. General improvement and fixing issues as communicated by project partners and study participants are not included in this list and are performed in the course of update for Y3.

ECAAD-Y3-F1 Continuous planning feedback

Requirements

The planning task is currently regarded as overhead, since the structure approach adds a significant overhead to the current practice (re-formulization, adaptation to new technology, adaptation to new procedure). The added value of a formal description of the plans needs to be made transparent to the planner (documentation, visualization, configuration/deployment/execution – all aspects during the execution of the plans) by adding functionality that provides continuous feedback during the design phase of the plan. Examples from other domains are e.g. software development - static code analysis and model-driven development approaches, risk management - validation of effective design using models through dynamic mechanism such as simulation algorithms and human interaction – walkthrough, energy profiles for architects and designers - pre-calculation of energy-efficiency in the planning phases by designers and architects, validation of design and pre-view of efficiency from various viewpoints and the like.




Functionality: Annotation

Annotation mechanisms on library and instance level in form of key-value pairs with the possibility for the planner to override annotation values. The annotation DB as a new element in the architecture provides means to track annotation settings using time-stamps (for historical views).

Figure 1 Annotation Mechanism

Functionality: Calculation Model

Flexible calculation model: to derive an overall result for each plan/model, the planner has the chance to a) select from pre-existing plans on a multi-dimensional level (not only 1 indicator, but a set of indicators to be calculated for each plan)

Figure 2 Calculation model and Visualisation




Functionality: Ad-hoc calculation engine

The annotation and calculation model are ad-hoc interpreted and update the indicators per plan on defined user interactions (new object added, object deleted, annotation modified, new library element added). From a technological perspective, the model of above is automatically translated into a close-to-platform language to optimize re-calculation and response times.

Functionality: Visualization of “Effectiveness”

Visualization of design “effectiveness” in model: the updated calculation results are graphically represented using the visualization techniques and libraries from other WPs in NEXT-TELL. Example views are available above in Figure 2, plan values and thresholds are set in the calculation model definition.

Visualization options foreseen as plan plugins:

Traffic-light coding

Bar and pie charts for comparison

Line charts for evolution view

Heat maps as overlays to the visualized plans



ECAAD-Y3-F2 Learner/Student persona

Requirements

The role of the learner with respect to ECAAD was until now not in scope from a planning perspective but is targeted in the executional view (WP3). As communicated in D2.3, this adjustment is performed for Y3.

From a planning perspective, different interaction models are envisioned for V3 of the method and tools moving from a consumer perspective (models are developed by teachers/experts, students “follow” the models”) to a consumer/provider interaction/negotiation view. From a methodology point of view an additional preparing step is prepared that allows the student to a) propose a certain approach on how to learn, what to learn, what means to use, b) start a negotiation process with the assessing entity and c) “release” the negotiated plan to be followed on an individual basis.

For V3, the student persona is not only a consuming one with continuous and transparent review/viewing capability as well as suggestion functionality but also a producing stakeholder in the planning cycle.

Functionality: ECAAD light

To support the student and integrate the persona in the approach, ECAAD is enhanced to support ad-hoc creative planning without formal restrictions on the modeling platform. The plans are regarded in a first step as sketches of proposal for teachers to approve and potentially further refine during the course of negotiation. Capabilities of the WP4 results are integrated to support negotiation mechanisms.

From an ECAAD perspective it is essential to allow for a release of information from the student to the teacher. It is foreseen to enable a common “sandbox” environment for all NEXT-TELL participants as trial area, specific sub-sections for project work and individual groups in the model-repository for private investigation and work. Similar as the OpenUniversities Flashmeeting system, the editing token can be passed on from the student to the teacher and back to trigger a structured discussion process.

ECAAD-Y3-F3 Lesson Plans as Apps

Requirements

To enhance application of ECAAD and support current practices, it is foreseen to continue the library building approach as already started in Y2. Through literature reviews, internet search and experience reports/expert knowledge of teachers involved, the repository is continuously extended. As a next step, the repository is further structured to enable out-of-the-box solutions for teachers by applying the apps-metaphor on the content available. Apps can be considered as full lesson plans for specific generic skills (in relation to the 21

st century skill focus) to be integrated in subject-matter courses.

Functionality: Packaging and Instantiation

As currently the repository is considered as a collection of best-practices, libraries developed bottom-up through application, within Y3, packages of plans and plan fragments are created with well-defined configuration interfaces and annotated description to validated applicability. These annotation come from different perspectives, such as technology available in school, NEXT-TELL tools used, NEXT-TELL external tool infrastructure and learning environments, etc.

As a concluding aspect for Y3 deliveries of ECAAD V3, it is planned to release the results achieved with respect to the planner to the Open Models Initiative (www.openmodels.at) to allow community-driven assessment and continuous evaluation/update. This corresponds to the content developed and packaged on one hand, but also on the implementation of the ECAAD method in the tools. Initial contact with the initiative hosted by the University of Vienna has been established, NEXT-TELL seeks to be included as a content and method development project on the platform.

http://www.openmodels.at/



5 Methods and Tools for STEM learning and assessment V3

Learning Analytics and educational data mining (EDM) are recent buzz words in educational research. In principle, the idea is to find theoretical frameworks, models, procedures, and smart tools to record, aggregate, analyze, and visualize large scale educational data. The principle goal is to make educational assessment and appraisal more goal-oriented, pro-active, and beneficial for students. In short, learning analytics is supposed to enable formative assessment on a large basis of all kinds of information about a learner, on a large basis. Usually, the benefits are seen in the potential to reduce attrition through early risk identification, improve learning performance and achievement levels, enable a more effective use of teaching times, and improve learning design/instructional design [Siemens, 2011]. Insofar, the concept of learning analytics is quite close to ideas and objectives of NEXT-TELL.

Methods used for learning analytics and educational data mining are very broad, for example, social network analyses, activity tracking, error tracking, keeping of e-Portfolios, semantic analyses, or log file analyses. An extensive overview of educational data mining applications, developments and definitions has been given in [Baker, 2009]. EDM has its origins in multiple research areas such as statistics, data mining, machine learning, visualization and computational modeling aiming to automatically discover patterns and models from huge and extensively growing datasets. While in the beginnings of educational data mining most data was retrieved from experimental learning sets not lasting longer than a couple of weeks, nowadays such data is often tracked over the duration of a whole course. Collected data is analyzed to gain valuable insights on learning processes and aspects. With these enormous loads of data new challenges arise especially in visualizing and modeling the information in a manner still readable and interpretable for human stakeholders. While EDM is discovering patterns and models on scaled data, Learning Analytics further includes the needs of different stakeholders and the strength of human judgment in addition to computational measurements. The focus areas of both communities seem to be slightly different, although they follow similar goals that are on the one hand the improvement of educational technology and on the other hand evaluation of pedagogically sound instructional designs [Baker, 2012].

Learning analytics emphasizes on supporting pedagogical approaches by providing assistance to teachers in practically relevant questions. Data gained by learning analytics tools can be used to evaluate pedagogically sound learning designs within classroom settings. It is highly focusing on monitoring learner actions and their interactions with learning tools and learning peers [Lockyer, 2011]. Many attempts go towards visualizing these learning traces in order to make significant relationships explicit or even to allow a stakeholder to discover such relations themselves. Research using dashboards in learning analytics is described in [Duval, 2011]. As known in other domains like for instance sports the visualization of collected interaction data and the comparison with like-minded may lead to insights about unwelcome habits and better practices but also to higher motivation due to the playful introduction of competitiveness. In [Dychkoff, 2012] a learning analytics toolkit is presented that takes a step forward by allowing the course instructors to define their own learning indicators via combining traced student actions. This toolkit is designed to support the instructors in optimizing their teaching strategies.

EDM and learning analytics require collecting as much digital information about learners as possible, ideally in standardized formats. The problem we are facing in the European classrooms and also in the context of the NEXT-TELL project is that oftentimes the technical infrastructure is lacking or not accessible. Despite the very sparse laptop and tablet classes the access to computer labs in rather limited in European schools. In the worst case - which is a rather common case, unfortunately – children attend computer-based work once a week. The second hurdle is that not all teachers have a sufficient level of technical know-how to use a broader spectrum of ICT in the teaching and, not least, the installation of software and the tracking of student information is restricted by school policies. In addition, a continuous (or too frequent) working with digital media is not wanted by teachers. Teaching is still a personal face-to-face process between students and teachers; mostly likely a great portion of activities in and outside the classroom occurs in a non-digital manner.

Facing such challenges, NEXT-TELLs attempts to broaden the solutions for activity tracking with having the students working with electronic devices and, on the other hand, to deepening the mechanism for educational



data mining and learning analytics. In the first instance we are designing and developing an activity tracking tool that works browser-based and with all sorts of devices such as PCs, tablets, or even smartphones. These developments are a direct result of co-design work with schools in Austria.

The platform is named myCLass and offers very simple and usable features for tracking activities of students by mouse/finger clicks (on tablet devices), and it allows adjusting competencies and learning goals. The following image shows a conceptual sketch of the tool.

Sketch of the myClass activity tracking tab.

After teachers log in through any kind of device, they have access to their classes. A teacher selects a class, e.g. a mathematics class. On the left all assigned students are listed and can be selected manually. For each student then, three major tabs are available.

Diary

The first tab allows a teacher to take notes and make a diary. In essence, the functionality is comparable to a blog; teacher can add, edit and delete texts. The information gather in these diaries can be directly linked to the ProNIFA tool and can be analyzed in terms of competencies. This function is similar to the chat log analyzer module, a function that allows a rule-based analysis of log files (e.g., of Second Life sessions) and an automatic update of competence / competence state probabilities. As a concrete example, if a diary entry says “did well in the fraction test today”, the ProNIFA analysis might update all task-relevant competencies (such a dividing numbers).

Activities

The second tab supports a very fast tracking of activities of all kinds. As shown in the image above, a teacher can specify various activities (e.g., helping others, showing great insight, or being late). During the lessons, activities are tracked by a simple button click. As described above, activities can be assigned to competencies, but also in a learning analytics centred sense, the data may serve as pool for educational data mining and reasoning processes over the total records of students.

Competencies

The third tab allows teachers to adjust the competency probabilities or the achievement/completion level of learning goals. As shown in the following image, the tab lists all competencies / learning goals related to and a related slider control. The slider shows the current probability/achievement level and allows simple adjustments, independent of device and location. Such adjustments, for example, can be made during an excursion in a museum.



myClass’s slider-based adjustment of competency levels.

Learning Analytics and Visualizations

myClass, as mentioned, has an interface to the ProNIFA tool and its analytical features. In principle, the global vision of NEXT-TELL is to collect information from various sources (and myClass is a new, highly practical, and practice-oriented source coming out of direct cooperation with schools), feed the information into ProNIFA which analyzes the results (primarily with a focus on competencies), and passes the results to the OLM platform and perhaps also to personal archiving and presentation solutions such as the Mahara e-Portfolio system. For practical reasons, myClass itself also has visualization features optimized for a cross-platform use. As indicated in the following image, a teacher might display the information about activities, competencies, and progress in real-time directly through myClass.

myClass has built-in features to display learning information in real time.

In addition to the describe features, the tool will be equipped with a range of additional features such as authoring tools to define activities, interfaces to connect to student models, to visualize information, or to print export/print results. Technically the system is based on PHP and as direct API to the NEXT-TELL software infrastructure. Of course myClass shares functionality with the ProNIFA web services. A strong connectivity will be established to the Montessori activity tracking platform that is developed and currently applied by TALK. In parallel to the developments of myClass, we are working on a broader range of data mining algorithms and reasoning services for ProNIFA to increase the usefulness of the entire framework. For example, we might increasingly focus on the reasoning over activity data in terms of social aspects in the classroom. For that purpose EDM techniques must be explored and reviewed for their practical suitability and technical integration.



6 Methods and Tools for TESL learning and assessment V3

6.1 OpenSim

After having developed a reliable environment for running and tracking learning activities in OpenSim we now see the focus of our next developments in improving the overall usability of the system. The works with teachers during the school studies have shown that mainly two areas need improvement:

1. Realtime-feedback: As it is quite challenging to supervise an activity that’s going on simultaneously in a whole virtual city. Our teachers need detailed information about what is going on, in order to decide whether intervention is necessary or not. At this point tracking and updating competencies is less important.

2. Tool integration: Forwarding data about competencies from OpenSim via ProNIFA over to the OLM should run seamlessly without any manual intervention; thus, as soon as the activity is over, the teacher should see immediately its results in the OLM.

6.1.1 Real-time feedback

Which data does a teacher need in order to find out how his/her students are doing and which interventions are necessary?

The two most frequently needed interventions are:

Students need help: they got lost, they don’t know what to do, etc.

Students are not working properly on their task

If this information is provided in real-time at the teacher’s mobile device, then we imagine, that it is possible that one teacher manages one class without further assistance.

We have found indicators for both cases in the data of our logfile. Most of them need some information aggregation steps (combine different entries in the log) to come to a conclusion. Some examples are:

Indicator (in logfile)

Possible conclusions Students might need help, because …

Students might not be working properly, because …

Further findings

student enters regions

students stay in the same region all the time

… they have technical problems

… they don’t understand what to do

… they just run or fly around

students change region very quickly

… have not understand what to do in each region

… they purposelessly run or fly around (or play tag)

all students of one team always stay in the same region

the team members work nicely together

student touches hints

students have not yet touched any hint at all

… they didn’t find hints

… they are not looking for hints

one hint has not been touched at all

… students can’t find this hint



Indicator (in logfile)

Possible conclusions Students might need help, because …

Students might not be working properly, because …

Further findings

students was the first who touched a certain hint

winner (if the hint is the goal)

text typed language of typed text

… they type in their mother tongue instead of the language they should practice

no text typed … they have technical problems

… they do not participate

words from a “ban list” are used

… they use inappropriate language

6.1.2 Tool Integration

We have learned from our school studies that seamless integration of tools is essential for the acceptance of software solutions by teachers. Currently the translation of the logfiles that are produced in OpenSim to updated competencies in OLM need too many manual steps in ProNIFA: start ProNIFA, open the file with the rules, open the logfile, start logfile-analyzer, log into the OLM, call web services and transfer updated competencies.

Two enhancements shall be implemented:

1. A button “feed competencies into OLM” (or similar) in ProNIFA

2. A batch job that only requires access to the logfile and then does all steps automatically

6.2 Moving the “Chatterdale scenario” towards a unified method for virtual and physical learning environments

The latest developments in mobile computing open up a door to broaden the process that we developed and applied in OpenSim towards physical learning environments. This gives us the chance to meet a recommendation from the last review and move from specifically designed scenarios to more general applicability. In fact the general approach can be applied for all subjects (including STEM and TESL) in the same way.

The diagram below shows

how the usage of learning objects that are either part of the virtual or the physical environment can be tracked (we use QR tags for the physical materials),

generate logfiles in a database,

generate real time formative feedback for immediate use in the classroom,

perform competence assessment in ProNIFA ,

and finally update competences in the OLM.



Tool chain

Introducing mobile devices into daily school routine demands certain criteria to be met:

1. Accessibility: If the application is not accessible in class it fails. Stable Internet, forgotten passwords

and malfunction software – all these are substantial risks and need to be taken care of. There is only

one chance for a first impression when approaching teacher: We address this by providing a mobile

internet connection, using token instead of passwords and NFC triggered application launch points

(like in Hong Kong public transport)

2. (In class) Performance: If the NEXT-TELL process consumes (subjectively) more time than the previous

pen-and paper process it will be abandoned. Every second counts in a classroom – and time spent

waiting for an application to resound counts twice: We address this by focusing on a lean, feature-

poor interface. We strip away everything not needed in the specific situation, creating small

applications tailored for the individual user and usage situation. We also optimize the data-structure

to be responsive in the analysis to provide useful in-class ad hoc visualizations.

3. Usability: If the users are not comfortable, believing upfront in their ability to succeed they probably

won’t touch it: We address this issue by providing an intuitive user interface that does not require any

upfront training. Everything more complex will be externalized, e.g. by importing / exporting data to

office tools which are “standard”.

4. Added value: As we are asking for a behavioral change we not only need to provide a vision for

improvement, but also a personal gain which is desired by the involved people: For learners this is

simply the interesting device itself – we are able to provide them with tablets through third party

funding. For teachers it is much more difficult, because the real individual desire does only surface

within the implementation process as we do not provide “off the shelf” products. In this specific case

it is the time saved and the real-time insight gained.



If these requirements are met, a large variety of use cases can be supported:

Use cases for all domains

Use case Description

1 A, B The teacher records his/her observations about student’s activities (A); the system provides testing facilities for the students and generates automated appraisal (B)

2 The teacher uses real time analytics for providing immediate feedback (or help) to the student

3 A, B, C Evidence for assessment is collected: either self-assessment (A) or peer-assessment (B) or artifacts (C)

4 A, B The system provides real time analytics for formative feedback (A) or generates recommendations for the student’s next steps (B)



For TESL some of these use cases are already realised:

Use cases for TESL

Use Case

1 A: Environments for students to practice: e.g. Moodle, GoogleDoc

B: for tests: e.g. Moodle

2 Environment for students to do collaborative assignments: e.g. Chatterdale quest in OpenSim, collaborative eWriting

3 A: interaction record is created: e.g. OpenSim logfile

B: formative feedback is given: e.g. by a native speaker in the Chatterdale quest

6.3 Integrating TESL with 21C learning: Facilitating team work

As with the other scenarios for Year 3, this one is also driven by the objective to relate NEXT-TELL tools and methods closer to today’s classrooms, which are not as technology-rich as often assumed. And even when they are technology-rich, the teaching/learning practices enacted often do not make full use of the resources available.

This line of activity will combine four goals: (a) to support the learning of speaking and writing in English as a second language; (b) to support ICT learning, in particular the use of spreadsheets; to support the acquisition of 21

st Century skills, in particular; (3) team work, and (4) self-guided learning.

6.3.1 The classroom practice

The classroom practice is based on having worked with three TESL teachers between December 2012 and March 2013 in the European School Bergen, The Netherlands. While in these classes the practice was conducted in a “low-tech” format (essentially, paper based), in a next step described here it will move to a realistic technology-enriched format, a format that then allows for NEXT-TELL methods to be applied more directly.



Students work over a couple of weeks on a larger project, creating together a digital artefact. The artefact is largely text-based (such as a newsletter), with some graphics elements. Google Docs is used as much as possible, so that students have to write in GDocs substantially.

The project is done in teams of 3-5 students. There is weekly group work scheduled (“meetings”), in the normal classroom, where these teams meet and discuss and move the project forward. At least one of these weekly meetings is prepared by a student, taking on the role of a “facilitator” (or process leader). That role changes each week, so that each student can make experiences in that role.

The task of the facilitator is to:

prepare the meeting (agenda planning, process planning)

conduct the meeting

document the meeting

reflect on the experience and on learning that has taken place.

In order to prepare the students, a 2-3 hour “course” needs to be provided by the project. It’s best to teach this course “blended”, with some life elements, but with all resources available for reviewing at the time when really needed to prepare a meeting.

About one week before the meeting, the facilitating student discusses the agenda with her teacher, and modifies the agenda and facilitation plan accordingly. A template for the meeting/agenda plan artefact is provided, which is identical with the meeting notes template. The students are also provided with some examples for conducting brainstorming and idea prioritisation activities, and know the website with further examples. A seed set of activities will be provided through the planner activities library.

The actual meeting is done around a table, with large sheets of paper (greater than A3) in the middle of the table. All participants have pens to write on this “group memory” artifact. The facilitator will usually have planned what kinds of notations will be used (e.g., tables, lists, concept maps).

The conversation is taped with a digital audio recorder. The facilitating student is supposed to use the audio recording to identify learning opportunities during reflection, and to provide evidence for mastering certain skills for the teacher. The teacher may want to listen to the tape for further insights into students’ English learning.

After the meeting, the facilitator writes meeting notes and distributes them with the team members. S/he also shares the notes with the teacher, accompanied by a reflection piece.

6.3.2 NEXT-TELL technology support

ECAAD Planner A “specialised” version that supports meeting plans and provides some feedback on meeting relevant parameters, such as overall duration or course, but also perhaps engagement level. The students are the “planners” here, hence it will be important to provide a suitable interface to the planner.

Task/Learning Environment

Most of the task work will be done on Google Docs, in particular with spreadsheets, documents, and the drawing tool. Template need to be provided for meeting agenda, meeting notes, and most importantly, for a number of group activities.

Tracing Trace and visualise all activities in GDoc. The core tools will be Google spreadsheets and documents.

Assessment, Appraisal

Manual appraisal of meeting artefacts and the recorded talk via rubrics (teacher, self, peer)

Some (aspects of) a meeting artefacts should be automatically scored, using ECAAD planner functionality for the planning artefacts, and Google Script for the meeting artefacts produced “at runtime”.



OLM Competence framework for meeting facilitation;

Support assessment from teachers, self-appraisal, peer appraisal.

Negotiation possible and practiced.

RGFA RGFA integrated into the facilitation toolbox as a method. Also try to integrate with the work that teachers set regarding specific subject matter content.



7 Conclusions

Our main conclusion is that in addition to attending to ‘individual’ usability (for teachers and students) and individual motivation and capacity (“Second Order Barriers”), ECAAD level tools need to be reactive to First Order Barriers (in particular computer:student ratio, connectivity issues) and to Third Order Barriers: classroom management practices building on the social relations between teacher and students, and students amongst each other. In concrete terms this means that V3 ECAAD level tools will be designed and implemented in order to make data entry practically possible in a variety of situations, including the “technology lean” classroom, and in a manner that takes into account challenges of classroom management.



8 References

[Baker, 2009} Baker, R. S., & Yacef, K. (2009). The State of Educational Data Mining in 2009 : A Review and Future Visions. (K. Y. E. I. Chief, R. S. J. D. B. A. Editor, T. B. A. Editor, & J. E. B. A. Editor, Eds.)Review Literature And Arts Of The Americas, 1(1), 3–17.

[Baker, 2012] Baker, R., Duval, E., et. Al. (2012). Educational data mining meets learning analytics. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12), Simon Buckingham Shum, Dragan Gasevic, and Rebecca Ferguson (Eds.). ACM, New York, NY, USA, 20-20.

[Béguin, 2003] Béguin, P. (2003). Design as a mutual learning process between users and designers. Interacting with Computers, 15(5), 709-730.

[Duvall, 2011] Duvall, E. (2011). Attention please!: learning analytics for visualization and recommendation. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge (LAK '11). ACM, New York, NY, USA, 9-17.

[Dyckhoff, 2012] Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and Implementation of a Learning Analytics Toolkit for Teachers. "Special issue articles." Journal of Educational Technology & Society 15.3 (2012) (P 58-67).

[Infante, 2010] Infante, C, & Nussbaum, Miguel. (2010). Third order barriers to the integration of technology in the classroom.

[Lockyer, 2011] Lockyer, L., & Dawson, S. (2011). Learning designs and learning analytics. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge (LAK '11). ACM, New York, NY, USA, 153-156. [Siemens, 2011]

[Siemens, 2011] Siemens, G., Gasevic, D., Haythornthwaite, C., Dawson, S., Buckingham Shum, S:, Ferguson,R., Duval, E., Verbert, K., Baker, R.S..J.d. (2011). Open Learning Analytics: an integrated & modularized platform: Proposal to design, implement and evaluate an open platform to integrate heterogeneous learning analytics techniques. Available online at http://solaresearch.org/OpenLearningAnalytics.pdf



9 Glossary

Terms used within the NEXT-TELL project, sorted alphabetically.

Partner Acronyms

JRS JOANNEUM RESEARCH Forschungsgesellschaft mbH, AT

UniRes UNI RESEARCH AS, NO

KMRC Medien in der Bildung Stiftung, DE

TUG Technische Universität Graz, AT

CBS Copenhagen Business School, DM

BHAM University of Birmingham, UK

IOE Institute of Education, University of London, UK

EXACT eXact Learning Solutions SPA, IT

TALK Verein offenes Lernen, AT

BOC-AT BOC Asset Management GmbH, AT

BOC-PL BOC Information Technologies Consulting SP.Z.O.O., PL

MTO MTO Psychologische Forschung und Beratung GmbH, DE

Abbreviations

BS Baseline Study

CbKST Competence-based Knowledge Space Theory Training Course

CBT Computer Based Training

DBR Design-Based Research

ECAAD Evidence Centered Activity and Appraisal Design (builds on the ECD)

ECD Evidence Centered assessment Design (e.g. PADI project)

EFL 'English as a Foreign Language'; EFL refers to learning English in a non-English-speaking region, such as studying English in an Asian or Latin American nation. Typically, EFL is learned as part of a student's school curriculum or for career purposes if working for an international corporation.

ENA Epistemic Network Analysis

ESL English as a Second Language; refers to learning English in the target language environment

HCI Human Computer Interaction

ICT Information and Communication Technology

IT Information Technology

LEPP Longitudinal Evaluation of Performance in Psychology (2nd generation e-portfolio)

NEXT-TELL Next Generation Teaching, Education and Learning for Life

OLM Open Learner Model

PADI The PADI project aims to provide a practical, theory-based approach to developing quality assessments of science inquiry by combining developments in cognitive psychology and research on science inquiry with advances in measurement theory and technology.

RA Requirement Analysis

RDS Researcher-led Design Study

SRI Stanford Research Institute



STEM The Science, Technology, Engineering, and Mathematics (STEM) fields are collectively considered core technological underpinnings of an advanced society, according to both the National Research Council and the National Science Foundation

TDS Teacher-led Design Study

TEL Technology Enhanced Learning

TESL Teaching English as Second Language

TISL Teachers Inquiry into Students Learning

NEXT-TELL partners responsible for generating tools and methods

BOC-AT ECAAD

BOC-PL SPICE

EXACT Moodle

JRS/ EXACT Google Docs and Google Spreadsheet

TALK OpenSim

CBS Rep5

JRS EVE

EXACT Mahara ePortfolio

BHAM OLM

Acknowledgement: The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 258114.