http://calibrate.eun.org wp4: validation andrea kárpáti, elu...

htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rg

WP4: Validation

Andrea Kárpáti, ELU ([email protected])In cooperation with Roger Blaimire, EUN

([email protected])

mailto:[email protected])



mailto:[email protected]

CALIBRATE Review 19 December 2006 2

htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rgValidation framework

Levels of validation– Systems

– Institutional - school level – Pedagogical - teacher/student level, educational usability– Technical - performance of project deliverables versus school

insfastructure: technical usability– Economic – sustainability, dissemination– Cultural/Linguistic – how content, technology, inherent teaching

style style etc. Travel

Methods of validation– Usability tests: National (pre)pilots, (inter) national workshops, (inter)

national discipline based collaborative groups

– Interviews: semi-structured individual, focus groups

– (Pre)pilots, school based usability tests

– Workshops: (inter)national


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rgInstitutional / school level

Assessment criteria e-maturity change in teaching

and in collaboration dissemination involvement of wider

community change in school ICT

culture communication and management)

Assessment instruments

ICT Competence Inventory (ICT/CI)

Educational Strategies Questionnaire (ESQ)

Focus group interviews (teachers, heads, students of same school)


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rgPedagogical - teacher/student level Authoring, adapting,

modifying, sharing of LO Implementation in classroom

teaching Impact on practice e.g. the

role of teachers, teaching strategies, teaching styles

Discipline-related experiences – Math, Science, Environmental Science, English as Foreign Language.

Lesson plans – online template (lesson type, content, ICT and traditional tools used, pedagogical strategy, evaluation methods etc.

Blog – online diary on working with Calibrate tools and their utilization: experiences, problems, desires for further development

National and international fore for discipline-based teacher groups

Focus group interviews (teachers of the same discipline from different schools)


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rgTechnical - performance of Calibrate

tools and the LRE

How deliverables work in practice

Implications for ICT resourcing in the school Localization issues 1: technical aspects

Technical support Integration with existing

ICT technology Adaptability to school

needs

Economic Cost for schools:

integration with existing environment, training, technical support

(Pre)pilots Blogs International forum for

piloting teachers SUSTAINABILITY STUDY I:

– school visits with interviews one year after project ends

– ICT infrastructure survey at the beginning of the project and one year after finish

– Interview with school leadership


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rgCultural/Linguistic

Localization issues 2: cultural aspects

Support for multiple languages

Intercultural (what aspects of national educational culture does the project support and what not?)

Trans-national collaboration

Culture of education (attitude to innovation, ethos of profession, private and public sector)

Attitude of stakeholders (students, researchers, support staff)

International workshop for teachers with focus group interviews and collaborative LO design experiments

Interviews with national educational policy makers experts for (ICT policy, European harmonization of curricula,

SUSTAINABILITY STUDY II:– Attitude surveys

(participating teachers, heads and peers at piloting schools, other stakeholders)

– On-site observations – peer review visits


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rgTimeline for Validation 1

January 2006: disciplinary areas for validations selected, curriculum analyses

January – March 2006: selection of piloting school / teachers

May 2006. 1. pre-pilot May-September 2006: elaboration / discussion of

validation framework September – November 2006: teacher training

where needed, pilot of validation instruments October – December 2006: elaboration of training

material for validating teachers December 2006: 2. pre-pilot, finalisation of validation

plan and instruments


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rgTimeline for Validation 2

January 2007: teacher training courses end with module on CMS, with special focus on LRE and Toolbox

February – May 2007: Validation Round 1: school usability studies followed up through lesson plans, interviews and site visits

June-July 2007: summary of VR1, further development / eventual improvement of tools, planning for VR2

September – December 2007: VR2 January – February 2008: summary of VR2,

finalisation of Calibrate tools, suggestions for dissemination


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rgPrepilot 1: First impressions

Aim: test basic functionalities of LRE Methods: guided walkthrough, structured

questionnaire for individual walkthrough followed by plenary discussion – all video/audio-taped

Participants: 2 teachers / country from 6 countries, LRE and Toolbox designers, WP4 coordinators

Results: design and help improved, relevance of functions for teachers clarified, further functions planned – co-evolutionary design


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rgPrepilot 2: User test

Experienced users (teachers of the 1th pilot) and newcomers to Calibrate tools piloted, 2x2 teachers / 10 piloting countries

Self-report questionnaire filled out while performing tasks in the LRE

Topics:– Ease of use (navigation, performance of tasks)– Visual aspects (pleasingness, communication)– Pedagogical aspects (relevance of tools / content)– Suggestions for tool design– General attitudes / feelings / motivational level changes

(moods) while using tools Summary by ELU will compare „expert” and novice”

users’ attitudes, usage patterns, levels of involvement, critical observations and satisfaction


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rgTeacher Training for Calibrate

Methods and content varies acc. to participating country / teachers selected. Common core: using learning content management systems incl. LRE and Toolbox – a unit developed by ELU

Phase 1, preparation for pilot: collaborative learning in school-based, national groups, theme: ICT in education

Phase 2, pilot: knowledge building community of teachers – discipline-based work in (inter)national groups

Course types:– accredited national training courses– „crash course” on LRE and Toolbox– EPICT – the European Pedagogical ICT Licence


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rg

Educational Strategies Questionnaire

F1: Aims and objectives of school education – present and future– Values, attitudes and beliefs about teaching and

education– The Role of school

F2: ICT at school: function and mission– Existing good practice / problems– ICT development plan

F3. The role / task / mission of the teacherF4 Own educational practice

– Teaching environment / infrastructure – Paradigms, strategies, methods– Preparation for teaching

F5. Opinion / plans / wishes about in-service trainingF6. Innovation experiencesF7. Personal efficiency / resilience / coping with change


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rg

ICT Competency Inventory (ICT-CI)

A self report questionnaire piloted in a project measuring ICT readiness in about 150 schools of a district in Budapest. The ICT-SI is an index based measuring tool. (The concept of indexing is used for the data reduction method employed: several variables are composed in one index.) Our test comprises the following indices:

Index (1) – School access to ICT Index (2) – Home access to ICT Index (3) – ICT related attitudes Index (4) – ICT related competencies Index (5) – Educational use of ICT


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rg

Four domains for every index: „low” (value range: 0,01–1), „medium” (value range 1,01–2) „high” (value range 2,01–3) 0 value, indicating the non-existence of any interpretable data

for the index. (E. g., low level ICT access means sporadic, rare access while 0 level indicates no access to a PC).

Items correlated with the presence / absence of certain personality traits measured by the California Personality Inventory (CPI) to identify

teacher profiles that tend to success or fail with ICT Individual training / coaching needs

htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rgICT Profile - Teacher 32

1,069

2,333

1,702

0,966

0,500

1. Iskolai IKT

2. Otthoni IKT

3. IKT att.4. IKT komp.

5. IKT okt.


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rg

Personality tests for better validation results?

Coaching -. A generally accepted performance enhancement method in industry and commerce, very rare in education

EPICT (European Pedagogical ICT Licence), www.epict.org – nationalization and adaptation: usability at stake

CPI was used to define basic personality traits relevant for the world of work (sociability, innovativeness, coopeartion, assertiveness etc.)

CPI results were correlated with ICT Competence Inventory and Education Strategies Questionnaires

LEARNER PROFILES defined – facilitators could improve mentoring and customize assessment

BFQ to be used for Calibrate in Hungary and offered for other participants for consideration

http://www.epict.org/

EPICT pilot in Hungary: Strong correlation between personality traits and

ICT skills, attitudes and competence

1. School ICT access

2. Home ICT access

3. ICT attitudes

4. ICT compe- tences

5. Educational use of ICT

Sociability 0,154 -0,104 0,153 0,222* 0,123

Social skills 0,120 0,090 0,139 0,275** 0,015

Self respect 0,059 0,039 0,055 0,094 0,034

Gen. wellbeing (Wb)

0,219* -0,057 0,218* 0,256** -0,074

Self control 0,133 -0,031 0,222* 0,120 -0,064

Tolerance 0,178 0,000 0,354** 0,351** 0,007

Establ. good relationships

0,179 -0,030 0,233* 0,262** -0,149

Building a community

-0,093 0,200* -0,118 -0,139 0,230*


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rg Intellectual efficiency (Ie)

0,183 0,016 0,238*

0,332** 0,002

Flexibility (Fl)

0,163 0,014 0,162 0,306** 0,126

Feminine qualities (Fm

-0,067 0,076 -0,020

-0,103 0,114

Extraversion (Ex)

0,151 -0,033 0,162 0,271** 0,052

Reliability (Re)

0,147 0,053 0,134 0,228* 0,055

Emotional stability (Es)

0,114 -0,030 0,115 0,226* 0,046

Intellect (In) 0,150 -0,016 0,142 0,208* 0,029

1. School ICT access

2. Home ICT access

3. ICT attitudes

4. ICT com- petences

EducationalUse of ICT

Pearson two-tailed correlation, significance levels: *: 0.05, **: 0,01


htt

p:/

/calib

rate

.eu

n.o

rgh

ttp

://c

alib

rate

.eu

n.o

rg

Correlation analysis of CPI and ICI findings, questionnaires and semi-structured interviews, show personality features that are associated with success and failure to acquire ICT skills on a level sufficient for educational use

Some of these variables may be affected by specially targeted mentoring methods, teaching environment course content (best practice examples) custom-made skills training.

Based on EPICT adaptation results in Hungary, in-service training of teachers may be improved to suit not only the initial level of ICT skills but also the mindset and personality of teachers.

http://calibrate.eun.org wp4: validation andrea kárpáti, elu...

Documents