maurice grinberg evaluation.nbu.bg alte meeting, lisbon, november 12-15
TRANSCRIPT
Maurice Grinbergwww.nbu.bg
evaluation.nbu.bg
ALTE meeting, Lisbon, November 12-15
Modern private university Year 1991: 4 000 students in 30
programs Year 2008: 15 000 students in more than
100 programs Bachelor, Masters, and PhD programs Distance learning For the first time in Bulgaria:
General Education Test as entrance exam four times a year
Assessment CenterALTE meeting, Lisbon, November 12-15
Since 1995: General Education Test as admission exam (more than 10 000 candidates per year)
Since 1998 for all NBU students:• L2 exams at the end of a course of study,
levels from A2 to B2 • 6 languages: English, German, Italian, Russian,
Spanish, and French • L1 exam for writing skills in Bulgarian language• Computer skills exam
Since 2002: placement testsALTE meeting, Lisbon, November 12-15
Staff: Experts in testing, psychometrics, cognitive science, languages
Advisory board:• Department of Psychology and Cognitive
Science, and department of Foreign Languages• NBU QA department• External consultant:
prof. D. Dimitrov, George Mason University Projects and research
• Leonardo da Vinci program (WELKOM)• Erasmus program (NETTLE)• Assessment in e-learning • Cognitive assessment, small group assessment
ALTE meeting, Lisbon, November 12-15
Monitoring and development of all examination procedures in NBU
Development of exams for NBU students Analysis and interpretation of test
results: test score reports to test takers and university administration
Item-banking and storage of data Training for item-writers and
standardized workshops for examinersALTE meeting, Lisbon, November 12-15
Fairness and equity• Standardized administration• Monitoring during examination• Special accommodation for candidates with
disabilities• Information about the content and format of
the examination: question types, rubrics and appropriate test-taking strategies, test samples
ALTE meeting, Lisbon, November 12-15
B2 is a required level in NBU for all students
Goals: use the language to express and understand opinions, to build up and understand arguments, to communicate with students from other countries in English, Spanish, Italian, French, German or Russian.
Certification decisions and graduation
ALTE meeting, Lisbon, November 12-15
Test takers: Bulgarian native speakers with different topical knowledge and second language experience
Tasks: both from the instructional and real-life domain
Scores are reported to candidates in grades
Tested abilities: speaking, listening, reading, and writing
ALTE meeting, Lisbon, November 12-15
Psychometrics: • IRT used for evaluation of test item
properties • Many-facet Rasch analysis in assessing essay
writing task (facets: topic, raters, and assessment criteria)
• Estimation of candidates’ abilities, raters’ severity, relative difficulty of tasks, and relative difficulty of assessment criteria
• Bias/interaction analysisALTE meeting, Lisbon, November 12-15
Meet all ALTE standards for quality in language tests: • Revise existing foreign language exams• Achieve a consensus about test qualities• Develop a new test for Bulgarian as a
foreign language
Exchange of best practices: • Investigation and assurance of test
reliability• Improvement of validation argumentation• Elaboration of procedures to involve all
stakeholders in the examination processesALTE meeting, Lisbon, November 12-15
Development of tests for Bulgarian as a foreign language• Situational analysis: need for a A1 test for
foreign university students in Europe and Asia and related textbooks
• Planning phase: users and test takers identified, resource balance performed
ALTE meeting, Lisbon, November 12-15
Development of tests for Bulgarian as a foreign language (cont.)• Design phase: theoretical background of the
test• Approaches considered:
Bachman model of CLA will underlie each task as a needed executive resource
CEFR for the domain, communicative goal and language function
Weir’s Frameworks for Developing and Validating tests
ALTE meeting, Lisbon, November 12-15
NBU: http://www.nbu.bg Assessment center:
http://evaluation.nbu.bg(in construction!)
ALTE meeting, Lisbon, November 12-15
About Impact: Our strive to achieve positive impact involves
Providing public information and advisory services for students
Seminars to familiarize university teachers with our research on test qualities
Training for item writers and examinersReports to the University BoardCooperative work with the Department of
Foreign Languages to improve continuous assessment in language classes
Problems: we have not investigated the educational impact of our examinations
Decisions: start collecting evidence about “face validity”
About Practicality: The Assessment Center develops and implements the appropriate procedures for managing all aspects of the examination process.
Problems: need to increase research studies in all aspects of test production
Decisions: identification of suitable experts
ALTE meeting, Lisbon, November 12-15