After completing this topic, you should be able to:•Explain the meaning of quality indicators•Describe the characteristics of quality indicators •Learn the importance of quality indicators in HEI•Understand the application of quality indicators in HEI

Quality Indicators

Indikator Kualiti dalam Pendidikan Tinggi(Quality Indicators in Higher Education)

• HEIs worldwide have undergone reform to improve quality •HEIs have implemented systematic and formalized quality assurance processes to achieve greater efficiency and accountability•Establishment of quality models and organizations to audit and review university performance•Institutional and national quality models and performance indicators are vital components to raise the standard of HEIs•Quantitative performance indicators are used to provide international comparisons

Quality Indicators

Performance cultures in higher education

• To ensure education provided by HEIs equips students for employment and provide the country with a highly skilled workforce that support economic growth.

•To contribute to educational, social, and political values.

Quality Indicators

Rationale for performance indicators

• To monitor own performance for comparative purposes.•To facilitate the assessment and evaluation of institutional operations.•To provide information for external quality assurance audits.•To provide information to the government for accountability and reporting purposes (Rowe, 2004).

Quality Indicators

Purposes of performance indicators in HEIs

• Ensure accountability for public funds•Improve the quality of higher education provision• Stimulate competition within and between institutions•Verify the quality of new institutions•Assign institutional status•Underwrite transfer of authority between the state and institutions•Facilitate international comparisons

Ref: Chalmers (2008)

Quality Indicators

The use of performance indicators in HEIs

• Simple indicators – expressed in absolute figures and are intended to provide an unbiased description of a situation or process.•Performance indicators – imply a point of reference; for example, a standard, objective, assessment, or comparator, are relative rather than absolute in character. Involve value judgements. •General indicators – externally driven and are not indicators in the strict sense; they are frequently opinions, survey findings or general statistics.

Ref: Chalmers (2008, quoted from Hanney, Henkel & Kogan, 1997)

Quality Indicators

Defining performance indicators

• Currently there is no common definition of performance indicators.•PI cannot be considered as facts, but are goal, value and context laden, and utilized in different ways depending on the performance model being used.•PI are defined as measures which give information and statistics context; permitting comparisons between fields, over time and with commonly accepted standards. They provide information about the degree to which teaching and learning quality objectives are being met within higher education sector and institutions. Ref: Chalmers (2008, p.3)

Quality Indicators

Defining performance indicators

• Input•Process•Output•Outcome

The types can be categorized as quantitative indicators and qualitative indicators.

Ref: Chalmers (2008, p.3)

Quality Indicators

Types of performance indicators

• define as those associated with the measurement of quantity or amount, and are expressed as numerical values. Input indicators• Human, financial and physical resources in supporting institutional programmes, activities and services.Output indicators• Output reflects the quantity of outcomes produced, including immediate measurable results, and direct consequences of activities implemented to produce results. Do not demonstrate quality of education, but quantities of outcomes.

Ref: Chalmers (2008)

Quality Indicators

Quantitative Indicators

• associated with observation based descriptions, rather than an exact numerical measurement or value. Relate to or involve comparisons based on qualities of non-numerical data such as policies and processes for assessing students’ learning, the experience, the content of a mission statement.

Outcome Indicators• Focus on the quality of educational program, activity and service benefits for all stakeholders.•Insightful, meaningful and accurate since they are related to the objectives of higher education.•Students are treated as customers. Ref: Chalmers (2008, p.5)

Quality Indicators

Qualitative Indicators

Process Indicators• include the means used to deliver educational programmes, activities and services within the institutional environment.• qualitative information on teaching and learning such as policies and practices.•Considered as most practical, useful and appropriate measures of quality teaching and learning. Ref: Chalmers (2008)

Quality Indicators

Qualitative Indicators

Look at 13 process indicators • Mission, Vision and Objectives•Teaching and Learning Plans and Policies•Teaching and Learning Indicators•Internal and External Performance Funds for Teaching and Learning•Organizational Unit Review•Curriculum Review•Assessment and Feedback Policies•Graduate Attribute Statement•Student experience Ref: Chalmers (2008)

Quality Indicators

Research conducted in Australia

Look at 13 process indicators • Professional Development•Appointment and Promotion Criteria•Review of Academic Staff – performance•Recognition of Excellence in Teaching and Enhancing Student Learning ExperienceRef: Chalmers (2008)

Quality Indicators

Research conducted in Australia

Look at four dimensions of teaching practice • Institutional climate and systems – commitment to the enhancement, transformation and innovation of learning. Measure student experience and level of satisfaction.• Diversity – relates to ethnic, cultural and socioeconomic diversity as well as students’ abilities, talents and learning approaches.

Ref: Chalmers (2008)

Quality Indicators

Research in Australia – Quality Teaching

Look at four dimensions of teaching practice •Assessment – the assessment tasks of student enrolled in the program of study. Look at the design, delivery and administration, provision of feedback, moderation, and review of assessment.• Engagement and learning community – student engagement, i.e. the student commitment and engagement with their own education. Also includes staff engagement.Ref: Chalmers (2008)

Quality Indicators

Research in Australia – Quality Teaching

• Peranan universiti dan ahli akademik (University’s and academics’ roles)• Perkembangan kurikulum mengikut keperluan pasaran (Curruculum development according to market needs)•Penyelidikan, pembangunan dan pengkormesialan dalam sistem inovasi kebangsaan (Research, development and commercialization in the national innovation system)•Kaedah pengajaran dan pembelajaran (Teaching and Learning Methods)•Perluasan akses dan mobiliti pengetahuan (Accessibility and knowledge mobility)

Quality Indicators

Higher Education in Malaysia– Challenges

Pekeliling Kemajuan Pentadbiran Awam Bilangan 2 Tahun 2005

Garis Panduan Bagi Mewujudkan Petunjuk-petunjuk Prestasi Utama Atau Key Performance Indicators (Kpi) Dan Melaksanakan Pengukuran Prestasi Di Agensi Kerajaan

Terminologi

Petunjuk Prestasi Utama (KPI) ialah salah satu kaedah bagi mengukur prestasi perkhidmatan agensi-agensi Kerajaan

Perkhidmatan Teras Bidang tanggungjawab agensi sejajar dengan visi (core business): dan misi agensi;

Proses Utama Fungsi-fungsi di bawah perkhidmatan teras yang (core process): perlu dilaksanakan bagi menghasilkan perkhidmatan untuk pelanggan;

Key Performance Petunjuk-petunjuk prestasi utama yang ditentukan Indicators (KPI): sebagai asas mengukur prestasi;

NKRA

Education

Crime (Public Safety)

Corruption

Low Income Households

Rural Basic Infrastructure

Urban Public Transportation

• Globalisasi dan piawaian melalui pemeringkatan dan penarafan (Globalization and standardization through development and accreditation)•Peluang guna tenaga (Resource utilization opportunities)

Quality Indicators

Higher Education in Malaysia– Challenges

• 20 public universities•21 polytechnics•37 community colleges• >400 registered private colleges• 21 private universities and university colleges• 11 local university branch campuses + 5 foreign university branch campus

Areas for indicators:• Academic staff• Educational programs• Student selectivity• Educational resources• Governance

The method used will be peer review.

Quality Indicators

Malaysia - Current scenario

Different ranking approaches:• League table – each university is assigned a specific rank. Higher ranks indicate higher quality, lower ranks indicate lower quality.•Quality criteria and indicators are used in this ranking methodology to assess universities.•Each indicator such as research impact as the number of citations per faculty in the Thompson Scientific Database or teaching quality as in THES are given weight.•This approach are applied to all universities assessed.

Quality Indicators

How are universities ranked?

Different ranking approaches:•A ranking of individual disciplines or departments instead of whole institutions. •A multidimensional concept of university quality instead of a ‘one-size-fits-all’ approach, taking into account the diversity of academic institutions, missions and goals as well as language and cultural specifics.•A separate measurement and presentation of single indicators.•A presentation of ranking results in rank groups (top, middle, bottom groups) instead of league tables.

Quality Indicators

How are universities ranked?

Ranking serves several purposes:•Responds to demands from consumers for easily interpretable information on the standing of HEIs. •Stimulates competition among universities.•Provides some rationale for allocation of funds.•Helps to differentiate among different types of institutions and different programs and disciplines.•Contributes to the definition of ‘quality’ of HEIs within a particular country.

(source: Berlin Principles of Ranking of Higher Education Institutions, 2006

http://www.che.de/downloads/Berlin_Principles_IREG_534.pdf)

Quality Indicators

Purposes of ranking of HEI

Purposes and goals of rankings:•Be one of a number of diverse approaches to the assessment of higher education inputs, processes, and outputs.

•Be clear about their purpose and their target groups.

•Recognize the diversity of institutions and take the different missions and goals of institutions into account.

•Provide clarity about the range of information sources for ranking and the messages each source generates.

•Specify the linguistic, cultural, economic, and historical contexts of the educational systems being ranked – should be aware of possible biases.

(source: Berlin Principles of Ranking of Higher Education Institutions, 2006)

Quality Indicators

Berlin Principles of Ranking of Higher Education Institutions

Quality Indicators

Berlin Principles of Ranking of Higher Education Institutions

Design and weighting indicators:•Be transparent regarding the methodology used for creating the rankings.•Choose indicators according to their relevance and validity.•Measure outcomes in preference to inputs whenever possible.•Make the weights assigned to different indicators (if used) prominent and limit changes to them.

Quality Indicators

Berlin Principles of Ranking of Higher Education Institutions

Collection and processing of data:•Pay due attention to ethical standards and the good practice recommendations articulated in these Principles.•Use audited and verifiable data whenever possible.•Include data that are collected with proper procedures for scientific data collection.•Apply measures of quality assurance to ranking processes themselves.•Apply organizational measures that enhance the credibility of rankings.

Quality Indicators

Berlin Principles of Ranking of Higher Education Institutions

Presentation of ranking results:•Provide consumers with a clear understanding of all of the factors used to develop a ranking, and offer them a choice in how rankings are displayed.

•Be compiled in a way that eliminates or reduces errors in original data, and be organized and published in a way that errors and faults can be corrected.

Quality Indicators

Purposes of Quality Indicator System

Colorado State, USA:•Encouraging continuous improvement by institutions in achieving high levels of performance.•Measuring institutional performance and accountability.•Determining funding recommendations and the funding distribution for the higher education system.•Build public support for increased funding for higher education.

Quality Indicators

Some Quality Indicators

•Baccalaureate graduation rates•Achievement scores of graduating students on various comprehensive examinations, tests, and /or professional specific licensure or certification examinations•Graduates employed or continuing their education•Institutional support expenditures – administration expenditure, expenditures per student•Undergraduate class size•Faculty teaching workload

• Categories of institutions

• APEX university• Research intensive•General

•The structure of Malaysian Qualifications Agency (MQA)• One-stop center for institutions for registration and accreditation of courses

Quality Indicators

Current scenario

Accountability and Quality

The concept of accountability and quality assessment in higher education is an international phenomenon

In America, many regions are moving toward ‘performance incentive funding’.

In Europe and Australia, the central government is directly involved in establishing quality mechanisms through:

Quality control,

Quality audit

Quality assessment

The agencies involved are like the Higher Education Quality Council and the Higher Education Statistics Agency (HESA)

Quality Indicators

The focus of attention in performance indicators in the U.S. has been cost efficiency, scientific and technical education, management of intellectual property produced at universities, and undergraduate education.

Less attention is paid to graduate education and research.

Categories of costs for higher education:Education and general: faculty and staff salaries, equipment, libraries, administrative and basic academic computing, and certain capital or such as rent.Cost for sponsored researchCosts of student living: room, board, clothing, laundry, entertainment, and etc.Cost of foregone earnings: While disengaged from the productive work force.

Quality Indicators

Accountability and Quality

The Movement in Setting PerformanceIndicators in Higher Education(U.S.A.)

1980sEra concerned with growth in enrollments and access was overEmerging issues include:Public accountabilityQualityProductivityUndergraduate educationIn 1986, all 50 states and the District of Columbia had developed initiatives to improve the undergraduate education

Shift from growth funding (formula funding) toward funding outcomes, results, and performance

These efforts paralleled developments in Europe and Australia

1990s

The development of performance indicators differs from that in 1980s

From voluntary institutional improvement to a system of mandated public accountability

By 1994, 18 states had developed indicator systems

Greater centralization of authority

Issue raised:

Will the federal government assume greater centralized control of higher education through areas such as accreditation and financial aids by using a set of national goals and performance standards?

The Future of Higher EducationThe White Paper 2003

Higher education must expand to meet rising skill needs

The social gap among those entering university remains too wide

Many of our economic competitors invest more in higher education

Universities are struggling to employ the best academics

Funding per student fell 36% between 1987 and 1997

Universities need stronger links with business and industry

Reports of Institutional Effectiveness(EOIE) Virginia’s Public Institutions of Higher Learning

Annual report to provide meaningful information on the academic quality and operational efficiency of Virginia’s public institutions

To provide evidence of institutional effectiveness – the extent to which institutions accomplish their missions and students achieve their educational goals.

Structure of the Reports(Five Points)

Institution’s missionThe mission statement sets a vision for the institution and defines how it will get there.Institutional profile In-depth views of enrollment and projections of future enrollment.System-wide measures Include 14 performance measures focused on operational efficiency and factors associated with academic quality:Example: Classroom and laboratory space utilization, percentage of professionally accredited programs and etc.

Institution-specific measures

Represent unique aspects of the mission that the college or university chose to highlight

Core competency reports

Explore student general education assessments in written communication and technology/information literacy.

Performance Indicators of California Higher Education, 2001

Describes the scope of the current set of indicators reported by the Commission, and highlights recent trends based on current information related to these indicators.

This report are divided into five categories:

Population Context,

Fiscal Context,

Student Preparation,

Student Access, and

Student Outcomes

Key Performance Indicators (KPIs)

Key Performance Indicators are quantifiable measurements, agreed to beforehand, that reflect the Critical Success Factors of an organization.

They defer depending on organization

KPIs must:

Reflect the organizational goals

Be key to its success

quantifiable

Curtin’s efficiency and effectiveness Key Performance Indicators (KPIs)

Teaching and learningEffectiveness indicatorsQuality of graduatesQuality of teachingStudent progress and achievementInputEfficiency indicatorsTeaching and learning expenditureStudent progress and achievement

EFFICIENCY & EFFECTIVENESS

EFFICIENCY means: saving TIME, MONEY or EFFORT

Efficiency measures the resources used to attain a certain level of output

EFFECTIVENESS means how well the the job gets done. i.e. the quality of the output.

Effectiveness measures the extent to which outcomes have been achieved

The End of Quality6th Quality in Higher Education International Seminar (Birmingham, UK) May. 2001

Three Major themes:

1. Has external quality review has its day?

2. Has control of quality been usurped by the market and by information technology?

3. Does the development of mass education necessarily mean the end of quality?

Transforming Quality7th Quality in Higher Education InternationalSeminar (Melbourne) Oct. 2002

Three main themes:

1. To reconceptualise how higher education engages with access, employability and funding issues

2. What constitutes a high quality learning process and outcomes

3. How might quality evaluation be transformed to help improve the quality of the experience and of the learning?

8th Quality in Higher Education InternationalSeminar (Sheffield) May. 2003

Two major themes:

1. How does student feedback inform quality?To what extent do institutions need to adopt new procedures to make student feedback effective?

2. What does the White Paper encourage a closer link between quality and learning?

The First Session of the Regional Follow-up Committee for the World Conference on Higher Education (WCHE), 2- 3 November 2000, Kuala Lumpur Malaysia

Recommendations to Member Countries:

1. Need for ongoing efforts to broaden access taking into account the disadvantaged groups (women and ethnic minorities)

2. Provide increased support for staff development and research

3. More participation of women in higher education particularly in decision making level

Indicators of Research Quality in Higher Education

The vast majority of discoveries have been made in higher education environment (Dill, 1986).

Review of literature on research productivity highlighted several indicators which include:

Productivity dollars

Productivity publications

Peer evaluation

Productivity dollarsThe number of dollars generated by research was the most often cited measurement of successThose universities that are ranked higher, their faculty have are adept at obtaining research grants

Productivity publicationsThe number of publications is frequently used as an indicator of quality in researchThe research that is published is taken as an indication of its quality

The types of publication which determine its quality:Journal articles, monographs, chapters, booksQuality: reputation of publication in discipline, distribution of publication, refereed vs. non-refereed journals, invited chapters/papers

Output versus Impact factor of publications

Output refers to how prolific the the research is producing acceptable articles/books

Impact was determined by checking citations of the articles over a period of years

OUTPUT & IMPACT FACTOR

Peer Evaluation

Assemble a group of peers to review the research efforts and make a determination of the quality of those efforts

The concerns of such approach include:

The visiting group doesn’t fully understand the work of the unit/individual being reviewed (especially when it is multidisciplinary)

PEER EVALUATION

Curtin’s Efficiency and Effectiveness Key Performance Indicators (KPIs)

Research and Development

Effectiveness indicators

Research Performance Index

Research Quantum

Comparison between Curtin and all Australian Universities

Research Funding

Research Publications

Efficiency Indicators:

Research Performance Index

Research Funding

Research Publications

Effectiveness measures the extent to which outcomes have been achieved

Efficiency measures the resources used to attain a certain level of output

Efficiency and Effectiveness Key Performance Indicators (KPIs)

Publications and Number of Ph.D. Graduates

2001 762002 892003 112

Publications2000 1,864 (319 international journals, 212 in local journals)2001 1,815 (303 international journals, 204 in local journals)2002 2,507 (496 international journals, 328 local

journals)Malaysia was ranked #63 by MASTIC (Pusat Informasi Sains dan Teknologi Malaysia) in terms of production referred journals

Critical Success factors (Research)

Wide academic base and facilities

Graduate students

Screening of IRPA application and monitoring

Evaluation of research and innovation

Research culture and administrative support

Strong research networking

Research cluster development

Incentives & in-house competition as incubator

Steps to be Taken by Institutions toPromote Excellence in Teaching

Define what they mean by excellent in teaching

Having well-defined criteria about excellent teaching and standards for weighting and rating of teaching/research/service

Weigh teaching more heavily

Increase sophistication of teachers

Promote excellent teaching, not just excellent teachers

Not treat promotion as a separate issue(Gibbs, 1995)

Theoretical Definition for Excellent Teachers

The character of the professor

Values, personality, and social intelligence

The knowledge of the professor

Disciplinary and pedagogical understanding

The actions of the professor

Problem-solving behaviors

The responses of the students

Learning outcomes

Quality Teaching in Higher Education

Flexibility in approaches to teaching and learning (including assessment)

Good organization of subject matter and course, including relevance and coherence of content and planned teaching/learning activities

Effective communication

Knowledge and enthusiasm for subject matter and teaching

Facilitation of learning through student interaction and active experience

Respect for and positive attitude toward students

Critically reflective orientation to teaching including effective use of feedback to guide learning and improve teaching

Appropriateness and fairness in assessment and grading

Reeders, E, & Marshall, H. 1996

GOOD TEACHING: THE TOP TEN REQUIREMENTS

By Richard Leblanc, York University, Ontario , 1998.

1. Good teaching is as much about passion as it is about reason.

2. Good teaching is about substance and treating students as consumers of knowledge.

3. Good teaching is about listening, questioning, being responsive, and remembering that each student and class is different.

4. Good teaching is about not always having a fixed agenda and being rigid, but being flexible, fluid, experimenting, and having the confidence to react and adjust to changing circumstances

6. Good teaching is also about styleThis is very important -- good teaching is about humor.

7. Good teaching is about caring, nurturing, and developing minds and talents.

8. Good teaching is supported by strong and visionary leadership, and very tangible institutional support -- resources, personnel, and funds.

9. Good teaching is about mentoring between senior and junior faculty, teamwork, and being recognized and promoted by one's peers.

10. At the end of the day, good teaching is about having fun.

Quality in College Teaching:A Research Approach

Flexibility in approaches to teaching and learning (including assessment)

Good organization of subject matter and course, including relevance and coherence of content and planned teaching/learning activities

Effective communication

Knowledge and enthusiasm for subject matter and teaching

Facilitation of learning through student interaction and active experience

Respect for and positive attitude toward students

Critically reflective orientation to teaching including effective use of feedback to guide learning and improve teaching

Appropriateness and fairness in assessment and grading