australasian journal of economics education 13, number... · 1. report research on the teaching of...
TRANSCRIPT
Australasian Journal of Economics Education
Volume 13 Number 1 March 2016
ARTICLES
A Report on the Development of Ross Guest
Learning Standards for Economics Allan P Layton
in Australia
The Impact of Lecture Capture on Chris Jones
Student Performance Matthew Olczak
Selecting Sessional Tutors: A New Carl W. Sherwood
and Effective Process Bruce Littleboy
BOOK REVIEW
Teaching Innovations in Economics Peter Docherty
M. K. Salemi & W. B. Walstad
ISSN 1448-448 X
Editorial Executive Co-Editor Professor Rod O’Donnell Telephone: (+61 2) 9514 7738 Email: [email protected] Co-Editor Dr Peter Docherty Telephone: (+61 2) 9514 7780 Email: [email protected] Co-Editor Mr Joseph Macri Telephone: (+61 2) 9850 6069 Email: [email protected]
Editorial Board Professor William J. Baumol, New York University, USA. Professor Harry Bloch, Curtin University of Technology, Australia. Professor Bruce Chapman, Australian National University. Professor Kenneth Clements, University of Western Australia. Professor David Colander, Middlebury College, Vermont, USA. Professor John Foster, University of Queensland, Australia. Professor Andrew Hannan, University of Plymouth, UK. Professor Yujiro Hayami, Foundation for Advanced Studies in
International Development, Japan. Professor Tim Hazledine, University of Auckland, New Zealand. Professor K.L. Krishna, Delhi School of Economics, India. Professor Alan Luke, Nanyang Technological University, Singapore. Professor Rod O’Donnell, Macquarie University, Australia. Professor David Round, University of South Australia. Professor Daniel Rubinfeld, University of California, Berkeley, USA. Professor Warren Samuels, Michigan State University, USA. Professor Amartya Sen, Harvard University, USA. Professor John Siegfried, Vanderbilt University, USA. Professor Jim Taylor, University of Lancaster, UK.
Secondary School Teaching Representatives Mr Doug Cave, Queensland Economics Teachers Association. Mr Ian Searle, Brisbane Boys Grammar.
AUSTRALASIAN JOURNAL OF ECONOMICS EDUCATION
MISSION STATEMENT The Australasian Journal of Economics Education is a peer-reviewed journal that publishes papers on all aspects of economics education. With a view to fostering scholarship in the teaching and learning of economics, it provides a forum for publishing high quality papers and seeks to bring the results to a widening audience. Given both the increasing diversity of the student clientele, and increasing calls for greater attention to the quality of tertiary teaching, this Journal seeks to foster debate on such issues as teaching techniques, innovations in the teaching of economics, student responses to such teaching, and the incentive systems which influence the academic teaching environment. The AJEE is interested in research involving both quantitative and qualitative analyses and also in interpretative analyses based on case studies. While the Journal is Australasian-focussed, it encourages contributions from other countries in order to promote an international perspective on the issues that confront the economics discipline. AJEE aspires to: 1. Report research on the teaching of economics, and cultivate heightened interest in the teaching of economics and the scholarship of teaching. Pedagogical issues will be a central feature, and will encompass work on the teaching of economics in diverse contexts, including large and small classes, undergraduate and postgraduate classes, distance learning, issues confronting foreign students on-shore and off-shore, and issues related to the teaching of fee-paying MBA and other post-graduate groups from diverse disciplinary backgrounds. Though economics is the prime focus, consideration will also be given to work on other subjects that have a demonstrated relevance for the teaching of economics. Such issues will also involve evolutionary issues in the teaching of economics, in terms both of effective ways to teach evolving theory and of evolving technology with which to teach that theory (including on-line teaching). Recognition will be given to the fact that economics as a discipline has not fared well in CEQ results (course experience questionnaire
results) since the reporting of those results began in Australia. Nor has economics teaching typically been well received in the USA or UK, according to survey evidence. In that context the relevance to teaching of changing administrative arrangements in universities will also be highlighted (eg in terms of contemporary quality assurance procedures and other government policy changes in Australia and New Zealand). 2. Report research on the nexus between teaching and research (including research on the diverse, changing and potentially conflicting incentives within the academic industry). Papers exploring the extent to which research and teaching activities are complementary or competitive will be welcomed. 3. Recognise the relevance of some more deep-seated implicit assumptions and issues of economic philosophy embedded in what is commonly taught, (as in Sen’s work on economics and ethics, for example). Inter alia, the question arises as to the way in which students respond to economics taught as a path to scientific certainty, as against economics taught as reflecting unsettled debate and vigorous controversy. 4. Recognise the place of history in the teaching of economics. Both HET and economic history tend to play a diminishing role in professional economics training, as emphasis on technique dominates. This a-historical approach to the teaching of economics has been criticised by many influential economists (including Joan Robinson, Leontief, Myrdal, Colander, and Robert Clower in his acerbic remarks about the value of much that is published in such prestigious journals as the AER). This line of criticism has been continued in the recent growth of heterodox economics associations in a number of countries (including one for Australia and New Zealand) and on the web through the Post Autistic Economics (PAE) newsletter. Historical and institutional factors will thus provide one focal interest. 5. Recognise interdisciplinary issues important to the presentation of economics in various contexts. On the one hand, economics students are not systematically exposed to the insights of other social sciences and the conformity or otherwise of their conclusions with those of economics. On the other hand, other disciplines within the social sciences and humanities (e.g. the Social Work profession) do not always include even an introduction to economics for their students, notwithstanding that economic issues are often very important
determinants of the environment within which they operate. More fundamentally, questions arise as to whether social science is more than the sum of its respective parts, and as to whether the roots of economics can be fully understood in isolation from the history not only of economics but also of politics and philosophy. 6. Establish a link to the teaching of economics in the secondary schools, given that tertiary enrolments in economics reflect fluctuating enrolments in economics in the secondary schools. 7. Encourage on-going surveys of student response to the teaching of economics across Australasian (and other) institutions, including response to experimental teaching and to differences between institutional approaches. (c.f. Colander and Klamer’s 1988 survey of economics students at USA ivy league institutions.) 8. Monitor trends in the teaching of economics both globally and in the Australian and New Zealand university systems (such as enrolments, staff-student ratios, international-domestic student ratios, offshore offerings etc), and the implications of those trends for various funding arrangements. 9. Promote a series of papers on specialised themes within the overall province of the teaching of economics e.g. on the teaching of Principles courses, the teaching of History of Economic Thought, the teaching of intermediate microeconomics and macroeconomics, the teaching of development economics, and likewise regarding teaching in such streams as Quantitative Methods, large first year classes, non-English speaking background students, the teaching of economics to non-economists, product differentiation in teaching economics, and professional education in economics in executive education programs outside conventional university contexts. 10. Monitor the measuring and rewarding of quality (economics) teaching within Australasian universities.
AUSTRALASIAN JOURNAL OF
ECONOMICS EDUCATION
Volume 13, Number 1
March, 2016
CONTENTS
ARTICLES
A Report on the Development of Learning
Standards for Economics in Australia
Ross Guest
Allan P Layton
1
The Impact of Lecture Capture on Student
Performance
Chris Jones
Matthew Olczak
13
Selecting Sessional Tutors: A New and
Effective Process
Carl W. Sherwood
Bruce Littleboy
30
BOOK REVIEW
Teaching Innovations in Economics
M. K. Salemi & W. B. Walstad
Peter Docherty
49
Australasian Journal of Economics Education
Volume 13, Number 1, 2016, pp.1-12
A REPORT ON THE DEVELOPMENT OF
LEARNING STANDARDS FOR ECONOMICS IN
AUSTRALIA*
Ross Guest
Griffith University
Allan P Layton
University of Southern Queensland
ABSTRACT
Guest (2013) motivated the development of learning standards for Economics in
Australia. Since then, under the auspices of an Australian Office for Learning and
Teaching (OLT) National Senior Teaching Fellowship, with support from the
Australian Business Deans Council (ABDC) and endorsement by the Economics
Society of Australia, a set of Learning Standards for Economics in Australia has
been developed through a collaborative and highly consultative process. In
addition, a set of exemplar assessment items has also been developed which align
with, and link to, the developed Learning Standards. This paper provides an update
on this development including: details of the process by which the Learning
Standards were developed, a brief discussion of the Learning Standards themselves
and of the accompanying assessment items, and, finally, a discussion of the
intended impact of the whole process on the teaching and learning of Economics
at Australian universities.
Keywords: learning standards, learning outcomes, student assessment.
JEL classifications: A22.
* Correspondence: Ross Guest, Griffith Business School, Griffith University, Business 3
(G06), 2.12, Gold Coast Campus, Parklands Drive, Southport, QLD, 4222, Australia.
Phone: (07) 555 28783; E-mail: [email protected]. This paper draws on material
from Guest (2013, 2014) and the Economics Learning Standards website
http://www.Economicslearningstandards.com. The authors acknowledge contributions
from members of the Economics Learning Standards Working Party and Expert Advisory
Group – listed on the website - who all contributed to the development of the Economics
Learning Standards.
ISSN 1448-4498 © 2016 Australasian Journal of Economics Education
2 R. Guest & A.P. Layton
1. INTRODUCTION
As discussed in Guest (2013, 2014), under the rubric of the Tertiary
Education Quality Standards Agency (TEQSA) Act 2011, Learning
Standards are “threshold standards”. This approach requires, inter alia,
that “the academic standards intended to be achieved by students and
the standards actually achieved by students in the course of study be
benchmarked against similar accredited courses of study offered by
other higher education providers” (Australian Government 2011, p. 17).
Furthermore, whilst the articulation of general teaching and learning
standards falls under the ambit of the Higher Education Standards Panel
(HESP), it is generally acknowledged that the TEQSA Act will
necessitate each discipline developing its own Learning Standards
consonant with the HESP general standards.
As has been acknowledged elsewhere, it is very important to note
that, whilst the Act uses the term “learning standards” it is not intended
that the ultimate outcome be standardization across Australia’s tertiary
institutions at either the general level of Learning Standards, nor at the
discipline level. On the contrary, learning standards describe what is
essential for graduates to have attained at graduation and this leaves
considerable room for diversity across institutions. Indeed, as will be
seen later, the Economics Learning Standards have been developed in
such a manner as to allow for great diversity in curricula across the
sector, whilst nonetheless allowing for the sort of benchmarking as
envisaged by the Act. Furthermore, as will be evident later, the term
“standards” may even be misleading and a more appropriate descriptor,
as least in the case of the Economics Learning Standards described
below, might be “learning outcomes”.
In the sections below, we will, in brief, traverse again some of the
motivations for developing Learning Standards in Economics (Section
2), provide a brief summary of the process used to develop the
Economics Learning Standards, the Learning Standards themselves, an
overview and flavour of the assessment exemplars (Section 3), the
intended impact of the Learning Standards on teaching and learning
(Section 4), with brief concluding remarks following in Section 5.
2. THE NEED FOR AUSTRALIAN LEARNING STANDARDS
The motivation for Learning Standards in Economics is threefold:
legislative compliance, educational improvement through curriculum
renewal, and earlier precedent developments in cognate disciplines. A
Learning Standards for Economics 3
widespread movement towards discipline-specific learning standards in
higher education has occurred both nationally and internationally over
the past decade (Guest 2013, 2014).
At the international level, these include: the OECD’s Assessment of
Higher Education Learning Outcomes (AHELO) Feasibility Study,
(which included an Economics strand); the Tuning Project which arose
from the European Bologna Process, and which interestingly overtly
avoids the term “standards” but defines “intended, expected or desired
learning outcomes” in Economics. The UK’s Subject Benchmark
Statements - including Economics - have been in place since 2000, and
the Economics Statement was revised in 2007.
At the national level, the Federally-funded Learning and Teaching
Academic Standards (LTAS) project developed a set of threshold
learning outcomes for a number of disciplines, with Accounting being
chosen as the first discipline from Business and Management to
produce a set of LTAS, and which were published in December 2010
(Hancock & Freeman 2010) with support from the Australian Business
Deans Council (ABDC). Since then, learning standards have been
developed, also under the auspices of the ABDC, for the business
disciplines of Marketing, Tourism Hospitality and Events Management
and Finance.
As previously mentioned, a key legislative catalyst for the
development of Economics Learning Standards was the TEQSA
legislation. Under this legislation all higher education institutions are
required to be able to demonstrate: (i) that their internal processes for
the design and approval of each degree take account of external
standards, and (ii) that the outcomes achieved by their students are
benchmarked against external standards. Learning Standards, such as
those provided in this document, developed through extensive
consultation with the discipline community, provide one reference
point for benchmarking. Another driver for these Learning Standards is
the Australian Qualifications Framework (AQF), which specifies
descriptors of learning outcomes in terms of knowledge, skills and
application abilities for graduates. These are generic descriptors, first
introduced in 1995, but revised in 2011. The AQF was the starting point
for the LTAS discipline-specific statements of threshold learning
outcomes, where the term “threshold” meant minimum.
Learning Standards provide guidance to a range of national and
international stakeholders, including academics designing new
4 R. Guest & A.P. Layton
programs, academics wanting to benchmark their existing programs,
employers who want assurance about what Economics graduates know
and can do, prospective students and secondary school course advisors
who want to know what Economics at tertiary level is about, and the
wider discipline community who wish to have assurance about the
relevance and value of Economics learning outcomes. Also, learning
standards provide the opportunity for widespread curriculum renewal,
with emphasis on ensuring that graduates of Economics have attained
the core skills and knowledge that are needed in order to effectively
practice the discipline of Economics.
Evidence from employers and elsewhere, both nationally and
internationally, indicates a need for such a renewed focus (Guest &
Duhs 2002; O’Doherty et al. 2007; Walstad & Rebeck 2002). In
particular this evidence suggests that, according to national and
international evidence about the lack of ability of Economics graduates
to address real world contexts, a refocussing on deeper understanding
and application of core knowledge and skills is needed. A UK study
(O’Doherty et al. 2007) found that “Economics graduates are not
particularly good at applying knowledge” to any “real world” situation.
Similarly US Economics graduates were found to have a weak
understanding of fundamental economic concepts and only a “slightly
better” ability to apply economic principles to solve economic problems
(Walstad & Rebeck 2002). In the Australian context, former Australian
Productivity Commission Chairman Gary Banks has argued that:
. . . in many cases the sort of Economics needed to inform policy
decisions is not very complicated or sophisticated. Much public policy
could go a long way with a few basic principles or precepts . . . . . But
we see policy proposals and decisions that violate those principles
almost on a daily basis.
(Banks 2011, p. 2)
3. THE DEVELOPMENT PROCESS AND OUTCOMES
The following four principles were considered fundamental to guiding
the development of the Learning Standards:
1. Learning Standards should reflect the minimum learning
outcomes that all graduates are expected to have attained.
Additional learning outcomes and those of an aspirational nature
are outside the scope of the Learning Standards;
2. Learning Standards should recognise both diversity among
Economics programs as well as the need to enable learning
Learning Standards for Economics 5
outcomes to be compared across institutions and student cohorts.
However, the standards will not prescribe a set of topics, learning
activities or assessment items;
3. Learning Standards should be consistent with AQF standards and
should be informed by international standards;
4. The process should be collaborative, evidence-based, transparent
and iterative, incorporating feedback from the discipline
community including academics and other economists from the
private and public sectors.
Guest (2014) describes the process in detail which can be summarised
here as follows. The approach was based on extensive consultation and
engagement with the Economics discipline community, both nationally
and internationally, followed by extensive dissemination activities.
A working party of academic economists - chaired by co-author, Ross
Guest - was appointed in late 2012 (by expressions of interest to
Australian academic departments followed by appointment by a
selection panel) to develop the learning standards for Australian
Economics degrees at Bachelor and Coursework Masters level. At the
same time an Expert Advisory Group (EAG) was established which was
chaired by co-author, Allan Layton.
See Guest (2014) for a complete list of members of both the working
party and the EAG. The Working Party held nine meetings from late
2012 through 2013. In April 2013 the first draft went to the EAG for
consideration, and in May 2013, a series of in-person workshops were
held in university departments covering all States including the ACT.
Each workshop was an open forum where the aim, process and
outcomes to that point were explained, followed by a discussion. The
aim of the workshop was to capture feedback to be shared with the
Working Party in developing the next draft. There were 19 such
workshops held over a six month period. A Symposium entitled
Learning Standards in Economics for Australia: A Draft for Debate and
Discussion was then conducted at the Australian Conference of
Economists (ACE 2013) in July, 2013. The symposium panel consisted
of members from the Working Party and the co-authors. All interested
stakeholders from the academic, private and public sectors attending the
conference were invited to participate in the ongoing discussion.
In September 2013 a second draft was distributed through a survey
of Australian economists (mainly academics). Over 800 respondents
were targeted and 137 responses were received. Consultation was also
6 R. Guest & A.P. Layton
held with the external evaluator of the National Fellowship, Professor
John Sloman, and other UK Economists at the Developments in
Economics Education conference at the University of Exeter. In late
2013 further revisions were made and in November 2013 the final
version – being considered final by both the Working Party and the
EAG - was disseminated to primary stakeholders and posted on the
website (http://www.Economicslearningstandards.com/). At that point
the Learning Standards were also presented at meetings of the ESA by
Ross Guest and of the ABDC by the ABDC Teaching Fellow, Mark
Freeman, who was also on the Working Party. Both the ABDC and the
ESA endorsed the Learning Standards at their respective meetings.
(a) The Outcomes: Economics Learning Standards for Australia
The Learning Standards apply to a program of study badged as an
Economics program. This would typically be the case for a Bachelor
(or Master) of Economics, or Bachelor (or Master) of
Commerce/Business degree with a Major in Economics. These
programs may carry a reference to Economics in the degree
nomenclature, such as Bachelor of Commerce (Economics). The
Learning Standards have been developed at both bachelor level (AQF
Level 7) and Masters (coursework) level (AQF Level 9). ‘Masters level’
applies to both entry-level Masters degrees and advanced Masters
degrees. The Learning Standards presented here do not apply to
research Masters or Honours degree programs in Economics.
The question then is: what constitutes the study of Economics? The
Learning Standards define Economics as “the study of the factors that
influence income, wealth and well-being. Its aim is to analyse and
understand the allocation, distribution and utilisation of resources and
their consequences for economic and social well-being”. The Learning
Standards provide a further contextual statement as follows. Economics
provides analytical methods to address problems and issues in society.
Analytical methods refer to theoretical models and empirical tools for
explaining and predicting both microeconomic and macroeconomic
behaviour. Microeconomics is concerned with the behaviour of
individual consumers, workers, firms, markets and industries, and the
way they interact. It is also concerned with the role of government
regulation in moderating behaviour and interaction among these
groups. Macroeconomics refers to the analysis of the behaviour of
economy-wide phenomena such as unemployment, inflation, economic
growth, the distribution of income and wealth, financial markets,
Learning Standards for Economics 7
exchange rates, international trade in goods, services and capital, and
the role of government policies, including fiscal and monetary policies,
in influencing these phenomena.
(b) The Learning Standards
The Economics Learning Standards are defined in terms of a set of
minimum learning outcomes rather than aspirational outcomes.
Bachelor and Masters learning standards are distinguished in terms of
knowledge and skills and their application. Compared with Bachelor
graduates, Masters graduates are expected to have attained knowledge
that is more complex, more integrated and more inclusive of recent
developments in the discipline. They should also be able to analyse
more critically and reflectively, communicate to wider audiences, and
be able to plan and execute a research-based project or piece of
scholarship.
The five learning domains and associated learning outcomes for
Bachelor and Masters graduates are summarised in Table 1. Whilst the
learning outcome domains are listed separately, this is simply for
taxonomic purposes. The work of Economics graduates often draws on
several of the learning domains simultaneously. Furthermore,
associated with the Learning Standards described in Table 1, there is
also a set of Economics concepts which are outlined as pertaining to the
Knowledge domain. These are included in Appendix A of the Learning
Standards document located on the website.
In addition, the full standards come with a set of notional micro and
macroeconomic examples of what could/should be expected of
graduates, one for each learning domain. For example, in the
Application domain (refer to Table 1), a macroeconomic example
provided in the Learning Standards is:
Bachelor graduates will be able to identify and evaluate possible causes
of a change in the rate of unemployment, including changes to labour
force participation, and broader economic influences such as
government policy and macroeconomic aggregates.
Masters graduates will have a more advanced understanding of
unemployment as a labour market outcome, and will be able to discuss
debates over the causes of the different types of unemployment.
Finally, the full set of Learning Standards also provide a set of real
life exemplar tasks – provided by members of the EAG – in which
graduate economists can expect to find themselves engaged, and as
8 R. Guest & A.P. Layton
Table 1: The Economics Learning Standards for Higher Education
Learning
Domain
Learning Outcomes
Bachelor Degree Masters Degree
Knowledge Bachelor graduates will be able to:
identify, coherently explain and
synthesise core economic concepts.
Masters graduates will be able to:
identify, coherently explain and
synthesise core and advanced
economic concepts, including
recent developments in the
discipline.
Application Bachelor graduates will be able to:
frame problems in terms of core
economic concepts and principles;
apply economic reasoning and
analytical skills, in order to make
informed judgments and decisions.
Masters graduates will be able to:
frame and critically analyse
problems in terms of core and
advanced economic concepts and
principles;
apply advanced economic
reasoning and analytical skills,
including quantitative techniques
where appropriate, in order to
make informed judgments and
decisions;
plan and execute a research-
based project.
Data Analysis
Bachelor graduates will be:
able to use economic data to address
typical problems faced by
economists;
aware of, and able to implement,
basic empirical techniques and
interpret the results.
Masters graduates will be able to:
select and apply an appropriate
empirical method to address
typical problems faced by
economists;
critically evaluate the results.
Communication Bachelor graduates will be able to: present a clear and coherent
exposition of economic knowledge,
ideas and empirical evidence both
orally and in writing, individually or
in collaborative contexts.
Masters graduates will be able to: communicate complex ideas
clearly and coherently, in written
form and interactive oral form to
expert and non-expert audiences,
individually or in collaborative
contexts.
Reflection
Bachelor graduates will be able to
reflect on: the nature and implications of
assumptions and value judgments in
economic analysis and policy;
interactions between economic
thinking and economic events, both
historical and contemporary;
the responsibilities of economists
and their role in society.
Masters graduates will be able to
reflect on and evaluate: the nature and implications of
assumptions and value
judgments in economic analysis
and policy;
interactions between economic
thinking and economic events,
both historical and
contemporary;
the responsibilities of economists
and their role in society.
Learning Standards for Economics 9
noted in the Learning Standards, such tasks will often draw on more
than one and often all of the five Learning Standards domains.
(c) Assessment of Learning Outcomes
The OLT subsequently funded a follow up project related to
assessment, which was co-led by the two co-authors. The Project
produced a set of exemplar assessment items which overtly aligned with
the newly developed Economics Learning Standards.
The purpose of the developed repository of assessment exemplars
was to illustrate how to design Economics assessment tasks which
explicitly and transparently map to the Learning Standards discussed
above. Full details are provided on the website. The assessment
repository was developed in a collaborative way by a core assessment
project team of academic economists drawn from the initial Economics
Learning Standards Working Party and the EAG as well as other
advisors on assessment design and business education.
For the Learning Standards to be fit for purpose, the assessment tasks
in each unit of study must be explicitly and transparently designed to
address particular Learning Standards, such that achievement of all of
these Learning Standards can be demonstrated by the end of the degree
program. This was the primary motivation for developing the set of
assessment task exemplars located on the Learning Standards website –
that is, to provide guidance to those academic economists who want to
ensure that their assessment is explicitly and transparently linked to the
Learning Standards.
Importantly, the exemplars seek to be authentic in that the tasks are
set in the context of real world issues/situations, and focus on the
application of knowledge and skills. The exemplars also illustrate
innovative types of tasks, including approaches for assessing
communication skills, both written and oral. The tasks typically ask
students to appropriately frame the problem or issue in terms of
economic principles, apply those principles and communicate the
outcomes to appropriate audiences. The tasks may also require an
ability to reflect on changes in economic thinking over time, and on the
interplay between economic events and development of theory and
policy.
The exemplars cover the broad areas of macroeconomics,
microeconomics, data analysis/econometrics, international economics
and economic thought. In each exemplar, an overview of the item is
provided first which outlines what the assessment task seeks to achieve
10 R. Guest & A.P. Layton
and how this is to be accomplished in terms of the use of the assessment
instrument. Details of the task are then provided assuming a Bachelor’s
assessment level with each part of the assessment item referencing
particular Learning Standards it is addressing. It is then demonstrated
how the item can be adapted to cater for the Master’s level (with similar
reference to the Learning Standards), followed by suggestions about
how the item might be used – perhaps in some adapted form - for
different purposes, and/or how it might be adapted to assess different
Learning Standards at different levels.
In total there are 18 assessment exemplars provided on the website
across the areas listed earlier. Readers are invited to visit the website1
to interrogate any exemplars potentially of interest to them to get a
sense of the resource. It is hoped that academic economists will find
this assessment repository a useful resource in guiding development of
their own assessment tasks that are explicitly and transparently linked
with the Economics Learning Standards.
4. INTENDED IMPACT OF LEARNING STANDARDS
The whole point of learning standards is, ultimately, to improve the
learning – and the teaching - of Economics, and, in particular, to
improve the ability of Economics graduates to practice the discipline of
Economics in a range of professional contexts and as citizens. Guest
(2013) sets out the case for this, citing national and international
evidence that Economics graduates fall short of expectations of
employers, academics and indeed themselves in their ability to apply
economic principles to real world problems outside of the classroom.
This evidence is based on surveys of Economics graduates and
employers (cited in detail in Guest 2013). It is also worth noting that
student enrolments in Economics programs have struggled for several
decades in the face of competition from other business disciplines
(again cited in Guest 2013).
All of this evidence points to a need for curriculum renewal. However
as we discovered in the 12 month process of producing the Australian
Economics Learning Standards, curriculum debates are fraught.
Considerable evidence suggests that ‘less is more’ in relation to content
coverage (Guest 2013). That is, learning is deeper, more lasting and
more useful when students are given more time on task, and more
1 http://www.Economicslearningstandards.com/search-for-an-assessment-exemplar -via-
field-of-Economics.html.
Learning Standards for Economics 11
opportunities to apply principles in a range of new contexts. This
necessarily however comes at the cost of breadth of topic coverage.
The problem then becomes that it is very hard to get agreement about
what content should be cut, and what core skills should be emphasised.
The pluralist or heterodox school argues for a broad range of economic
thinking perspectives to potentially be brought to bear on an economic
problem or issue (Guest 2013). This tends to imply a broadening rather
than narrowing of content, taking in schools of thought including
Austrian, institutionalist, evolutionary, behavioural, post Keynesian,
feminist and Marxist approaches, as well as perspectives from
alternative social science disciplines. It is fair to say that the set of
developed Economics Learning Standards tend toward the
traditional/orthodox perspectives on Economics, but not entirely – the
learning domain, “reflection”, recognises the contestability of
assumptions and value judgements, as well as the evolution of
economic thinking.
The Australian Economics Learning Standards focus on application
of core knowledge and skills. It is hoped that the examples in the
Learning Standards of the way in which each of the learning domains
could be operationalised will focus teachers and students on real world
applications. Similarly, the exemplars of assessment items, produced
subsequently and posted on the website, are explicitly focussed on
applications using real data/contexts. Finally, the Learning Standards
also include the important domain of “communication”, reflecting
feedback from employers that graduates – and not just of Economics
programs - fall short of expectations in their ability to present ideas and
analytical results effectively to a range of audiences.
5. CONCLUDING REMARKS
The Economics Learning Standards for higher education in Australia
are certainly not meant to be the last word on minimum learning
outcomes in Economics in this country. They will evolve as the
discipline itself evolves and as the student body evolves in terms of
prior learning and preparation for tertiary education. The Economics
Society of Australia and the Australian Business Deans Council, both
of which supported the development of the Learning Standards, have
an important role to play in leadership of the evolution of learning
standards in Economics. Indeed the success of the Learning Standards
will depend on the commitment of the whole of the academic
Economics community. It is important that the approach to Learning
12 R. Guest & A.P. Layton
Standards is not one of simple compliance, but rather continues as a
dynamic driver of curriculum renewal along the lines suggested above.
REFERENCES
Australian Government (2011) Higher Education Standards Framework
(Threshold Standards) – F2012L00003, available at www.comlaw.
gov.au/details/F2012L00003/Html/Text#_Toc311791711.
Banks, G. (2011) ‘Does Australian Public Policy Get the Economics It
Deserves?’, Agenda, 18 (3), pp.21-29.
Guest, R. (2013) “Towards Learning Standards in Economics in Australia”,
Economic Papers, 32 (1), pp.51–66.
Guest, R. (2014) Economics Learning Standards for Australian Higher
Education, Australian Government Office for Learning and Teaching,
Final Report, available at http://www.olt.gov.au/resource-economics-
learning-standards-Australian-higher-education.
Guest, R. and Duhs, A. (2002) “Economics Teaching in Australian
Universities: Rewards and Outcomes”, Economic Record, 78 (241),
pp.147–60.
Hancock, P. and Freeman, M. (2010) Learning and Teaching Academic
Standards Project Business, Management and Economics. Learning and
Teaching Academic Standards Statement for Accounting, Strawberry
Hills, Australia: Australian Learning and Teaching Council, available at:
http://www.olt.gov.au/resource-accounting-ltas-statement-altc-2010.
O’Doherty, R., Street, D. and Webber, C. (2007) The Skills and Knowledge
of the Graduate Economist: Findings of a Survey Conducted on Behalf of
the Royal Economic Society and the Economics Network, Bristol, UK.
Walstad, W.B. and Rebeck, K. (2002) “Assessing the Economic Knowledge
and Economic Opinions of Adults”, Quarterly Review of Economics and
Finance, 42, pp.921–35.
Australasian Journal of Economics Education
Volume 13, Number 1, 2016, pp.13-29
THE IMPACT OF LECTURE CAPTURE ON
STUDENT PERFORMANCE*
Chris Jones and Matthew Olczak
Aston Business School,
Aston University, Birmingham, UK
ABSTRACT
This paper investigates whether watching online recordings of live lecturers
improves student performance on a large first year introductory economics module
taught in a Business School. Our results show that performance on other modules
and previous experience of economics are the key determinants of performance on
this module. However, we also show that watching the lecture recordings may go
some way to counteracting a student’s lack of previous experience in the subject.
Keywords: technology, lecture recording, lecture capture, and student performance.
JEL classifications: A22.
1. INTRODUCTION
Technology that enables live lectures to be recorded and subsequently
made available for students to watch online has become widely used in
university teaching.1 Certainly from our experience at Aston
University, lecture capture has developed from initial experimentation
to virtually all lectures over a period of less than five years.
Furthermore, a broad range of evidence discussed below suggests that
this innovation is extremely popular with students. However, the
perceived benefits also appear to differ significantly across the diverse
* Correspondence: Matthew Olczak, Aston Business School, Aston University,
Birmingham, B4 7ET, UK, email: [email protected] Tel: +44(0)121 204 3107. Chris
Jones may be contacted at Aston Business School, Aston University, Birmingham, B4
7ET, UK, email: [email protected] Tel: +44(0)121 204 3036. Thanks to two
anonymous referees for comments and suggestions.
1 See Stephenson & Cortinhas (2013) for a detailed overview of how this technology
works.
ISSN 1448-4498 © 2016 Australasian Journal of Economics Education
14 C. Jones & M. Olczak
student population. Furthermore, the views of instructors are less
positive.
Given these mixed views, there is clearly the need for direct evidence
that investigates the effect of watching lecture recordings on student
performance. Therefore, the aim of this paper is to investigate this in a
large first year introductory economics module taught in a Business
School. We estimate the average effect watching the recordings has on
student performance in this module and, in addition, consider whether
this effect differs depending upon the characteristics of the student.
Our paper relates to a growing body of literature that tries to assess
the impact of various technologies on student learning of business and
economics. The approach typically taken has been to compare a control
group who have face-to-face lectures with a treatment group who only
receive online lectures. In contrast, we are interested in examining the
effect of watching lecture recordings on student performance when this
technology is made available to all students taking the module.
Therefore, more closely related to our methodology are a number of
papers, described in detail below, that examine student choice over
whether or not to make use of the lecture recordings provided.
The remainder of the paper proceeds as follows. In section 2 we
summarise the alternative views in the literature of lecture capture and
outline the existing literature that assesses the impact of technology on
student learning. In section 3 we provide an overview of the module
studied and outline our empirical methodology. In section 4 we first
show that the majority of the students on the module made at least some
use of the lecture recordings; some for a considerable amount of time.
Interestingly, these students were more likely to have no previous
experience of economics. We then report our baseline econometric
results. These show that performance on other modules and previous
experience of economics are the key determinants of performance on
this module. However, we also find that watching the lecture recordings
has a small positive effect on performance and that this may go some
way to counteracting a student’s lack of previous experience in the
subject. Nevertheless, this finding should be interpreted with some
caution since there is an obvious concern that it may be affected by self-
selection bias. This is because the students that watched the recordings
may have been more able students who would have performed better in
the module regardless of whether or not they watched the recordings.2
2 See Greene (2003, p.788) for a formal demonstration that this results in an upward bias.
Lecture Capture & Student Performance 15
Below we outline how we attempt to control for this potential bias.
However, despite this, our finding of the impact of watching lecture
recordings is best interpreted as an upper bound and therefore any
significant effect can be interpreted as being small. At the end of section
4 we discuss various robustness checks on our baseline results. Finally,
in section 5 we make some concluding remarks.
2. RELATED LITERATURE
Our treatment of the literature is divided into that which considers
alternative views of lecture capture and that which assesses the impact
of lecture capture on student learning.
2.1 Alternative Views of Lecture Capture
The literature which addressed the issue of how lecture capture is
perceived may also be divided into that which reflects the views of
students, the heterogeneity of the benefits that students perceive from
lecture capture, and that which reflects the views of instructors. Each of
these issues is addressed separately.
2.1.1 Popularity of lecture capture with students
A reasonably large body of survey evidence from a range of disciplines
shows how popular lecture capture is with students. For example,
Gosper et al. (2008) reports the results of a large survey conducted
across a number of Australian universities and a wide range of
disciplines showing that the majority of students perceived their
experience of lecture capture to have been positive and claimed that it
helped them to achieve better results.3 Furthermore, in our module
student feedback the overall satisfaction score was high and many
comments commended the provision of lecture recordings.
However, evidence on how much students actually value and make
use of recorded lectures is mixed. Taplin et al. (2011) try to more
accurately assess student valuation of lecture recordings by measuring
their willingness to pay for them. They found that many accountancy
students had a low valuation both in terms of willingness to pay and the
fact that they would only on average be prepared to substitute 2 of 11
tutorials for the provision of recorded lectures (Taplin et al. 2011,
p.183).4 Survey evidence in Wong (2013) also suggests that students
found other online resources, such as solutions to exercises, lecture
3 See also other consistent evidence in for example Taplin et al. (2011) and Elliott & Neal
(2016). 4 This evidence is also confirmed for a wider range of disciplines in Taplin et al. (2014).
16 C. Jones & M. Olczak
notes and other information more useful than recordings. In contrast, in
Elliott & Neal (2016) most students made at least some use of the
recordings and typically viewed a reasonably large proportion of the
lectures, whereas usage was much lower in Larkin (2010) and Taplin et
al. (2011). It is also clear that students value face-to-face class time and
don’t regard recordings as a substitute (Wong 2013; Gosper et al. 2008;
Larkin 2010; and Taplin et al. 2011).
2.1.2 Heterogeneity of perceived benefits from lecture capture
Not only is the evidence on valuation and usage of lecture recordings
mixed, it also appears that the benefits can differ substantially across
students. For example, Wong (2013) highlights the fact that
technological innovations can be especially useful as support for a
diverse group of students, in particular those who are in employment as
well as studying. The survey results in Flores & Savage (2007) show
that students who substitute online recordings for attending the live
lecture valued the availability of recorded lectures highly. Furthermore,
watching the lectures improved performance. However, they also find
that a nontrivial subset of students do not use the technology in this way
and therefore don’t value its provision. Likewise, despite the Taplin et
al. (2011, 2014) estimates of student valuation of lecture recordings
being low, their results suggest that the overall benefits of providing the
recordings exceed the costs. This is because a few students value their
provision very highly. As they go on to discuss, this raises equity and
ethical issues.
2.1.3 Instructor views on lecture capture and its impact
Instructor views on lecture capture also appear to be less positive than
those of the students. In Gosper et al. (2008, pp.22-23) only 29% of the
staff surveyed stated that their overall experience was almost always
positive.5 Furthermore, whilst 49% of staff said the technology made it
easier for students to study, only 30% agreed that it helped the students
to achieve better results. This compares to 67% of students who claimed
that it had helped them to achieve better results. Staff concerns included
a negative effect on student attendance, reduced communication with
5 Interestingly, good experiences were positively correlated with staff having chosen to
introduce the technology rather than being required to do so. They also find that about
one third of the lecturers using recording technology had made no substantial change to
their teaching approach as a result. In contrast, they argue that such technological
innovations should ideally impact on the whole course design.
Lecture Capture & Student Performance 17
students and deterioration in their ability to motivate and inspire
students.
In terms of a negative effect on attendance, the actual evidence is
mixed (see, for example, Taplin et al. 2011).6 This may be because, as
Larkin (2010) points out it is conceivable that watching recordings
enables students to see what they are missing and gain confidence in
the subject and that this will actually encourage them to subsequently
attend lectures.
Like Gosper et al. (2008), a number of other papers have also raised
similar concerns about the impact of lecture recordings on staff-student
interaction, and questioned the effect on student engagement (see, for
example, Flores & Savage 2007; Taplin et al. 2014 and Bennett &
Maniar 2007). On the other hand, Larkin (2010) highlights the fact that
recordings provide an opportunity for the lecturer to listen back and
reflect on their teaching. In addition, the lecturer can potentially
respond to student requests for additional material by providing
supplementary online recordings. Therefore, in this way interaction
between students and lecturers can actually be enhanced.
Overall, given these mixed views on the benefits of watching lecture
recordings and the likelihood that the benefits that do arise may differ
across students, there is clearly the need for direct evidence on the effect
on student performance.
2.2 Assessing the Impact of Technology on Student Learning
There is now a growing literature that tries to assess the impact of
various technologies on student learning of business and economics
(see Agarwal & Day 1998, as an example of one of the first papers).
Specifically on lecture recordings, Savage (2009) did not find evidence
that students for whom recordings were made available performed
significantly better than a control group in which no recordings were
available. In addition, student attendance did not differ between the two
groups. In contrast, Wong (2013) finds that the introduction of a range
of technologies coincided with an increase in module pass rates and that
the best students used the online resources much more than students that
failed the module. A number of other papers haven taken a similar
approach by comparing a control group who have face-to-face lectures
with another group who only get online lectures. For example, Brown
6 In addition, the actual evidence in Gosper et al. (2008) was also mixed and in Larkin
(2010) and Stephenson & Cortinhas, (2013) the survey evidence attendance suggests it
was not affected.
18 C. Jones & M. Olczak
& Liedholm (2002) find that student performance in the latter group
was inferior, especially on questions that aimed to assess a deeper level
of understanding. In addition, there was also some indication that the
students taught online worked less hard on the course. More generally,
the results on the impact of teaching online on performance are mixed
(see Williams et al. 2012 for a summary). This suggests that, whilst the
availability of recorded lectures may be a useful complement to live
lectures, caution should be exercised in any moves towards replacing
these with recorded material.
More closely related to our methodology are papers that examine
student choice over whether or not to make use of lecture recordings
when they are provided to complement live lectures. Chen & Lin (2012)
examine evidence from an intermediate microeconomics course in
Taiwan. They find that most lecture views occurred during the revision
period; and on average, watching the relevant recording just prior to the
exam improved performance by around 5%. Whereas, viewings
immediately after the lecture did not have a positive impact on student
performance. Their data also shows that it was the students with poorer
attendance records that made the most use of lecture recordings. Crucial
to their approach is the need to isolate the impact of a particular lecture
for understanding a given topic.7 However, if key threshold concepts
are introduced which, once understood, facilitate the understanding of
subsequent material, then this approach becomes less valid. Evidence
suggests this may well be the case in first year introductory economics
modules (see for example Shanahan et al. 2006). Therefore, we instead
focus on the impact recorded lecture viewings have on overall module
performance.
Williams et al. (2012) use data on self-reported lecture attendance to
examine whether viewing lecture recordings are a substitute or
complement to lecture attendance and to test the impact this choice has
on student performance in their first year undergraduate economics
class. They identify two groups of students. Firstly, those that attended
very few lectures, some because of employment commitments, and
report that they use the recordings to catch-up. Secondly, those that
attended the most lectures and used the recordings to revisit material.
Whilst the first group of students were the ones that made most use of
7 This is because they attempt to control for the possible self-selection bias discussed
earlier by looking at the variation in a given student’s lecture viewings and matching this
with their performance on the relevant exam questions.
Lecture Capture & Student Performance 19
the recordings, they show that it was the latter group of students that
derived the most benefit. Recordings were useful as a substitute, but not
enough to eradicate the negative effect that non-attendance has on
performance. In our concluding section we compare our results with
those of Chen & Lin (2012) and Williams et al. (2012).
3. DATA AND METHODOLOGY
Economic Environment of Business is a first year Business School
module that provides an introduction to economics for students that will
typically not go on to major in economics. The teaching of the module
comprises 11 lectures, one of which is a revision session delivered at
the end of term, and 5 one hour tutorials per student taken fortnightly.
Each lecture lasted between 1 and 2 hours and every lecture was
recorded live using the Panopto software that the university has
adopted. The recordings were then made available for the students to
watch (but not download) from the university’s Virtual Learning
Environment (Blackboard) shortly after the lecture.8 This meant that
we could readily obtain detailed data on which students watched the
lecture recordings and for how long. The module had one piece of
assessment at the end of term - a 2 hour examination. This comprised a
40 question multiple-choice test (accounting for 40% of the total marks)
to assess breadth of knowledge and then two essays (a choice of one
from two on micro and one from two on macroeconomics) to assess
depth of knowledge. Our sample comprises the 380 students that sat this
examination for the first time (i.e. not for reassessment) in the Spring
term of 2012.
We are interested in explaining a student’s module performance (yi),
which will be measured by their overall examination mark. This will be
modelled as:
𝑦𝑖 = 𝛼𝑂𝑀𝑂𝐷𝑖 + 𝑥𝑖′𝛽 + 𝛿𝑊𝐴𝑇𝐶𝐻𝑖 + 휀𝑖 (1)
Most importantly given our focus, WATCHi is a binary variable
capturing whether or not student i decided to watch the online lecture
recordings. The variable OMODi is student i’s average year mark across
all the other modules they study in their first year (excluding this
module). This is an important control variable as it will pick up a 8 The recording software captures both the lecture and the PowerPoint slides. Unless
students request that the recording is paused, student interaction with the lecturer could
be captured. However, the microphone may not pick up some audience participation.
20 C. Jones & M. Olczak
student’s overall ability and level of engagement. Furthermore, it is
important to note that lecture recordings were not provided in any of
these other modules. This means that including this variable helps to
alleviate concerns that our results will be affected by the self-selection
bias discussed above. We return to discuss the possibility of self-
selection bias at the end of the paper. In addition, xi is a vector of other
characteristics of student i (summarised below). We also include fixed
effects to control for the student’s degree programme, parental
occupational class, school and ethnicity. Finally, εi is an error term.
Table 1: Descriptive Statistics for the Module Intake
Variable Obs Mean Std. Dev. Min Max
EXAM MARK (%)
380
58.35
13.77
9
95
GENDER (1 if male) 380 0.47 0.50 0 1
OVERSEAS (1 if overseas
student) 380 0.15 0.36 0 1
AGE (on entry in years) 380 19.1 2.37 17 42
OMOD (Average mark (%)
for all other first year
modules taken) 380 57.53 11.49 12.29 81.47
ECON (1 if prior study of
economics) 380 0.15 0.36 0 1
Table 1 summarises the key characteristics obtained from the
university records of the 380 students in our sample and their overall
exam performance. As Table 1 shows, the average overall mark for the
module was 58% and this was the same as the overall average mark
across all other modules.9 Most of the students had come to university
soon after completing their schooling and the majority had not studied
economics prior to joining the university.10 Finally, overseas students
(defined as those from outside the European Union) represented a
relatively small, but non-trivial proportion of the students.
9 This explains why the data is from Spring 2012. After this date lecture capture use
became more widespread and was then made compulsory. 10 For those that had studied Economics this was typically via an AS- or full A-level or
as part of the International Baccalaureate.
Lecture Capture & Student Performance 21
Table 2: Descriptive Statistics for Students Who Watched the Recordings
Variable Observations Mean
Std.
Dev. Min Max
MINS (total mins watched) 235 452.19 498.10 0.52 2668.27
NUM LECS (no. lectures
watched) 235 5.28 3.45 1 11
4. RESULTS
In total 235 (59%) of students watched part of at least one lecture
recording. Table 2 provides further information on exactly how much
these students watched the recordings. Out of the 235 students who
watched the lecture recordings, on average students watched about 5
of the lectures and for a total of around 7.5 hours. Figure 1 plots total
viewing activity across the term up to the examination date. It is clear
that the viewings peak substantially leading up to the exam. This is
entirely consistent with other evidence in the literature (for example
Elliott & Neal 2016 and Williams et al. 2012). Furthermore, consistent
with this previous evidence, there is also some indication that students
were being selective and only watching specific parts of the lectures.
Figure 1: Number of Minutes Watched Across the Term
Table 2 also suggests that a few students watched for a very short
period of time. This was perhaps to check that the recording had
worked, with the intention to go back and watch at a later date.
However, it was clearly not for sufficient time to benefit from using the
0
2000
4000
6000
8000
10000
12000
26/01/2012
05/02/2012
15/02/2012
25/02/2012
06/03/2012
16/03/2012
26/03/2012
05/04/2012
15/04/2012
25/04/2012
05/05/2012
Min
ute
s
22 C. Jones & M. Olczak
resource. In total, there were 18 students who viewed some of the
recordings, but for a total of less than 15 minutes. Therefore, in our
baseline specification we define having watched lecture capture as
having watched more than 15 minutes in total.11 Table 3 compares the
key characteristics of the students between those that did and did not
watch the lecture recordings for more than 15 minutes.
This shows that the students that made use of the recordings did better
across all the other first year modules. Therefore, this provides
additional evidence of the importance of including this control variable
in our subsequent econometric analysis. Interestingly, the students that
made use of lecture capture were less likely to have previously studied
economics and more likely to be home students.
We will discuss these findings further below. It is also worth noting
that in contrast, the 18 students who watched the recordings for less
than 15 minutes were more likely to have studied economics previously
than those that didn’t watch at all.12 Therefore, this provides some
additional justification for treating this small group of students
differently.
The first results column of Table 4 reports Ordinary Least Squares
(OLS)13 estimation results for our baseline specification where, as
outlined above, the WATCH variable captures the students who watched
a total of more than 15 minutes. These students represent 55% of our
sample and had a mean viewing time of just over 8 hours. It is clear
from these results that previous experience of economics and
performing well across all the other first year modules are key
determinants of student performance on the module. In contrast, a
student’s age, gender or whether or not they are an overseas student
appear to have no significant effect on their performance.
Most importantly, the key coefficient of interest on the variable
WATCH is also positive and significant and at the 10% level (p =
11 Since only 2 students watched for between 15 and 30 minutes, our results do not differ
substantially if we instead made the cut off at 30 minutes. 12 One possible explanation is that these students were keen at the start of the module, but
become disengaged early on in the term. Some evidence to support this is provided by the
fact that their brief viewings of the recordings were typically early on in the term. The
possible disengagement of this subset of students clearly merits further investigation. 13 Our dependent variable is bounded between 0 and 100. However, using OLS does not
appear to be problematic since the predicted values from our baseline specification all lie
within this interval.
Lecture Capture & Student Performance 23
Table 3: Conditional Mean Values of Key Student Characteristics
Students that watched
< 15 mins Recordings
Students that watched
> 15 mins Recordings
OMOD ** 56.073 58.725
ECON ** 0.192 0.111
GENDER 0.494 0.447
OVERSEAS ** 0.198 0.111
AGE 19.169 19.043
Two-sample t-test (allowing for unequal variance) testing whether means are
significantly different: *** p < 0.01; ** p < 0.05; and * p < 0.1.
0.053).14 However, the size of this effect is relatively small (95%
confidence interval of -0.031 to 4.146). Since most of the students that
watched for more than 15 minutes had not previously studied
economics,15 this result suggests that watching the lecture recordings
may partially offset their lack of previous experience. More
specifically, ceteris paribus, not having previously studied economics
is estimated to reduce a student’s expected examination performance by
6 marks, whereas watching the recordings for more than 15 minutes
increases this by 2 marks.
Therefore, the results from our baseline specification suggest that
watching the lecture recordings may have a small positive effect on
student performance.16 In order to test more precisely whether it
matters how much the students watched the lecture recordings, in
columns 2 and 3 of Table 4, we replace the WATCH variable with the
total number of minutes and lectures viewed as reported in Table 2. The
14 If we instead exclude the 18 students that watched the recordings but for less than 15
minutes then the size of the estimated effect from watching the recordings increases and
is slightly more significant (p = 0.045). 15 If we don’t control for previous experience of economics, the WATCH coefficient is
insignificant. This suggests that previous experience of the subject is the key determinant
of performance. In other unreported results we have also tried interacting the WATCH
variable with ECON. However, given that very few students with previous experience of
economics made use of the lecture recordings, it is not possible to disentangle the two
effects. 16 We have also allowed the effect of watching lecture capture to vary according to the
student’s performance on other modules, whether they had previously studied economics,
whether they were an overseas student and by gender. However, there was no evidence
to suggest that the effect of watching the recordings depends upon these characteristics.
24 C. Jones & M. Olczak
Table 4: OLS Regressions of Overall Examination Performance
Regression (1) Regression (2) Regression (3)
OMOD 0.776*** 0.788*** 0.782***
(0.0539) (0.0541) (0.0545)
WATCH 2.058*
(1.062)
MINS 0.00108
(0.00113)
NUM LECS 0.202
(0.132)
ECON 6.349*** 6.048*** 6.242***
(1.408) (1.408) (1.419)
GENDER 0.463 0.421 0.422
(1.011) (1.017) (1.015)
OVERSEAS -2.296 -2.188 -2.231
(1.725) (1.724) (1.721)
AGE -0.158 -0.164 -0.161
(0.298) (0.285) (0.290)
Programme Yes Yes Yes
Occ. Class Yes Yes Yes
School Yes Yes Yes
Ethnicity Yes Yes Yes
CONSTANT 8.429 7.218 8.351
(13.554) (13.392) (13.425)
Observations 380 380 380
Adj. R-squared 0.571 0.567 0.567
Robust standard errors in parentheses. These are calculated using the standard
Huber-White procedure to correct for possible heteroskedasticity. Symbol ***
indicates p < 0.01; ** p < 0.05; and * p < 0.1.
results show that neither of these variables has a significant impact on
performance. One interpretation of this is that it supports the evidence
that students are selective in the material that they watch. It may be
that additional information on how the students use the recordings, for
example on the extent to which students focus their viewings on the
parts of the lectures covering key threshold concepts and specific advice
on assessment, would provide further insights. As outlined in section 3,
we also include fixed effects to control for the student’s degree
programme, parental occupational class, school and ethnicity. To ease
the burden on the reader, we don’t report estimates for these here.
However, the full results for our baseline specification are reported in
the Appendix.
Lecture Capture & Student Performance 25
6. CONCLUSION
Consistent with Chen & Lin (2012) and Williams et al. (2012) we find
that watching lecture recordings may have a positive effect on student
performance. Also, in line with Chen and Lin, our results suggest that
watching the recordings during the revision period accounts for much
of this. However, our estimated effect on student performance is smaller
than in these previous studies. In Williams et al. (2012), despite using
the recordings less, it is the students that attended most lectures and then
use the recordings to revisit material that derived the most benefit from
watching the recordings. Recordings were useful as a substitute, but not
enough to eradicate the negative effect non-attendance has on
performance. Since we do not have data on attendance the smaller effect
that we estimate may partly be attributable to the effect of non-
attendance.
Our results also highlight another heterogeneous impact from
providing lecture recordings. In our sample, students with no previous
experience of economics were more likely to watch the lecture
recordings and our results indicate that this may have gone some way
to counteracting this. Our analysis of which students made use of the
recordings also shows that these were less likely to be overseas
students. This suggests that there may be scope for investigating how
to encourage this subset of students to make more use of the technology.
Overall, especially given the possibility that self-selection bias is still
present, our results suggest that whilst there may be some small benefits
for student performance from providing lecture recordings, this
technology is unlikely to result in substantial improvements. However,
additional evidence would be beneficial. Whilst perhaps difficult to
implement due to ethical issues, comparisons between a treatment and
a control group who (unlike in earlier research) receive the same live
lectures, but differ in terms of whether or not they are provided with
lecture recordings, would be one useful way to further evaluate the
benefits of this technology. Such an approach has been used to examine
other teaching innovations (see, for example, McMahon 2011).
APPENDIX
This appendix provides details of regressions that included fixed effects to
control for the students’ degree programme, parental occupational
class, school and ethnicity that were not reported in the main text.
26 C. Jones & M. Olczak
Table 5: OLS Regression of Overall Exam Performance with
Fixed Effects for Control Variables
Explanatory Variable Coefficient Estimate (s.e.)
OMOD 0.776 (0.0539)***
WATCH 2.058 (1.062)*
ECON 6.349 (1.408)***
GENDER 0.463 (1.011)
OVERSEAS -2.296 (1.725)
AGE -0.158 (0.298)
Programme
BSc Business Computing & IT -5.689 (4.633)
BSc Business & Computer Science -2.919 (3.503)
BSc Business & French -7.515 (3.254)**
BSc Business & Int. relations -2.749 (3.187)
BSc Business & Mathematics 6.363 (3.110)**
BSc Business & Politics -0.615 (2.682)
BSc Business & Psychology 4.881 (2.674)*
BSc Business and Public Policy Management 4.125 (2.469)*
BSc Business & Sociology 1.019 (7.314)
BSc Business & Spanish 3.794 (3.576)
BSc Computing for Business 0.586 (2.604)
BSc Human Resource Management -0.620 (2.465)
BSc International Business and Modern
Languages 3.859 (2.176)*
BSc International Business & Management 2.972 (1.882)
BSc Marketing 2.967 (2.034)
LL.B. Law with Management -1.900 (2.448)
Occ. Class
Higher Managerial 8.480 (6.003)
Intermediate Occupations 7.329 (5.934)
Lower Managerial Occupations 7.948 (5.929)
Lower Supervisory and Technical Occupations 5.857 (6.643)
Semi-routine Occupations 7.112 (5.995)
Small Employers and Own Account Workers 8.361 (5.906)
Lecture Capture & Student Performance 27
School Coefficient Estimate (s.e.)
Comprehensive School -3.973 (5.318)
Grammar School -9.473 (5.950)
Independent School -9.272 (5.290)*
Language School -10.333 (5.596)*
Sixth Form College -5.466 (4.795)
Special School 9.719 (5.757)*
Tertiary School -8.422 (5.331)
Unknown -4.879 (4.754)
Ethnicity
Asian Other 0.971 (4.721)
Bangladeshi 0.963 (6.253)
Black Other 8.780 (18.179)
Black or Black British - African 2.527 (4.196)
Black or Black British - Caribbean -0.338 (4.013)
Chinese 7.334* (4.177)
Indian 3.282 (3.535)
Information Refused 4.586 (4.364)
Other -2.457 (5.671)
Other Mixed Background 3.665 (5.012)
Pakistani 3.691 (4.241)
White 2.503 (4.450)
White & Black African 1.067 (5.248)
White & Black Caribbean -9.117 (6.270)
White British 2.551 (3.552)
White Other 3.073 (4.106)
CONSTANT 8.429 (13.554)
Observations 380
Adj. R-squared 0.571
Ethnicity classifications are those used by Aston University for reporting to the UK
Higher Education Statistics Agency (HESA). See http://www.aston.ac.uk/registry/for-
students/facts-figures/ as well as https://www.hesa.ac.uk/data-and-analysis/students.
Reference category for Programme is ABS Undergraduate Exchange, Occ. Class is
Unknown Category, School is Art Design and Performing Arts and Ethnicity is White
Irish. Robust standard errors in parentheses. Symbol *** indicates p < 0.01;
** p < 0.05; * p < 0.1.
28 C. Jones & M. Olczak
REFERENCES
Agarwal, R. and Day, A. E. (1998) “The Impact of the Internet on Economic
Education”, Journal of Economic Education, 29 (2), pp.99-110.
Bennett, E. and Maniar, N. (2007) Are Videoed Lectures an Effective
Teaching Tool?, Mimeo, http://podcastingforpp.pbworks.com/f/Bennett
%20plymouth.pdf.
Brown, W.B. and Liedholm, C.E. (2002) “Can Web Courses Replace the
Classroom in Principles of Microeconomics?”, American Economic
Review, 92 (2), pp.444-448.
Chen, J. and Lin, T-F. (2012) “Do Supplementary Online Recorded Lectures
Help Students Learn Microeconomics?”, International Review of
Economics Education, 11 (1), pp.6-15.
Elliott, C. and Neal, D. (2016) “Evaluating the Use of Lecture Capture Using
a Revealed Preference Approach”, Active Learning in Higher Education,
17 (2), pp.153-167.
Flores, N. and Savage, S.J. (2007) “Student Demand for Streaming Lecture
Video: Empirical Evidence from Undergraduate Economics Classes”,
International Review of Economics Education, 6 (2), pp.57-78.
Gosper, M., Green, D., McNeill, M., Phillips, R., Preston, G. and Woo, K.
(2008) The Impact of Web-based Lecture Technologies on Current and
Future Practices in Learning and Teaching: Report for Australian
Learning and Teaching Council, available at https://www.mq.edu.au/
ltc/altc/wblt/research/report.html.
Greene, W. H. (2003) Econometric Analysis, 5th edtion, Upper Saddle River,
N.J.: Prentice Hall.
Larkin, H.E. (2010) “ ‘But They Won’t Come to Lectures …’ - The Impact
of Audio Recorded Lectures on Student Experience and Attendance”,
Australasian Journal of Educational Technology, 26 (2), pp.238-49.
McMahon, M.F. (2011) “Classroom Games in Economics: A Quantitative
Assessment of the ‘Beer Game’ ”, Warwick Economics Research Paper,
No.964, Warwick University, UK.
Savage, S.J. (2009) “The Effect of Information Technology on Economic
Education”, Journal of Economic Education, 40 (4), pp.337-53.
Shanahan, M. P., Foster, G. and Meyer, J. H. F. (2006) “Operationalising a
Threshold Concept in Economics: A Pilot Study using Multiple Choice
Questions on Opportunity Costs”, International Review of Economics
Education, 5 (2), pp.29–57.
Stephenson, J. and Cortinhas, C. (2013) “Creative Uses of In-class
Technology”, The Handbook for Economics Lecturers, Economics
Network, available at https://www.economicsnetwork.ac.uk/handbook/
technology.
Lecture Capture & Student Performance 29
Taplin, R.H, Kerr, R. and Brown, A.M. (2014) “Opportunity Costs
Associated with the Provision of Student Services: A Case Study of Web-
based Learning Technology”, Higher Education, 68, pp.15-28.
Taplin, R.H, Low, L.H. and Brown, A.M. (2011) “Students’ Satisfaction and
Valuation of Web-based Lecture Recording Technologies”, Australasian
Journal of Educational Technology, 27 (2), pp.175-91.
Williams, A., Birch, E. and Hancock, P. (2012) “The Impact of Online
Lecture Recordings on Student Performance”, Australasian Journal of
Educational Technology, 28 (2), pp.199-213.
Wong, L. (2013) “Student Engagement with Online Resources and its Impact
on Learning Outcomes”, Journal of Information Technology Education:
Innovations in Practice, 12, pp.129-46.
Australasian Journal of Economics Education
Volume 13, Number 1, 2016, pp.30-48
SELECTING SESSIONAL TUTORS:
A NEW AND EFFECTIVE PROCESS*
Carl W. Sherwood & Bruce Littleboy
School of Economics,
University of Queensland
ABSTRACT
Devising an effective way to select high-performing sessional (casual) tutors from
a pool of applicants is relatively unexplored. Standard methods are to rank them
by their grade point average or to interview them one at a time. We report on a
new selection process based on observing a small group activity where applicants
discuss an unseen newspaper article and devise a tutorial question based on it. By
selecting tutors who demonstrate good knowledge, communication and inter-
personal skills, student evaluations of tutors have increased. Costs and benefits of
switching to this new process are identified and discussed.
Keywords: tutorials, small group teaching, sessional teaching, tutor selection.
JEL classifications: A22.
1. INTRODUCTION
Teachers and administrators aspire to improve the quality of the entire
learning process. We focus here on tutorials, small classes conducted
to support lectures. Tutor training is widely recognised as important,
but the quality and effectiveness of tutor training may vary. Training
and support is also costly. Better and more cost-effective tutors are
more likely to emerge if the applicants with greater potential are
initially selected. We argue that ‘better’ tutors are those who are
* Correspondence: Carl W. Sherwood, School of Economics, University of Queensland,
St Lucia, QLD 4072, Australia; Email: [email protected]; tel: +61-7-3365-6563;
fax: +61-7-3365-7299; Bruce Littleboy may be contacted at School of Economics,
University of Queensland, St Lucia, QLD 4072, Australia; Email: [email protected].
We are most grateful for the referee’s helpful comments and suggestions.
ISSN 1448-4498 © 2016 Australasian Journal of Economics Education
Selecting Sessional Tutors 31
already well organised, articulate and have strong interpersonal skills.
The task is then to identify such applicants before hiring them.
Applicants for senior teaching or research positions typically deliver a
staff seminar, but tutors are not similarly screened prior to appointment.
Remarkably little attention has been devoted to selecting tutors from
a pool of applicants. In the corporate world, the filtering of prospective
employees can be elaborate. However, in universities, tutors are
regarded as at the bottom of the academic hierarchy. Yet tutorials are
often students’ most direct source of engagement with a course,
especially when enrolments are very large. If tutorials matter, so does
the selection of tutors. Whatever ‘good teaching’ is, obtaining high tutor
evaluations nowadays from students is a managerial imperative. This
paper shows how the median overall evaluation scores of tutors can be
raised by changing the selection process.
We have located no literature that directly relates to any tutor
selection process and consequently, no analysis of subsequent tutor
performance. In particular, there is no analysis relating to tutor
communication and interpersonal skills during interview and to
subsequent tutor evaluation by students. There is a broader literature
on training and certification to achieve competence; for example,
Angrist & Guryan (2004), Becker (1997, p. 1351). Bosshardt & Watts
(2001) note that while students value speaking-skills and enthusiasm,
they value preparation and organisation more. Ragan & Walia (2010)
likewise point to the value of preparation. Bianchini et al. (2013) show
that instructor characteristics such as age and seniority, and perhaps
gender, can affect student evaluations. Korur & Eryilmaz (2012) report
on the ethnic and gender preferences of students for teachers in Turkey.
Some of these criteria are clearly inappropriate for appointing tutors.
This paper discusses a pilot selection process used during 2012 and
2013. Rising student tutor evaluations of tutors has encouraged its
ongoing use. Additional data have been collected since 2013 with the
intention to perform a more detailed statistical analysis of the selection
process and tutor evaluations. Here we simply provide an analysis
using descriptive statistics after the first year of implementation.
Processes for selecting tutors often falls into one of two conventional
kinds. One is to rank applicants using administrative criteria such as
Grade Point Average (GPA). The other is to judge an applicant’s
performance in front of an interview panel. We argue for a third
method, a small-group interview that allows better identification of
32 C.W. Sherwood & B. Littleboy
those likely to receive high student evaluation scores. We hold that
tutor skills revolving around communication, organisational, and
interpersonal skills matter in a tutor far more than grade point average
(GPA) or performance in a standard interview. Good teachers have an
ability to communicate well, and this attribute is looked for in
applicants. Successful applicants later only receive one day of training
before taking their first tutorial, so choosing well is important.
We aim to find tutors with an aptitude for communicating and an
ability to facilitate work in smaller groups. Tutorials typically have up
to 25 students. We assigned a group of three applicants a task that
relates to a graduate attribute outcome, namely an ability to apply
economic concepts in practice. We assigned an unseen newspaper
article story to the group and invited them to design a suitable tutorial
question based on the economic issues that the story related to. A panel
observed how well they responded.
First, we aim to identify those tutors likely to be high performers and
to reduce the number performing below a specified benchmark.
Success is measured by increases in the median overall tutor
effectiveness using Tutor Evaluation survey data. Second, we
investigate which of the criteria may help account for any improvement
in tutor evaluations. The criteria used to evaluate applicants in
interviews deliberately match criteria students use in official surveys to
evaluate tutor effectiveness. Third, we investigate the link between an
applicant’s GPA and their perceived effectiveness as a tutor. There are
three hypotheses:
Hypothesis 1: The new process increases the median
evaluation score.
Hypothesis 2: Good communication skills are essential for
being a good tutor.
Hypothesis 3: An applicant’s GPA is a necessary but not
sufficient condition for being an outstanding
tutor.
The following section describes the old selection process and its
shortcomings. The next section explains how the new interview process
works. A statistical analysis section then evaluates the results of the
new process. We then consider the wider costs and benefits of making
the change to the new interview process. Conclusions and reflections
follow.
Selecting Sessional Tutors 33
2. THE OLD SELECTION PROCESS
Since at least 2001, the School has advertised in October for tutors for
the following year. There are more high-quality applications each year
than available tutoring positions. Applicants were filtered according to
their overall course work GPA. Applicants with a GPA of 6 or more
were typically shortlisted and interviewed. Some categories of
applicants were given priority for an interview such as the fourth-year
Honours and PhD students. A tutorial position for these students would
provide income during their studies while also giving them teaching
experience useful for possible future academic careers.
After applicants were shortlisted, a panel of two academic staff and
at least one administrative staff-member interviewed each applicant
individually. This would take place in November and lasted for about
15 minutes. The interview used standard questions such as: What do
you think makes a good tutor? Why do you want to be a tutor? How
would you encourage students to participate in a tutorial? But clichéd
questions invite clichéd responses, and those already interviewed were
clearly informing those yet to be interviewed of the questions being
asked. Applicants had limited opportunities to demonstrate their ability
to perform in a tutorial setting. The process was also time-consuming
and regarded as a chore. The same people were reused as interviewers
as it was difficult to encourage other staff members to participate.
Typically, those involved were lecturers from very large undergraduate
courses that required many tutors, a senior School administrator, and
the administrative staff who interacted regularly with the tutors. While
promoting consistency within and between years, the selection process
was not transparent or even known to exist by most staff in the School.
At the end of the interviews in November, the great majority of
applicants interviewed were offered positions. In essence, the real
selection process was occurring pre-interview by administrative staff
and was driven by applicants’ GPAs. The absolute minimum number
of suitable applicants were shortlisted for an interview, and the process
had simply became a ritual, a procedure used to confirm each
applicant’s suitability. By 2011, up to 60 applicants were required to
be interviewed to fill an estimated 50 to 60 new tutor positions in 2012
(with around 30 existing tutors continuing). This ritual had also now
become very time consuming. With only 3 applicants interviewed per
hour, the process was requiring up to 20 hours to complete. The process
was clearly in need of change.
34 C.W. Sherwood & B. Littleboy
3. THE NEW SELECTION PROCESS
To widen the pool of applicants able to be interviewed, the overall GPA
threshold required was lowered from 6 to 5.5. This meant that almost
100 applicants were interviewed at the end of 2012. This was easily the
most applicants the School ever interviewed (about double the number
from 2011). To do so, the interview process needed to be completely
redesigned.
The new group-based interview process was designed so each
interview session was strictly limited to 20 minutes, followed by 10
minutes for staff to discuss the applicants. Each group interviewed
consisted of 3 applicants, and so 6 applicants could be processed in an
hour. Another room was used to conduct a concurrent interview,
thereby ensuring 12 applicants per hour could be interviewed. On any
one day, interviews were conducted for 1.5 hours, resulting in a total of
18 applicants being process per day. Therefore, by conducting
interviews on five different days across a two week period, up to 90
applicants were able to be interviewed.
The group-based interview involved organizing three randomly
assigned applicants to arrive at the interview and sit around a table. A
minimum of two applicants in a group was permitted if an applicant
failed to arrive, but a group of four was not allowed. Scribble paper and
pens were provided, and a whiteboard was also available for use by the
applicants. A chief interviewer welcomed the applicants and read out
brief instructions on what was required. The group’s task was to
collaboratively create a tutorial question from a randomly assigned
newspaper article presented to them within ten minutes.
We selected nine different newspaper articles prior to the interviews,
with each article of similar length and complexity. Some of the titles
were “Nowhere to run if you're on a menu”,1 “Hospital parking fees
enough to make you sick”,2 and “Easter holidays to deliver fuel price
hikes for motorists”.3 Each interview group was assigned an article so
as to reduce the chance that later groups received the same article.
When two different groups were being observed, in two different
rooms, both rooms used the same article.
Applicants were not forewarned of the newspaper article they would
be given or who would be in their group. They were advised of the
1 N. Mirosch, Courier Mail, 23 March 2011. 2 P. Mickelburough, Herald Sun, 17 March 2012. 3 A. Kelly, The Sunday Mail (Qld), 1 April, 2012.
Selecting Sessional Tutors 35
interview process and requirements, but they were prevented from
preparing or sharing model answers. The three applicants were in a
sense directly competing in the interview for limited positions.
However, those who dominated or tried too hard to impress were likely
to be assessed lower. Tutors need to work well with students, other
tutors, lecturers and course administrators. The process thus aims to
identify natural collaborators and facilitators. The selection approach is
based on the belief that good teaching requires both an individual ability
and teamwork.
During each interview, there would be the chief interviewer present
in all interviews, while other observers would assist the chief
interviewer. These observers typically consisted of another two
academics and one administrative staff, giving a total of four staff in
one interview. At the end of the collaborative 10 minutes working time,
each applicant was then given the opportunity to individually answer a
question asked by the chief interviewer. Each applicant had to answer
the same question. At least three questions were asked, with a different
applicant given the opportunity to answer each question first. The
questions involved each applicant reflecting on the interview process,
what they had learned, and how their experiences in the interview could
help them as a tutor. This question time was limited to 10 minutes. At
the end of these questions, the 20 minute interview was closed by the
chief interviewer, the applicants thanked for their contributions and
interest in the position, and the applicants left the room. The staff
members in the room then recorded a score for each applicant using a
scoring sheet with certain criteria.
There were five criteria used to score and rank the applicants,
namely:
i. appropriateness of their questions and answers;
ii. communication skills;
iii. interpersonal skills;
iv. provides evidence of encouraging student participation;
v. potential for being a tutor.
Each applicant was scored out of five on each criterion by each staff
member, giving a maximum possible score of 25. Both the observers
and chief interviewer provisionally scored each applicant
independently. Then the observers and chief interviewer discussed all
three applicant’s performance for no longer than five to six minutes.
36 C.W. Sherwood & B. Littleboy
The chief interviewer then determined and recorded a score for each
applicant taking into account the observers’ scores and comments.
Brief qualitative comments were also recorded on all marking sheets
for each applicant. In practice, it was found a score of 21 would see an
interviewee offered a position. This new tutor selection process was
aimed at eliminating applicants assessed as likely to perform below the
School’s high level of expectation. Many of the rejected applicants
interviewed would likely be good tutors, but competition is always
strong.
4. DATA COLLECTION
At the University of Queensland, the Teaching & Educational
Development Institute (TEDI) prepares and processes a paper-based
student evaluation survey on each tutor’s performance. This is done for
each tutorial at the end of every semester. The results are then made
available electronically to the School and a copy provided to individual
tutors several weeks later. These surveys are the key source of data
used. We collected data for each semester from semester 1, 2011 to
semester 2, 2013. TEDI’s Tutor Evaluation survey uses a 5 point Likert
Scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 =
strongly agree) to generate various TEVAL (Teaching Evaluation)
scores. These scores are an average of all student responses to survey
questions in a particular tutorial group. The survey asks students to rate
the extent to which:
Q1. The tutor was well prepared;
Q2. The tutor communicated clearly;
Q3. The tutor was approachable;
Q4. The tutor inspired me to learn;
Q5. The tutor encouraged student input;
Q6. The tutor treated students with respect;
Q7. The tutor gave helpful advice; and
Q8. Overall, how would you rate this tutor?
The vast majority of tutors typically take a minimum of two tutorial
groups each semester, with about 10 to 15 students per tutorial
completing each tutor evaluation at the end of the semester. Therefore,
a tutor employed for two semesters typically receives about 50
responses during a year to each survey question. It is Q8 score on the
TEVAL that typically attracts the most attention from administrative
Selecting Sessional Tutors 37
and teaching staff in assessing a tutor’s overall effectiveness and
performance. A Q8 TEVAL score of 4 is regarded as satisfactory and
deserving automatic re-appointment, subject to enrolment numbers, in
the following semester. Tutors with a score at or below approximately
3.8 each semester are categorised as underperforming. Prior to 2013,
tutors scoring below 4 were normally offered support or mentored to
help them to improve. However, if no improvement occurred, it seemed
these underperforming tutors remained in the system.
5. DATA ANALYSIS
We analyse the data in relation to each of the hypotheses outlined in the
Introduction.
Hypothesis 1: The new process increases the median
evaluation score.
In 2011, 407 tutorials were evaluated involving 96 different tutors. The
median number of responses per class was 12.5. In 2012, 367 tutorials
were evaluated with 98 different tutors. All tutors employed during
2011 and 2012 were interviewed using the old selection process. In
2013, 427 tutorials were evaluated with 96 different tutors consisting of
45 interviewed using the new process and 51 continuing tutors
appointed using old selection process. The median number of responses
per tutorial group in 2013 was 10.8.
Figure 1: Relative Frequencies of Q8 TEVAL Scores in
Various Cohorts of Tutors.
0%
5%
10%
15%
20%
25%
30%
35%
3.2 3.4 3.6 3.8 4 4.2 4.4 4.6 4.8 5
Rel
ati
ve
Fre
qu
ency
Q8 TEVAL Score
2011 old process 2012 old process
2013 new process 2013 old process
38 C.W. Sherwood & B. Littleboy
Using 2011 and 2012 as benchmark control years, and 2013 as the
intervention year, the distribution of Q8 TEVAL average scores for all
tutors, a measure of their overall effectiveness, is presented in Figure 1.
In reading Figure 1, consider the four bars shown where the Q8 TEVAL
score is 5. The four relative frequency values, for each year, have been
calculated using all tutors’ Q8 TEVAL average scores that were more
than 4.80 and less than or equal to 5.
Figure 1 shows the proportion of tutors and their overall effectiveness
score in a particular year. The proportions of tutors recruited using the
new interview process in 2013 (the solid black bar), who score at 4.4,
4.6, and 4.8, are clearly higher than the proportions where tutors were
recruited under the old interview process used in 2011 and 2012. In
addition, for these same scores of 4.4, 4.6 and 4.8, the proportions of
tutors recruited under the new process in 2013 are also clearly higher
than the proportion of tutors recruited under the old process and tutoring
in 2013. The distribution has shifted toward the right reflecting an
increase in the median evaluation of the median score of the tutors
recruited under the new process compared to the old process. This is
confirmed by the data presented in Table 1 using individual student
evaluation data, with the median Q8 score being 4.57 in 2013 (new
process) compared to 4.42 and 4.41 in 2011 and 2012 respectively using
the old process.
Table 1: Summary Statistics for Q8 TEVAL Scores and Different
Tutor Cohorts.
Cohort
Year
Interview
System Used
Number of
Tutors
Median
TEVAL
Score
Mean
TEVAL
Score
Standard
Deviation
2011 Old 96 4.42 4.36 0.32
2012 Old 98 4.41 4.36 0.39
2013 New 45 4.57 4.49 0.27
2013 Old 51 4.48 4.41 0.40
It is noted that our tutors under the old system had a reputation for
being high quality and performing well. Indeed, a median score of 4.4
for almost 100 tutors would generally be considered impressive. It was
anticipated that it could be difficult to observe any clear improvement.
Selecting Sessional Tutors 39
Figure 2: Cumulative Frequency Distribution (% Cohort with
Q8 TEVAL Score at a Particular Value or Better).
Indeed, there are about 8% more tutors scoring at 4.8, recruited under
the new process in 2013, compared to the next best cohort of old-system
tutors (also 2013). At scores of 4.4 and 4.6 this margin is just less than
5% and 3% respectively. Similarly, at the highest scores of 4.8 and 5.0,
the new system cohort tutors are not appreciably superior. Indeed, these
old process tutors outperform the new process tutors in 2013. This
likely reflects the fact that better performing tutors are more likely to be
retained over time.
An alternative approach to evaluate the impact of the intervention is
presented in Figure 2. This allows for comparison of the cumulative
frequencies of Q8 TEVAL scores of tutors received from students. In
reading Figure 2, the cumulative frequencies for each year represent the
proportion of tutors scoring at a particular score or better. For example,
in 2013 using the new process, for a Q8 TEVAL score of 4.4, 91% of
tutors scored this value or more.
The scores for tutors recruited using the new process in 2013 are
shown, along with the scores for tutors from the old system employed
in 2013, 2012, and 2011 to benchmark against. What this reveals is that
91% of scores using the new interview process in 2013 were 4.4 or
better. This compares with only 79% of scores in 2013 from the old
process, and 71% and 69% of scores using the old process in 2012 and
2011 respectively. A cohort of tutors with 91% scoring at a TEVAL
score of 4.4 or better is considered to be excellent. It is rare to encounter
0%
20%
40%
60%
80%
100%
120%
3.2 3.4 3.6 3.8 4 4.2 4.4 4.6 4.8 5
Cu
mu
lati
ve
per
cen
tag
e o
f co
ho
rt
Q8 TEVAL Score
2011 old process 2012 old process
2013 new process 2013 old process
40 C.W. Sherwood & B. Littleboy
individual tutors able to achieve scores of over 4.8, let alone 5.
Moreover, when appointing around 100 tutors per year, it would be
extremely difficult to devise any selection process able to identify
sufficient tutors to reach such high TEVAL scores. However, the
incremental improvements that have been observed, where the
proportion of tutors scoring 4.4 has increased from 70% to over 90%,
has effectively shifted the quality of the School’s tutors to a higher
level. The median TEVAL increased from around 4.4 to almost 4.6,
which is a change from very good to outstanding. This improvement is
also reflected in the lower standard deviation value reported in
Table 1 for the new process compared to any value obtained under the
old process.
Hypothesis 2: Good communication skills are essential in being a
good tutor.
Over many years, student written and verbal feedback on tutor
performances have indicated students decided not to attend tutorials
because they simply could not understand what their tutor was trying to
say. This could result from not being able to clearly communicate
technical content in meaningful ways as well as not speaking clearly.
To evaluate the link between communication skills and being a good
tutor, TEVAL data relating to Q2 and Q8 were analysed. It revealed
that the correlation coefficient between communication (Q2 TEVAL)
and overall effectiveness as a tutor (Q8 TEVAL) was 0.91 (n= 96), 0.91
(n=98) and 0.94 (n=96) for tutors appointed in 2011, 2012, and 2013
respectively. There is clearly a strong positive correlation between a
tutor’s ability to communicate and their overall effectiveness score.
The new interview process pays particular attention to an applicants’
communication skills. Indeed, the five criteria used in the new selection
process in essence revolve around communication. Figure 3 shows the
relative frequency distribution of Q2 TEVAL scores (communication
ability) for the benchmark year of 2011, using the old process, and 2013
using the new process. This clearly shows that the proportion of tutors
having a Q2 TEVAL score from 4.2 to 5 is higher using the new process
compared to the old process. Figure 4 presents an alternative to
appreciate the impact on the new process on tutors’ communication
skills using a cumulative distribution. This reveals that more than 80%
of tutors have a Q2 TEVAL score of 4.4 or higher using the new process
compared to 65% using the old process. This again reflects an increase
in the median Q2 TEVAL scores, and is captured by the descriptive
Selecting Sessional Tutors 41
Figure 3: Relative Frequencies of Q2 TEVAL Scores for
Tutors in 2011 and 2013.
Figure 4: Cumulative Frequency Distribution (% Cohort
with Q2 TEVAL Score at a Particular Value or
Better).
statistics presented in Table 2. Presumably, it would be desirable, other
things equal, to have tutors with higher communication skills, even if
student evaluations of tutor overall effectiveness were unaffected.
Hypothesis 3: An applicant’s GPA is a necessary but not sufficient
condition for being an outstanding tutor.
0%
5%
10%
15%
20%
25%
30%
35%
3.4 3.6 3.8 4 4.2 4.4 4.6 4.8 5
Rel
ati
ve
Fre
qu
ency
Q2 TEVAL Score
2011 old process 2013 new process
0%
20%
40%
60%
80%
100%
120%
3.4 3.6 3.8 4 4.2 4.4 4.6 4.8 5Cu
mu
lati
ve
per
cen
tag
e o
f co
ho
rt
Q2 TEVAL Score
2011 old process 2013 new process
42 C.W. Sherwood & B. Littleboy
If GPA could be found to be strongly and positively correlated with
tutor performance, then the interview process arguably could be
discarded. GPA data on tutors recruited for semester 1, 2013 for both
their overall program to the end of 2012, and their GPA on only the
economics courses they had completed were compiled. These were
compared with their semester 1 Q8 TEVAL and Q2 TEVAL scores.
The details of the correlation analysis is shown in Table 3. This reveals
that the tutor’s overall GPA is a weak predictor of a tutor’s overall
effectiveness (Q8) and communication skills (Q2). Further, the tutor’s
GPA for economics-coded courses is even a less reliable predictor.
Table 2: Summary Statistics for Q2 TEVAL Scores and Different
Tutor Cohorts.
Cohort
Year
Interview
System
Used
Number of
Tutors
Median
TEVAL
Score
Mean
TEVAL
Score
Standard
Deviation
2011 Old 96 4.37 4.30 0.38
2013 New 45 4.46 4.42 0.36
Table 3: Correlation Analysis for New Tutors who Started in 2013 (n=40).
Program GPA ECON GPA Q8 TEVAL
Average
Q2 TEVAL
Average
Program GPA 1
ECON GPA 0.74 1
Q8 TEVAL
average
0.22 0.07 1
Q2 TEVAL
average
0.25 0.11 0.93 1
Table 3 reveals there is only a weak positive correlation between both
program GPA and Economic (ECON) GPA scores with a tutor’s Q8
TEVAL score (0.22 and 0.07) and Q2 TEVAL score (0.25 and 0.11).
For tutors newly employed during 2013, their GPA for the economic
courses completed was also obtained, and it revealed that GPAs for
economics courses are even less important than the overall GPA. (There
Selecting Sessional Tutors 43
were 5 tutors appointed in 2013 whose evaluations were not included.
Either they were not currently economics students or their program
GPAs were not available).
This analysis suggests there is no strong evidence for selecting tutors
based on GPA alone. Instead, the much stronger correlation between
the Q8 TEVAL (effective tutors) and Q2 TEVAL (communication)
scores of 0.93 suggests applicants’ communication abilities are a key
element needed as a second filter of applicants. A first filter, using a
GPA threshold as an indicator, still needs to be applied to at least ensure
the applicant knows the course material.
These findings overall are quite stark, but it is improper to move from
finding correlations to attributing causality. Here, however, there is a
simple and intuitive connection between cause and effect. Good
communication is not a controversial attribute of a good tutor.
Furthermore, there is quite a deal of overlap between the interview’s
selection criteria and the criteria on the evaluation forms, which is not
accidental. However, to claim that causality flows from the new
selection process to an improvement in tutor evaluations is warranted
only if other factors may reasonably be regarded as constant. (Clearly,
reverse causality is not an issue here as the prior decision consciously
to change the process was made independently). Other factors appear
to be substantially the same. There are no grounds to suppose that there
has been a sudden and inexplicable improvement in the quality of the
applicant pool. (An administrative officer confirms that the overall
profile of the applicants appears comparable over the last few years).
Neither is there anecdotal evidence of a sudden and mysterious rise in
the students’ appreciation of tutors of the kind selected and trained by
the School. We do not currently have access to data about survey
evaluations of tutors in other Schools, so we cannot gauge relative
performance or test whether numerical evaluations are somehow
framed by experience elsewhere.
Neither has the composition of the student intake suddenly changed.
Although tutorial attendance has declined a little, it is implausible that
the drop-outs had a particular distaste towards tutors of the kind
selected. (Our results should be viewed against the background of
declining average class-attendance rates across the campus, perhaps
reflecting improved internet access or the need to earn income). Tutor
training has also evolved over the period, but it has not been
transformed. The results are so clear and immediate that it is reasonable
44 C.W. Sherwood & B. Littleboy
to attribute the improvement in survey outcomes to the new tutor-
selection process. It is difficult to improve on something that is already
operating very well, but overall improvement has occurred. It may be
possible to move our tutors to an even higher level.
6. WEIGHING THE COSTS AND BENEFITS
Change requires that the extra benefits are expected to exceed the extra
costs. The new process appears no more costly in terms of material and
administrative resources. On the cost side, there are small additional
financial costs (such as photocopying of newspaper articles), and there
are extra time costs in preparing the interview activities, collating the
paperwork and in recruiting (voluntary) academic observers.
Comparing the new and old interview systems, there is some extra
administrative load, but the total commitment in time, once the
procedures are settled, is about the same and possibly less. There is
more paper per applicant to collate and store, but there is also a better
trail of documentation that justifies the decisions reached. The greater
rate of throughput means that junior administrative staff can better
manage the flows of people who attend the sessions. A senior
administrative officer no longer attends the interviews, a considerable
advantage. Instead of interviews taking at least 15 minutes per
applicant as under the old system, they take 10 minutes per applicant
under the new. Quicker processing of applicants has permitted
interviewing more diverse applicants. Several students with a high
GPA, who would likely have done well under the old process, did
poorly under the new process.
There has been an increase in the time devoted by academics, but this
involves not only a time cost but also diverse benefits to the volunteer
that were not anticipated. Academic staff had many conversations at
the end of each interview session about the strengths and weaknesses
of the applicants. In particular, in a School that currently has many new
academic appointments, this was an opportunity for real collegiality.
Academics, beyond those teaching large first year courses, saw how
well or poorly their star students in later year courses performed. This
prompted reflection on how they designed, prepared and delivered their
own courses.
The sessions are far more unpredictable and engaging for the
academic staff than the former, more bureaucratic, method. Applicants
soon realised that questions from others required them to respond with
intelligent answers framed creatively in the specific context. The
Selecting Sessional Tutors 45
experience for the academic staff is unlike being dragooned to form part
of an old-style interview panel. During November 2013, 22 academic
staff (3 professors, 6 associate professors, 3 senior lecturers, and 10
lecturers) and 3 administrative staff participated. This is more than
eight times the number of academics involved in the old process.
Previously, it was also unlikely for any professors or associate
professors to have been involved, especially if they were not lecturing
in the large courses requiring tutors. Instead, coordinators in the first-
year courses ranked the applicants. This was often regarded as one of
their administrative duties as well as being in their interests to monitor
the quality of the intake. Having academic staff acting as observers
during interviews has clearly encouraged increased staff involvement.
We hope that positive experiences by academic staff may also result in
wider participation during our tutor training (and induction) day, and
perhaps to some revision of their own teaching aspirations and
practices.
Observers quickly gain experience as to how to evaluate the
applicants. In turn, this allows them to be promoted to chief
interviewer, which means the system tends naturally to spread the
workload over time. Promotion of observers to chief interviewer
occurred for interviews conducted at the end of 2012 and 2013.
Previous observers volunteered for the role of chief interviewer, simply
for the experience, on three separate occasions. The transition was
easily implemented and the observer was keen to participate in the role
again.
Importantly, the new tutor selection process also gives a head start to
subsequent tutor training. Applicants are given practical experience in
group work during the interview process. We confirmed what we
expected: as students, most applicants had not undertaken anything like
the interview activity during their classes. They realised how valuable
it can be as a learning experience (or had the good sense to say so). In
addition, academic staff observing the process also realised the
potential of using such activities in their classes and tutorials. If similar
group exercises were embedded in tutorials, then key graduate
attributes, notably an ability to apply theory to given problems, would
be demonstrably embedded in the program. One applicant observed
astutely that the interview set-up reversed the more standard teaching
process of using an example to fit a given concept.
46 C.W. Sherwood & B. Littleboy
The sessions are open to wider scrutiny and are observably arm’s-
length. In former practice, interview performance has not been the
exclusive criterion for tutor selection. Applicants who were Honours
or PhD students received automatic preference. (Honours students are
required to add an extra year to their three-year undergraduate
program). Positions are no longer offered on the basis of student
category. Those interviewees that ‘fail’ (score below about 20.5) are
not offered a tutoring position in the large first-year courses.
Supervisors of Honours, Masters or PhD students have no scope to
lobby or alter the rankings if their protégé does not ‘pass’ the interview.
They may, however, hire them for their own courses and accept
responsibility for their performance.
Some may believe it is more reliable having applicants assessed by
the same interview panel and under conditions as close as possible to
identical. However, any advantages offered by uniformity in this
selection process may be overstated. Uniformity can lead to a
predictable selection system meaning applicants can more readily
simulate the attributes desired. All sessions in the new selection process
are conducted according to the same protocols. However, what varies
are the newspaper articles, questions and knowledge applicants must
draw upon, and the combination of observers in each session. The
observers provide moderation. Almost all interviews are currently
conducted by only two chief interviewers, so basic consistency is
readily established.
Applicants with poor language skills or shy dispositions tend to fall
in the rankings. Selecting for desired attributes (interpersonal,
organisational and communication skills) still results in a gender-
balanced and ethnically diverse range of tutors. Note that a more
diverse range of staff involved in a more open selection process is also
likely to guard against interviewer bias.
7. REFLECTIONS AND CONCLUSIONS
The new process is quicker, more engaging for a wider range of
participants, and it delivers better results. Although tutor performance
is multi-factorial, which can make reliable statistical associations
difficult to identify, the net benefits of switching to the new process of
selection seem clear. When resources for training are scarce, it makes
sense to select those who are more likely to emerge as good tutors.
Since a tutor position is essentially the bottom rung of the academic
ladder, and appointments are increasingly on a casual basis of payment
Selecting Sessional Tutors 47
per hour, university administrators and academic staff perhaps pay
rather less attention to ranking applicants to perform tasks widely
regarded as being less skilled and of low financial value. Risks of
deterioration in the quality of the tutorial experience need actively to be
countered by devoting greater care to tutor selection.
Those entrusted to decide appointments should use criteria that
withstand inspection. Selection criteria need to align with goals. If the
goal of providing tutorials is remedial and intended for students in
difficulty, then patience and clarity in a tutor are at a premium.
Guidance during laboratory or practical work calls on different skills
and conducting fieldwork still others. There appears to have been little
published reflection, and still less empirical work, done on framing
selection practices to suit whatever learning or motivational goals are
assigned to tutorials.
A potential objection to a more targeted selection process is that we
should be more neutral about traits and should not select on too narrow
a range of preconceived criteria. A portfolio of different types of tutors
may be appropriate, at least in high-enrolment courses in which means
exist for students to migrate towards tutors with qualities they find
congenial. So far as we know, however, Australian universities do not
yet provide quiet tutors for shy students or run designated bilingual
classes specifically for international students, but this day may come.
Change requires that the extra benefits are expected to exceed the extra
costs. The new process appears no more costly in terms of material and
administrative resources. On the cost side, there are small additional
financial costs (such as photocopying of newspaper articles), and there
are extra time costs in preparing the interview activities, collating the
paperwork and in recruiting (voluntary) academic observers.
REFERENCES
Angrist, J.D. and Guryan, J. (2004) “Teacher Testing, Teacher Education
and Teacher Characteristics”, American Economic Review, 94 (2), pp.241-
246.
Becker, W.E. (1997) “Teaching Economics to Undergraduates”, Journal of
Economic Literature, 35 (3), pp.1347-1373.
Bianchini, S., Lissoni, F., and Pezzoni, M. (2013) “Instructor Characteristics
and Students’ Evaluation of Teaching Effectiveness: Evidence from an
Italian Engineering School”, European Journal of Engineering
Education, 38 (1), pp.38-57.
48 C.W. Sherwood & B. Littleboy
Bosshardt W. and Watts, M. (2001) “Comparing Student and Instructor
Evaluations of Teaching”, Journal of Economic Education, 32 (1), pp.3-
17.
Korur, F. and Eryilmaz, A. (2012) “Teachers’ and Students’ Perceptions of
Effective Physics Teacher Characteristics”, Eurasian Journal of
Educational Research, 46, pp.101-120.
Ragan, J.F. and Walia, B. (2010) “Differences in Student Evaluations of
Principles and Other Economics Courses and the Allocation of Faculty
across Courses”, Journal of Economic Education, 41 (4), pp.335-352.
Australasian Journal of Economics Education
Volume 13, Number 1, 2016, pp.49-55
BOOK REVIEW
Review of Salemi M.K. & Walstad W.B. (eds.) (2010), Teaching
Innovations in Economics: Strategies and Applications for
Interactive Instruction, Cheltenham, UK & Northampton MA,
USA: Edward Elgar, 274 pp + xiii.
Peter Docherty
Business School Economics Group,
University of Technology, Sydney
Teaching Innovations in Economics is an extremely useful book that
every thoughtful economics instructor ought to read. It is a set of
collected essays produced from the Teaching Innovations Program
(TIP) initiative of the American Economic Association’s Committee on
Economic Education between 2004 and 2010. As explained in Michael
Salemi’s opening chapter, this program was taught by a number of
leading economic educators in the United States, and was aimed at
enhancing the teaching skills of university and college instructors. It
thus represented a type of professional development program for these
instructors. Salemi describes its three part structure as made up: firstly,
of a set of ten face to face workshops that explored a range of non-
traditional pedagogical strategies; secondly, of a set of follow-up, on-
line modules completed when participants returned to their respective
universities; and thirdly, of projects that applied one or more of the
pedagogical strategies considered in the face to face and online sessions
to participants’ own classrooms.
The first three chapters of the book explore the TIP initiative in
further detail. Salemi’s opening chapter provides some background to
the initiative, describes its overall objectives, and outlines the structure
of the workshop phase. Mark Maier and Tisha Emerson’s second
chapter examines the on-line phase of the program and describes how
it built upon the workshops that participants had attended in phase one.
KimMarie McGoldrick’s chapter then discusses the program’s project
phase but she does this within a broader discussion of the scholarship
50 P. Docherty
of teaching and learning in economics. The last chapter of the volume
by William Walstad reflects back on the program at its conclusion and
carefully considers participant feedback. The intervening seven
chapters, all authored by program participants, report on a variety of the
projects conducted during stage three of the program. They include such
initiatives as cooperative learning, use of experiments in the classroom,
discussion techniques, formative assessment strategies, context rich
problems and case studies, and techniques designed for large classes.
One of the most valuable features of the book is its detailed description
and evaluation of these techniques. Each of these chapters provides a
rationale for the initiative described, a detailed description of the
initiative itself, and some kind of evaluation of or reflection on the
effectiveness of the strategy.
A number of the chapters on specific pedagogical strategies are
particularly interesting. Two addressed the issue of involving students
in class discussion, one of the most obvious strategies for increasing
student engagement. These were Chapter 6 entitled Classroom
Discussion by Michael Salemi, Kirsten Madden, Roisin O’Sullivan and
Prathibha Joshi, and Chapter 9 entitled Case Use in Economics
Instruction by Patrick Conway, Derek Stimel, Ann E. Davis and
Monica Hartmann.
In the first of these, Salemi et al. make a distinction between
structured and unstructured discussion. Structured discussion requires
students to read an article or book chapter prior to the class and answer
a series of questions which focus on either some factual dimension of
the reading or a key concept developed in the reading. Students are then
asked to contribute to answering these questions in class and the
instructor withholds any evaluative comment at least until towards the
end of the class to see how students react to one another’s answers.
Alternatively students may be required to interpret the article and
explain their interpretation. Unstructured discussion on the other hand,
also requires students to read an article in advance of the class but poses
a smaller number of general questions, and requires the instructor to
provide more context and guidance as the discussion progresses.
Three case studies that employed various discussion techniques and
collected feedback data from students are provided by the authors of
this chapter, reflecting their experiences in History of Economic
Thought, Principles of Macroeconomics and Intermediate
Macroeconomics classes. They conclude that discussion techniques are
Book Review - Teaching Innovations in Economics 51
less preferred by more junior students, especially those in first year, but
that exam performance is enhanced compared to classes where lecturing
is the only pedagogical technique employed. They also find that student
feedback improves as discussion is used more frequently and students
learn how to engage with it more effectively.
This chapter is useful because it formalises one of the most basic but
productive pedagogical strategies for moving teachers from simply
lecturing, towards a mode of delivery that more actively involves
students in classroom learning. William Becker has consistently argued
that such a movement needs to be one of the highest priorities for
enhancing university economics instruction, both because active
engagement of students is more educationally effective than simply
lecturing, and because economists under-utilise active techniques in
favour of lecturing (see, for example, Becker 1997, pp.1354-55; and
2000, p.113). Classroom discussion is a technique that some of us use
somewhat intuitively; in fact the description of structured discussion
provided by Salemi et al. sounds remarkably like the tutorials of
between 10 and 15 students I taught in first year economics at the
University of Sydney at the beginning of my career. To some extent one
may wonder whether this approach is really that innovative. But by
formalising the technique, Salemi et al. also extend it, because
formalisation allows us to analyse how the method works and thus to
consider possibilities for variation and adaptation that have the potential
for even more interesting classroom experiences.
The chapter on Case Use in Economics, extends the range of
possibilities for classroom discussion even further, but also relates these
possibilities to the additional idea that what one discusses is important.
Conway et al. argue that discussing real world phenomena provides
additional motivation for students who seem to care about the
applicability of the ideas they encounter in university courses. The case
method, used so widely in management education, enables the
instructor, they suggest, to bring together active student involvement in
class discussion, a consideration of real economic phenomena, and
greater use of formative assessment structures that provide on-going
feedback to students as the case unfolds.
I have personally been convinced of the value of the case method in
economics for some time, partly because of the number of MBA
students taught by my department and partly because of the career-
focused undergraduates that my university attracts. The value of
52 P. Docherty
looking at real cases is also supported philosophically since the
objective of economics is to explain actual economic phenomena. Even
high theory, for which I had a passion as a graduate student, is
ultimately of value only if it provides insight into the nature of
economic reality. Teaching students good theory and using that theory
to examine actual economic problems, whether of a business or a policy
nature, does not seem to me a difficult proposition to support, whoever
are the students under consideration. The Conway chapter is thus one
of the most useful chapters in the book from this perspective. If I have
any criticism, it is that the particular cases chosen by the authors do not
go far enough in terms of realism. The case reported to have been
employed in a macroeconomics subject of whether hypothetical George
should accept baseball cards as payment for washing a car, while
perhaps highlighting the essential features of money, is nonetheless a
relatively trivial and overly individualised case for macroeconomics, in
my view. Something like an examination of whether the FOMC should
purchase more securities in the face of continuing weakness in the U.S.
economy after the Global Financial Crisis (see Whiting 2006 for a case
along these lines) would be much more realistic.
The chapter by Mark Maier, Joann Bangs, Niels-Hugo Blunch and
Brian Peterson on Context-rich Problems to some degree unpacks the
value that active student engagement and real world problems have for
learning. This chapter argues that providing students with a well-
developed context that requires them to sift what is important and what
isn’t from the information at their disposal, and to use economic
principles in this identification process, helps them to understand these
principles more thoroughly and also develops application skills they
will use after graduation. The chapter’s emphasis, however, is mainly
on hypothetical scenarios designed to highlight the particular concepts
under consideration. My feeling, once again, is that while such
techniques may be more interesting and productive than simply
lecturing the textbook model, looking at real problems has an
authenticity and integrity that is hard to beat.
The chapters discussed so far tend to focus on the primary means
used to have students confront economic ideas that are new to them.
The chapter by William Walstad, Michael Curme, Katherine Silz
Carson and Indradeep Ghosh examines the question of how assessment
structures can be used formatively as part of the learning process rather
than simply summatively to evaluate post-facto what students have
Book Review - Teaching Innovations in Economics 53
learned (Ramsden 1992, p.184). A range of methods and course
contexts is explored in this chapter for using formative assessment
methods from in-class quizzes in microeconomic principles to short
assignments in econometrics, to multiple writing assignments in
monetary economics.
But a curious and disappointing shortcoming characterises this
chapter. A fundamental dimension of the distinction between formative
and summative assessment is that formative assessment provides
students with feedback that they may use to modify their understanding
of the material. A well-structured formative assessment regime also
gives students the opportunity to use this modified understanding in
further assessment tasks (Sadler 1998). The nature and quality of
feedback is thus central to the effectiveness of formative assessment as
a teaching/learning device. But this point is almost completely absent
from the chapter. In the opening section that lays the conceptual
foundations for the chapter, the definition of summative assessment
offered focuses only on its ex-post dimension and not on the more
central dimension of taking stock of student knowledge or skill
development. In describing formative assessment, the concept of
feedback is raised, but it is principally feedback received by the
instructor about the student (an essentially summative idea) that the
instructor may then use to modify his or her approach later in the course.
While this is certainly a useful aspect of formative assessment
structures, it is not the kind of feedback that lies at their heart, and
careful consideration of feedback to students on their learning is
essentially absent from the opening section.
To a large degree the same neglect characterises each of the
assessment initiatives described in the rest of the chapter. The test-retest
strategy of the microeconomic principles course involved a test on day
one before anything in the course had been learned at all, and a retest
on the final day. This seemed like a more elaborate summative
procedure that tested the increment to student knowledge developed
across the course rather than a formative strategy, although the group
project of looking at one of the questions in the pre-test did move in the
right formative direction. Little information about the nature of
feedback from the pre-test was provided in the discussion of the
initiative. The regular in-class testing in the econometrics initiative may
well have provided effective formative feedback if the final assessment
was properly aligned with these earlier assessments and feedback drew
54 P. Docherty
student attention to the knowledge or skills that were the focus of this
alignment, but insufficient information was provided to determine
whether this was the case. The most interesting assessment design was
the monetary economics writing assignment where students had three
opportunities to write across two assignments and a seemingly aligned
final exam. Here again, the nature of feedback to students on the earlier
writing opportunities and the ability of students to feed learning forward
into the next assessment task did not appear to be an important focus of
the authors’ discussion of this initiative. In the end, it was difficult not
to be disappointed with this chapter.
By contrast, it was good to see a chapter on interactive approaches in
large class teaching and the various suggestions offered by Gail Hoyt
and her co-authors provide a useful guide to making this challenging
teaching assignment more productive for the student and more
enjoyable for the instructor. I was also convinced by Denise Hazlett and
her co-authors that class experiments can be extremely effective for
teaching microeconomics in particular, because the class becomes a
micro-market and the behaviour under investigation is more directly
observable.
A set of questions, however, ought to be posed and carefully
considered in every course where class experiments are deployed.
These questions involve: the degree to which the Hawthorn effect
operates in such circumstances and whether the contrived nature of the
experimental micro-market potentially alters behaviour; a discussion of
what it actually is that is being directly observed, and what assumptions
are being explicitly made about such observations; and a consideration
of the differences that real economic institutions and contexts might
make to the behaviour in question.
I also found KimMarie McGoldrick and her co-authors’ chapter on
Co-operative Learning useful, especially the section outlining the key
elements that should characterise an effective design of this strategy.
The ability to work co-operatively is clearly a skill that most business
students will need after graduation, but designing effective group work
is a bane for many business school instructors. The set up cost for such
approaches to teaching is unfortunately high but the potential for much
more interesting student learning experiences is also significant, and
having some direction for shaping these approaches that avoids wasting
time on less effective strategies is an extremely useful contribution of
the McGoldrick et al. chapter.
Book Review - Teaching Innovations in Economics 55
Overall, Salemi and Walstad’s book is a useful contribution to
thinking about economics education. The range of pedagogical
techniques considered, the rationales outlined, and the evidence
collected from student feedback and performance on many of the
initiatives trialled, provide useful stimuli for university teachers of
economics to reflect upon and to use in designing their own approaches
to teaching.
REFERENCES
Becker W.E. (1997), “Teaching Economics to Undergraduates”, Journal of
Economic Literature, 35, September, pp.1347-73.
Becker W.E. (2000), “Teaching Economics in the 21st Century”, Journal of
Economic Perspectives, 14 (1), pp.109-119.
Ramsden P. (1992) Learning to Teach in Higher Education, London:
Routledge.
Sadler D.R. (1998), “Formative Assessment: Revisiting the Territory”,
Assessment in Education: Principles, Policy & Practice, 5 (1), pp.77-84.
Whiting C. (2006), “Data-based Active Learning in the Principles of
Macroeconomics Course: A Mock FOMC Meeting”, Journal of Economic
Education, 37 (2), pp.171-177.