prof keithia wilson - flinders university keithia wilson griffith university 2010 altc national...

Prof Keithia Wilson Griffith University

2010 ALTC National Fellow for the FYE Program Director for the FYE, Griffith Health

2007 Australian University Teacher of the Year

Prof Keithia Wilson ALTC National Fellow


Acknowledgment to Country In the Spirit of Reconciliation Following on from Sorry Day I would like to acknowledge & honour

the Traditional Custodians of this land that we are meeting on today, the Kaurna People, and pay respect to their Elders past & present

Overview 1. Principles & practices for enabling commencing

students success with FY Assessment Designing assessment Preparing students to undertake assessment Marking Assessment Post-Assessment feedback process to enable & support

learning 2. Strategies for academic recovery with at-risk

commencing students


Overview 1. Current Context for the FYE 2. Purposes of FY Assessment & Student

Motivations 3. The FY Assessment Lifecycle from the

perspectives of – Students and Staff 2. Principles & Models for enhancing effective

Practice with FY Assessment


Information Sources

The First Year Experience & Assessment literature generally

Findings from a recent ALTC Grant on First Year Assessment Practice (2009-2010)

Findings from a current ALTC National Fellowship on the FYE working with FY teaching teams (2010-2012)

Focus on the student voice & understanding the student experience of assessment


Griffith University context • Large metropolitan university in Brisbane (1 of 7 in S-E

Qld, & 1 of 4 in Brisbane) • Multi-campus - 5 campuses x 60 k corridor • Student enrolment of 42,000 • 70% of students are first-in-family at uni - FIF correlates

with low SES & lower entry scores/OPs • 6th largest low SES student intake in Australia • 3rd largest Indigenous student intake in Australia • 25% International student enrolment • 10% students studying in distance mode


Step 1 – Context

How can we improve the assessment process and experience for commencing students? Understanding the current context for the FYE


Federal Gov’t reform agenda in Higher Education

Aims to – Widen student participation in Higher Education – A FAIR GO! Increase the access of students from low SES & disadvantaged

backgrounds to university (higher numbers - Target increase from 12% to 20% by 2020)

Increase the success of students from low SES & disadvantaged backgrounds at university (higher retention)

Moving from an elite model of HE (0-15% population participation) to a mass model of HE (16-50%) (Trow, 2004)

25% participation in Australia (50% USA) 1.3% of Indigenous Australians attend university


Evolution of Approaches to FY Orientation & Engagement & the FYE First Generation Strategies = Co-Curricular - A focus on

designing FYO&E supplemental activities & strategies which are outside of the classroom(e.g., Orientation, peer mentoring)

Second Generation Strategies = Curricular - A focus on enhancing FY curriculum design, pedagogy & assessment practices

Third Generation Strategies = Whole-of-Institution - A focus on an Institution wide approach to 1st & 2nd generation strategies, with practice standardisation & QA mechanisms for continuous improvement

= Whole-of-School/Program - A focus on the strategic combination of 1st & 2nd generation strategies for a particular disciplinary context (School or Program)


Understanding the Current Context for the FYE

Meta Model 1 - FY Transition Practice Student Diversity Student Transition * Course Design * Course Delivery * Course Assessment


Student diversity

So what is important to understand about student diversity? Defining student diversity Assumed knowledge


What is Student Diversity? Traditional Students

(TS) medium-high SES second generation higher entry levels full time on-campus

Elite Model

Non-Traditional Students (NTS)

low SES first-in-family lower entry levels full-time & working not on-campus much Indigenous NESB, International, refugees disability home care responsibilities from rural & remote settings

Mass Model


What are assumed knowledge & capabilities? Academic Skills Information Literacy Computer Literacy Reading Skills Written Communication Numeracy Skills Critical thinking & analysis Independent Learning (self-regulation) viz. time on

task, self-study, time management, uni-work-social life balance, successful student behaviour


What are assumed knowledge & capabilities? Academic/Cultural Capital – the “Hidden Curriculum” 1. Student role Understanding student role expectations &

appropriate & successful behaviour 2. Performance Expectations Reading the academic context

to accurately determine performance requirements regarding studying & assessment

3. Help-seeking Capacity for help-seeking without fear of negative labelling (dumb/stupid)

4. Identity as a student Sense of belonging & personal fit with university (overcome the “outsider within” phenomenon – “A stranger in a foreign land”)


Are NTS capable of being successful at university? The national research evidence shows - The success rate (or tendency to pass their year’s

subjects) of low SES students is 97% of the pass rates of their medium & high SES peers & has been stable over the last 5 years (Bradley et al, 2008:30)

However, they require higher levels of support to succeed e.g., financial assistance, academic support, mentoring & counselling services (Commonwealth of Australia, 2009)


Therefore....implications for assessment practice Taking account of assumed entry level knowledge in

unit content & assessment design & making this explicit to students

Building foundational academic skill development into unit content & assessment design

Developing commencing students skills as independent, self-regulating learners across the first year (shared responsibility between students & staff)

Creating a unit & program level culture that respects & values diversity in all of its forms (age, gender, race, socio-economic status etc.)


Student Transition So what is important to understand about Student

Transition? An evidence-based model for conceptualising student

transition Senses of success (Lizzio, 2006) Predictors of student success


Meta-model 2: The ‘Five-Senses’ of Student Success

Sense of Student Identity

Sense of Connectedness

Sense of Capability

Sense of Purpose

Sense of Resourcefulness


(Lizzio, 2006)

What predicts commencing students’ satisfaction with their degree program?

Sense of Purpose

Sense of Capability

Good Teaching

Sense of Connection

Perceived Effectiveness of Orientation

Strongly Enhances

Enhances

Enhances

Enhances

Enhances

Time on Task Enhances

Commencing Student

Satisfaction


What predicts commencing students’ academic outcomes?

Semester 1 Academic

Results -GPA

Academic Capital Low Socio-economic status

First in Family English as Second Language

Competing Demands Time in employment

Time as carer

Prior Academic Achievement Entry Scores (OP)

Task Engagement @ Uni Attendance at Orientation

Time on task Strongly Enhances

Enhances

Reduces

Reduces


What predicts commencing students’ retention?

Semester 1 Academic

Results-GPA

Academic Capital -

Competing Demands -

Prior Academic Achievement +

Task Engagement @ Uni +

Sense of Purpose + + +

Student Satisfaction +

Student

Retention into yr 2


Therefore....Implications for practice Curriculum strategies for developing time on task &

self-regulation skills assists assessment success Strategies for enhancing sense of purpose in

curriculum & assessment practices Building academic & social capital through unit &

assessment design throughout the FY Effective Orientation Programs & early student

engagement strategies to encourage attendance at Orientation


Step 2 – Assessment Principles & Practices So...what do students say about their

experience of assessment?

A quick overview of some salient

research evidence!


Assessment characteristics which positively influence student learning & engagement

1. perceived validity of assessment tasks & approaches 2. perceived extent to which the learning environment

is empowering 3. perceived fairness of the learning environment,

especially with assessment tasks & practices


Students’ perceptions of the validity of assessment tasks & approaches (Sambell et al., 1997)

3 sets of priorities - 1. Educational values – authentic/meaningful tasks,

perceived to have long term benefits, applying knowledge

2. Educational processes – reasonable demands, encourages independence by making expectations clear

3. Educational consequences – rewards effort and breadth & depth in learning


Students’ meta themes in assessment (Savin-Baden , 2004)

2 forms of student disempowerment: 1. Unrewarded learning – the relationship between

quantity of work & its weighting 2. Disabling assessment mechanisms including: Processes – lack of information & inadequate

feedback Forms – assessment methods that do not fit with

espoused forms of learning


Students’ perceptions of fairness (Lizzio, Wilson & Hadaway, 2008)

Strongly influenced by – Extent to which they feel personally respected by

academic staff (convenors & sessional staff) in the learning & assessment process – relational culture Adequacy of the information & support systems

provided for them to “do their job” in relation to assessment


The Reality of Assessment for Commencing Students

Student feedback from multiple sources indicates: Strategic nature of Assessment for student

engagement, success & retention For many students assessment IS the learning Assessment items which are too difficult, not

meaningful, and not clearly explained are a key cause of student drop-out in the first semester

Well designed & implemented assessment is key to student engagement, learning & retention


Assessment Practice in the First Year So what is important to our understanding? A model of purposes of FY assessment & student

motivation with assessment A model for management of the FY Assessment

Lifecycle informed by : Student perspective Staff perspective Principles of Good Practice


Purposes of FY Assessment Diagnostic Transition enabling

Motivating

Formative Learning Summative/evaluative Terminal

Meta-reflective

Assessment of student readiness Assessment to aid transition to &

engagement with uni Assessment as stimulation for

learning Assessment for learning Assessment as learning Assessment of learning Assessment of capability/mastery Assessment of learning process


Student view of ‘motivating assessment’ Engaging Design Relevance (personal, academic, professional),

intellectual challenge, teacher enthusiasm Enabling Management Task clarity, task scaffolding, formative

conversations with & support from staff Teacher Authority Clear and firm expectations & standards,

consequences explained, responsibility invoked Prof Keithia Wilson ALTC National Fellow

Meta-model 3: The first-year assessment lifecycle – student experience 1. Student’s

prior experiences

2. Formal Framing &

student appraisal

3. Progressive Enabling &

Student Engagement

4. Submission Culture

5. Feedback Mechanisms

7. Feed-forward Mechanisms

6. Academic Outreach and Recovery


The first-year assessment lifecycle – staff experience 1. Student’s

prior experiences

2. Formal Framing &

student appraisal

3. Progressive Enabling &

Student Engagement

4. Submission Culture

5. Feedback Mechanisms

7. Feed-forward Mechanisms

6. Academic Outreach and Recovery

Design Phase

Student Preparation Phase

Assessment Phase

Post-Assessment Phase


Principles of Effective Practice in First Year Assessment Good Practice Principles rely on: 1. A lifecycle approach to assessment (4 Phases) Design phase Student Preparation phase (pre-submission) Assessment phase (marking) Post-Assessment phase (feedback & feed-forward) 2. A system’s approach to assessment (3 Systems levels) Individual Unit/Course/Subject level First Year Program level – horizontal & vertical integration Degree Program level


1. Design Phase: Individual Course/Subject Fit for purpose – Optimising student motivation &

engagement by designing assessment tasks which are perceived to be relevant & meaningful to students & their learning (sense of purpose)

Level of difficulty & complexity – Designing tasks with a conscious understanding of the assumed entry level knowledge & appropriate learning level of commencing students

Progressive knowledge & skill development – Designing assessment tasks across a semester in such a way as to ensure effective, cumulative knowledge & skill development

Variety – Ensuring a variety of different types of items both within a unit/subject, and across units /subjects in the same semester (FY program level planning)


1. Design Phase: Individual Course/ Subject

Early success -Optimising an experience of ‘early success’ for students to build academic & personal efficacy & confidence

Formative Assessment - Emphasising early formative assessment designed to develop skill & confidence viz.

Early – weeks 4-6 is optimal Smaller pieces for either no marks, or fewer marks (10-20%)

to encourage recovery from possible failure (building hope) Speedy, quality feedback (peer or staff, individually or

collectively) with feedback by weeks 7-8 Self-assessment of all written items by self-evaluating on

the identified criteria (builds meta-competence)


1. Design Phase: Individual Unit/ Subject Level

Written presentation of assessment tasks which present the task clearly and without ambiguity–

Start with the aim of the task Present the sequence of steps involved in the task,

from the simple to the complex, thus breaking down large tasks into manageable chunks Refer students to the marking criteria for the task Ensure that all available information is provided as

much as possible in the one place (see handout)


1. Design Phase – Essays – a “special case”

There are a range of data sources to indicate that essays in the first year are a source of great angst for both students and staff, and may be more complex than we (staff) think :

Professional Staff feedback Academic Staff feedback Student feedback Quality of FY essays, especially in the first semester Quality of student essays in the second & third years &

beyond! Wilson & Lizzio – ALTC Grant 2009-2010


Assessment Tasks – exams & laboratory reports

Engaging Assessment

Design

Assessment Management

& Support

Deep Approach

to Learning

Assessment Outcomes:

Student Grades

Assessment Task Efficacy


Assessment Tasks– Oral Presentation

Engaging Assessment

Design


& Support

Deep Approach

to Learning

Assessment Outcome -

Student Grades



Assessment Tasks - Essay

Engaging Assessment

Design


& Support

Deep Approach

to Learning

Assessment Outcome -

Student Grades



1. Design Phase – Essays – a “special case”

Health Group Assessment Consultant Semester 1 – essay tasks that are: Relatively short (e.g., 750-1000 words) Relatively simple (e.g., summary/description & simple interpretative

analysis such as answers to how or why questions) Include limited research requirements (e.g., simple searches of readily

obtainable information) Semester 2 – essay tasks that are: Longer (1,200-2,500 words) Incorporate more complex analysis (e.g., critical analysis & evaluation

tasks – compare & contrast) Include more complex research requirements


1. Effective FY Assessment Practice Principles – Design Phase: FY Program Level Workload distribution – Ensuring roughly equal

workload between all units in a semester to assist development of time management skills

Scheduling – coordination of submission dates to stagger the student workload

Threshold/Difficult Units - Identify & front-load any threshold units in a semester to enable student success

Variety of assessment types – Ensure variety across units in a semester

Group Work – Coordinate any group-based assessment tasks across the FY Program & preferably reduce to 1 unit only if the assessment requires additional meetings outside of class


Task Design Provide me with ‘fit for purpose’ tasks and roles

Meta-model 4: Progressive Enabling: How might we effectively manage the assessment process?

Self-Management

Help me by managing yourself

Systems Awareness Help me by

coordinating the experience


Reflective Task: Assessment Design Phase How does this information apply to your FY context?

What are the opportunities for enhancing FY

assessment design?

What are the challenges with enhancing FY assessment design?


2. Student Preparation Phase: Individual Course/Subject Level

Motivating students – 3 Domains of Motivation Articulating explicitly the academic relevance of each assessment

task (the knowledge and skills that will be useful to students later in their course/degree);

Articulating explicitly the personal relevance of each assessment task (the knowledge and skills that will be useful for students in the future);

Articulating explicitly the professional relevance of each assessment task (the knowledge and skills that will be useful to students later in their career);


2. Student Preparation Phase: Individual Unit/Subject Level

Motivating students – 2 Types of Relevance Articulating future positive relevance (personally &

professionally) of the knowledge and skills to be gained from an assessment task to the course/degree/future employment;

Articulating future negative relevance of the knowledge and skills to be gained from an assessment task to the course/degree/future employment for students who may not engage sufficiently;



Motivating students – Challenge Articulating explicitly the intellectual challenge of each assessment

task (the challenge offered to students to think and learn); Articulating to students the investment of work required to be

successful with the task (encourages development of self-regulatory behaviour);

Staff Stance Staff conveying to students their own personal enthusiasm for the

task (staff engagement increase student engagement); Being clear and well organised (assists anxiety management &

increases performance ability).



Management Strategies - Scaffolding learning -Designing a process for scaffolding assessment

preparation which prepares students for each assessment task: Providing detailed goals, criteria, standards for each task to clarify

what good performance is – viz. clear goals & standards Providing opportunities for students to actively engage with, &

potentially modify those goals, criteria, standards Providing high & low quality examples of performance for each

assessment task or a similar task Providing targeted resources such as practice items, quizzes, mini-

essay writing, step-by-step processes organised sequentially etc. Providing multiple & regular opportunities for discussion of

assessment tasks & requirements Ensuring consistent information & resources are provided from all

teaching staff (convenors & tutors) on assessment tasks


2. Student Preparation Phase: First Year Program Level Clear Goals and Standards Ensuring all course convenors have

detailed criteria for all assessment tasks in their units/courses Responsive culture Ensuring a responsive FY Program culture in all

units/courses, including training of sessional staff consistent messages about performance & success

Consistency of referencing style Providing a single referencing style only for FY students for their FY of study for all units/courses in their Program

Consistency of information storage by unit convenors for web-site information for all units (same folders)

Consistent terminology to describe same types of assessment tasks across a program e.g., critical reflection/critical analysis/essay/critique


Facilitating Motivation

Help me to engage

Building Capacity Help me to be task

capable


Progressive Enabling: How might we effectively manage the assessment process?

Self- Management Help me by

managing yourself

Systems Awareness Help me by coordinating expectations


Facilitating Motivation

Help me to engage

Relating Functionally

Help me to feel understood

Managing Information

Help me to understand the task

Facilitating Process

Help me to solve problems

Managing Procedures

Help me to navigate the rules of the

game

Building Capacity Help me to be task

capable


Progressive Enabling: How might we effectively manage the assessment process?

Self- Management Help me by

managing yourself

Systems Awareness Help me by coordinating expectations


Reflective Task: Student Preparation Phase How does this information apply to your FY context?

What are the opportunities for enhancing FY student

preparation with assessment ?

What are the challenges with enhancing FY student preparation with assessment?


3. Assessment Marking Phase: Individual Unit/Subject Level Transparency & Consistency – Ensuring

consistency of marking standards between markers by:

Marker preparation - having agreed, transparent , detailed standards for

preparing markers to assess Marking trial – initially marking 5 or so & then meeting to

compare standards of marking


3. Assessment Marking Phase: Individual Unit/Subject Level Marker Moderation - facilitating high quality moderation of final assessment marks

(procedural justice) by staff teams viz. * staff team meeting face-to-face to discuss education & capacity

building for staff re marking standards * all staff reading all fails & HDs (depending on numbers) to clarify

understanding of high & low end standards * all staff reading & re-assessing all assessment items on the margins of

each grading category (high Ps & low Cs etc) final re-adjustment of marks * keeping copies of high & low end examples to be used anonymously

with future cohorts (with student permission)


3. Assessment Marking Phase: Individual Unit/Subject Level

FY Assessment Debriefing & Learning for students providing speedy feedback (2 weeks optimal) Providing quality individual feedback which can feed-

forward into other assessment tasks - * identifying achievements (encouragement) * identifying clearly & explicitly what students need to do to improve their performance in an encouraging way (developmental) * sufficient comments to justify the mark (fair & just) summarising cohort strengths & weaknesses


4. Post-Assessment Phase: Individual Unit/Subject Level Timely feedback – Ensuring students receive

feedback within a short time frame (1-2 weeks is optimal) during semester

Empowerment - Ensuring opportunities for individual and group discussion of marks and grades

Academic recovery – identifying and intervening with at-risk students who have failed their first assessment item.


4. Post-Assessment Phase: First Year Program Level Empowerment - Creating a first year culture of

ensuring opportunities for individual and group discussion of marks and grades

Academic recovery – Program level strategy by identifying & front-loading the threshold (most difficult) unit/s with the aim of ensuring intervening with at-risk students who have failed their first assessment item to assist in their academic recovery

.


Strategies for Improving Assessment Design & Practices – Assessment Review Conducting a macro level review of all first year assessment (viz.

Types of tasks, range of word counts, weightings, weightings x word counts, profile of semester submission dates) & discussing findings as a FYT team

Conducting a micro level review of all first year assessment items from a student perspective viz. Levels of difficulty, clarity, alignment with course objectives, etc, and including suggestions & resources for scaffolding student engagement with assessment items – whole team & individual work

Focusing on individual; courses, as well as a programmatic approach to first year assessment


First Year Assessment Consultant

Completing written Assessment Reviews for first year courses & first year programs

Evaluating assessment tasks Assisting academics with developing curriculum and

assessment tasks Assisting academics with writing assessment tasks Assisting academics with the development of resources

to scaffold and support student learning and assist with understanding of assessment tasks


Some Strategies for Academic Recovery Preventative Strategies – Draft submissions for first assessment tasks with feedback

provided Re-submission of a failed assessment task for a possible

passing grade only (1 only in each semester, or semester 1 only, or the first year)


Potential hierarchy of academic recovery interventions 1. Students responsible for initiating help-seeking without

prompting; 2. Email communication to all students inviting contact to

discuss first assessment and providing information regarding support services;

3. In-class activity discussing first assessment performance and feedback;

4. Targeted written communication to students who failed or nearly failed first assessment inviting contact to discuss first assessment and providing information regarding support services;


Potential hierarchy of academic recovery interventions 5. Self-reflective workbook distributed to students for

independent completion and invitation for follow-up contact;

6. Phone call to students who failed or nearly failed first assessment inviting contact to discuss first assessment and providing information regarding support services;

7. Targeted invitation to students who failed or nearly failed first assessment inviting participation in structured face-to-face consultation and planning session with tutor.


First-Assessment First-Feedback Academic Recovery Intervention Key Idea: Efficacy building for students who ‘fail’ or ‘marginal pass’

first assessment in a core/threshold course Key Aspects: Students complete a self-directed workbook Individual structured session with tutor leading to an

action plan Follow-up phone or email contact 40% uptake Participation results in a 10% increase in submission rates

& 20% increase in pass rates for 2nd assessment item, & 40% increase in passing the course overall


First-Assessment First-Feedback Academic Recovery Intervention At-risk students’ self-reported evaluations of the process

were uniformly positive – Academic related learning (5.7/7) Personal development (5.02/7) Insight into reasons for under-performance (5.6/7) Increased efficacy & optimism (5.6/7) Process rated as non-aversive (5.3/7) Tutors reported stronger relationships with students, higher

attendance at tutes by those students, & greater student engagement


Reflective Task: Post-Assessment Phase How does this information apply to your FY context?

What are the opportunities for enhancing FY post-

assessment feedback processes?

What are the challenges with enhancing FY post-assessment feedback processes?


Working with Diverse Students - implications for curriculum design & assessment Identifying our assumptions about the entry levels of Assumed

Knowledge of our students (x course + x first semester & then first year overall) & designing course content, structure & assessment tasks to take account of FY students’ entry levels

Scaffolding student understanding of assessment tasks by providing resources to make expectations explicit & assist task understanding

Scaffolding student learning in each unit/course by building required attitudes, knowledge & skills progressively across the first semester & first year

T&L in the FYE needs to be conceptualised as a social experience where students are provided with rich & varied opportunities for interaction & dialogue with peers & academic staff & we are building a relational school/program culture


prof keithia wilson - flinders university keithia wilson griffith university 2010 altc national...

Documents