Interim Evaluation of the Graduate AttributesReport for AESC, January 2013Rhona Sharpe & Greg Benfield, OCSLD

Executive summaryThis paper reports on the first year of a three-year programme of work to evaluate the impact of the specification of the (post)graduate attributes in the Strategy for Enhancing the Student Experience, and their adoption within programmes of study at Oxford Brookes. Work so far has assessed how the attributes have been incorporated into and expressed within programme documentation.

The overall purpose of presenting this interim report is to seek feedback on the evaluation approach. Will this evaluation produce outputs that are going to be of use to programme teams and those responsible for their quality assurance?

This first year of the evaluation conducted an interpretative analysis of the programme learning outcomes in Programme Specification documents from 90 undergraduate programmes. This has suggested revised and extended definitions of the two of the five Brookes Graduate Attributes (research literacy and personal literacy) and illustrative examples of how they have been articulated and contextualised by programme teams from a variety of disciplines. This activity has produced a set of reusable staff development resources.

Overall, the analysis found that the graduate attributes are often not yet explicit in the expression of programme learning outcomes (PLOs) in programme documentation. It was inevitable that the initial 2012 mapping exercise of existing PLOs onto the five graduate attributes would sometimes give an imperfect fit, because the PLOs had been written for other categories. It was envisaged that following the initial mapping, the next stage would be to refine the expression of PLOs for graduate attributes during the normal course review processes. The report discusses this further and makes recommendations for future staff development to be integrated into quality assurance and course approval processes.

This analysis confirms that the implementation of the (post) graduate attributes is an ongoing piece of work.

AESC are asked to comment on and approve this approach to the evaluation and the outputs it will produce. After which, the process will be repeated for the three remaining Graduate Attributes and the Postgraduate Attributes

Comments are welcome and should be directed to Rhona Sharpe, OCSLD, rsharpe@brookes.ac.uk.

1

1. OverviewThis paper reports on the first year of a three-year programme of work to evaluate the impact of the specification of the (pos)graduate attributes in the Strategy for Enhancing the Student Experience, and their adoption within programmes of study at Oxford Brookes. The ultimate aim of the three-year programme is to be able to assess student achievement of the attributes, so that we can say with confidence that students at Brookes graduate with these attributes and that their experiences here contribute to their development. There are some necessary precursors to this ambitious aim in consolidating our understanding of the attributes (year 1), understanding how to create conditions in which their development is facilitated (year 2) and developing tools for measuring their achievement (year 3).

Year 1 (2011/12): Evaluating definitions of the attributesThe aim is to assess how the graduate attributes have been incorporated and expressed in programme documentation. Expected outputs:

Reflections on the attribute mapping process leading to recommendations for the mapping of collaborative provision in 12/13 (see Section 2)

Analysis of programme learning outcomes resulting in revised and extended definitions of the (post)graduate attributes (see Section 3)

Examples of programme learning outcomes, which show how the (post)graduate attributes have been contextualised by a range of disciplines (see Section 4).

Recommendations for continuing the embedding of the attributes within Brookes programmes.

Year 2 (2012/13): Evaluating development of the attributes

The aim this year is to explore how the attributes are being developed and promoted within programmes. Expected activities and outputs:

Further analysis of programme documentation will focus on how the attributes are being taught, practised and assessed, and will identify programmes which have made significant efforts to embed the attributes into their design.

Interviews with Subject Coordinators and Module Leaders of these programmes will provide additional detail. The main outputs will be case studies to add to the ‘Graduate Attributes in Action’ wiki1, which illustrate examples of teaching, learning and assessment activities promoting each of the attributes.

Piloting of instruments, developed from the taxonomies, to measure students’ achievement of each attribute.

Year 3 (2013/14): Evaluating achievement of the attributesThe aim this year is to gather evidence from students of how they develop the attributes and what it means to them to graduate from Brookes with these attributes. Expected activities and outputs:

Students from programmes identified in Year 2 will be interviewed about their experiences, resulting in video resources of student experiences.

The measurement instruments will be finalised and data collected from final year students.

2. Implementation: reflections on the process1 Graduate Attributes in Action Wiki at https://wiki.brookes.ac.uk/display/GAA/

2

2.1 The implementation processStaff and students at Oxford Brookes have experienced a number of curriculum change initiatives in recent years, including semesterisation, the Academic Progression Initiative2, internationalisation of the curriculum and the adoption of the Assessment Compact3. These have encouraged decision making at the programme level in order to create a coherent student experience. In addition, in 2011, programme level changes could be managed and resourced within the revised faculty academic management structure, led by Programme Leaders. OCSLD recommends that programme teams work together in extended, multi-professional groups that include students, and preferably alongside teams from other disciplines (such as the Course Design Intensives4).

When we are rethinking programme design, it is important to rethink how we document the learning experience most effectively. Documentation that is required for QA processes should reflect the programme design process and be used to communicate the impact of design decisions to all interested parties, including approval panels, members of teaching teams and current and prospective students.

As part of the Strategy for Enhancing the Student Experience, all programmes are required to show how they develop the Brookes (post)graduate attributes. As a first step, in 2011/12 all programmes undertook an exercise to map (and, if necessary, revise) their programme learning outcomes against the Brookes attributes, and to document these changes in a revised Programme Specification template. Figure 1 shows the expected implementation process including the intended relationship between programme level decisions and documentation. The actual implementation process is summarised in Table 1.

Figure 1: Slide from the SESE Symposia5 held in July and September 2011 for educational change agents (ADSEs, HoDs, Prog Leads, PLSEs, Student Representatives)

Table 1. Summary of actual implementation process2 Academic Progression Initiative at http://www.brookes.ac.uk/about/directorates/asa/registry/apqo/news/archive/api3 Brookes Assessment Compact at http://www.brookes.ac.uk/aske/BrookesACompact/4 Course Design Intensives at https://wiki.brookes.ac.uk/display/CDIs/Home5 SESE symposia for change agents at https://wiki.brookes.ac.uk/display/GAA/SESE+Events

3

Jan – May 2011 OCSLD produced a series a ‘graduate attributes in action’ case studies. https://wiki.brookes.ac.uk/display/GAA/Home

June 2011OCSLD ran roadshows to find out more about the attributes and discuss the case studies. https://wiki.brookes.ac.uk/display/GAA/SESE+Events

July – Sept 2011

OCSLD ran days for change agents to devise materials to support the process of mapping graduate attributes across their programmes. https://wiki.brookes.ac.uk/display/GAA/SESE+Events

Sept - Nov 2011

OCSLD devised tools for mapping the graduate attributes and revised templates for Programme Specification and Module Descriptors. https://wiki.brookes.ac.uk/display/GAA/Tools+for+mapping

Sept 2011 – March 2012

Mapping workshops for programme teams. OCSLD developers available to facilitate and to provide feedback on draft Programme Specifications.

Jan – Feb 2012

Planning meetings in faculties to finalise their own processes. Some produced additional guidance materials e.g. in TDE, the PLQA produced a short video presentation which described how to complete the new Programme Specification template.

Feb – March 2012

Extra OCSLD workshops on the two tricky GAs: digital and information literacy and global citizenship.Faculty led support e.g. ‘Trigger events’ on each GA and ‘Embedding events’ in Business led by the PLSEs.

30 April 2012 Deadline for submission of GA mapping documents, coversheet and revised Programme Specification to Faculty Quality Officer.

May – Sept 2012Faculty approval processes. Subgroups of FAESCs met to scrutinize documentation, provide feedback and record progress.

2.2. Reflections on the graduate attributes mapping process

A meeting was held with APQO Quality Officers and OCSLD Developers, which elicited the following reflections on the graduate attributes mapping process conducted in 2011/12. An early version of the following summary was discussed at the PESE Steering Group in September 2012.

2.2.1. Investment of effort

The process was an enormous investment of effort, both for programme teams and those involved in the faculty approval processes. This was partly as a result of working with both the (post)graduate attributes and a new Programme Specification template in the same year. The initial guidance was that programmes were to map their current teaching and learning activities and learning outcomes against the attributes, with three possible outcomes:

I. The programme team is able to map the existing programme learning outcomes onto the (post)graduate attributes without changing them

4

II. The existing programme learning outcomes require some amendment in order to map onto the (post)graduate attributes.

III. The mapping exercise reveals that more radical changes are required to the programme in order to meet the (post)graduate attribute initiative

The vast majority of programmes came into the second category of some change to the programme documentation, including the re-writing of learning outcomes, to reflect more clearly and accurately what was already happening within the programme.

In practice, the faculty approval processes both identified which of the three possible outcomes were appropriate for each programme and immediately started work on improvements. The light touch minor change allowed by APQO this year for this purpose was widely used and appreciated.

2.2.2. Articulating our expectations of students

Well-articulated PLOs are important to prospective and current students, teaching staff, employers and professional bodies. They express to students what they are expected to be able to do by the end of their course, clarify for teaching staff what students must achieve, and enable the assessment process to become more fit for purpose. Brookes’ use of (post)graduate attributes to organise PLOs should help graduates to better articulate to external agents their achievements across a broad curriculum.

Many programmes took the opportunity to improve their learning outcomes so that they were defined more precisely and better captured the student experience of the programme. Frequently these were relatively minor amendments, such as splitting a PLO into two component parts and distributing these across two attributes. Sometimes rewriting of PLOs seemed necessary because they had not been considered between validations or periodic review and/or those events had not prompted a consideration of the PLOs. There are considerations here for those who chair and sit on periodic review and approval panels. Sometimes the abandonment of the transferrable skills category and the way that it had been used much as a checklist resulted in items that needed rewriting to turn them into well-specified PLOs.

It should be acknowledged that it is difficult for teams to articulate their expectations of students at a programme level. It is a challenge to adequately summarise the totality of the learning outcomes achieved by students from their modules without being overly general and abstract. Despite the Academic Progression Initiative, there are ways in which programmes are still thought of as aggregations of modules. For example, it is a continuing challenge of the Assessment Compact to design assessment at programme level. And while we should be able to assume that PLOs are mapped and aligned at periodic review time, in large, complex programmes, documentation might not keep pace over time with accumulating changes at the module level.

Two attributes seemed particularly problematic to articulate within documentation. Digital and information literacy and global citizenship tended initially to be underspecified in the programme documentation, even though staff were able to explain verbally how they were developed within the curriculum. Sometimes we could see how the abandonment of transferrable skills contributed to this. We could also see that hitherto programmes had simply not seen such capabilities as important to be explicit about, despite them being developed through the student experience. OCSLD and faculties responded with additional events on these outcomes (see Table 1).

The intention is that Programme Specifications should now be written for an external audience. There is a need to develop a shared understanding of what a well-expressed Programme Specification looks like, perhaps through the sharing of good examples and discussion of the purposes of the document.

There was discussion of whether there needed to be different attributes for different exit awards. Foundation degree teams were unsure of whether to incorporate the

5

graduate attributes or not and some which have recently been validated did use the graduate attributes. Attributes have since been produced for Foundation Degrees.

2.2.3 Is mapping the start or the end?

The process did not always result in discussion about the curriculum as was intended. It had been hoped that the mapping process would trigger discussions of how attributes are contextualised for the discipline and developed within the programme. The IiP assessor noted in his report on the July 2012 visit that ‘It is worth broadening the debate about whether the work on implementing the Graduate Attributes is about curriculum development, or staff development, or both.’, explaining that a small minority of staff he spoke to during the visit preferred to focus on teaching their subject.6

The process did not always engage entire programme teams as was intended – programme leads or subject coordinators sometimes took the lead in rewriting the documentation. There are examples of the same learning outcomes being used across a number of programmes in cognate disciplines.

Very few programmes used the ‘no change’ option or the ‘major change’ option to trigger wider changes to the programme or modules. There is a danger that the mapping exercise is seen as a completed job, rather than as a first step towards embedding the graduate attributes within the student experience. However, as noted above, our analysis of the programme documentation shows that there is still work to be done in expressing the (post)graduate attributes clearly and explicitly in programme documentation. In Section 4 we make recommendations for supporting this work.

6 Kearns, P. (2012) Investors In People Review Report, July 2012, included in papers at IiP Steering Group on 23 August 2012

6

3. Understanding the Brookes graduate attributesThe descriptions of the Brookes (post)graduate attributes are currently generic. In order to mean anything to students they need to be interpreted in relation to each programme’s own disciplinary practices and assessment demands. The phase of the evaluation was led by two enquiry questions:

To what extent are the graduate attributes made explicit in programme documentation?

To what extent have the graduate attributes been contextualised for the discipline?

3.1 Method

The analysis used the 90 undergraduate Programme Specifications available on the APQO website as of 8 Jan 2013 (see full list in Table 3, Appendix 1). These had been completed in the new template following the mapping exercise. We imported these into NVivo and coded the PLOs by graduate attribute and its sub elements. Two additional codes were used to highlight PLOs that could be used as exemplars:

Articulated: an example that expresses the graduate attribute beyond the definition in the SESE

Contextualised: an example of a PLO where the graduate attribute has been contextualised for the discipline.

More details of the method of analysis are given in Appendix 1.

This analysis is interpretive. Such an exercise is always fraught. We are familiar with some of these courses and were aware in several instances that other, possibly more sympathetic, readings were possible. However, our aim was to read the PLOs as ‘dispassionate outsiders’ would. We deliberately tried to avoid making inferences about content of the curriculum and relied only on the explicit wording of the statements we read. As a result we found that the way PLOs were expressed sometimes belied the richness of the curriculum. We hope that the analysis that follows will help programme teams to evaluate and improve their public documentation.

3.2 Research Literacy

‘The ability to be a critical consumer of research, and also, where possible, to design and undertake at least a small-scale research project in the discipline, using appropriate methodology’. (SESE Section 4.1)

This definition can be interpreted as having three component parts:

ability to be a critical consumer of research, to design and undertake at least a small-scale research project in the discipline, using appropriate methodology

In addition, programme learning outcomes that articulated an expression of the attribute that went beyond the definition in the SESE were coded as ‘articulated’. Outcomes which expressed how research literacy looked distinct to the discipline were coded as ‘contextualised’.

3.2.1 Critical consumer of research

What does it mean to be a ‘critical consumer of research’ at the undergraduate level? There were 14 PLOs in 12 programmes which were coded for both ‘critical consumer of research’ and ‘articulated’. These are listed below:

7

1. An ability to read and interpret texts (print, oral, film, multimedia) within their historical, social and theoretical contexts (Anthropology)

2. Identify and utilise trustworthy information sources, such as the ACM Digital Library to develop a coherent understanding of issues in the domain. (Computer Games and Animation)

3. Interpret experimental data in terms of current paradigms in the biosciences (Biomedical Sciences)

4. Critically evaluate contemporary theories, arguments and empirical evidence in accounting and finance (Accounting)

5. Identify, gather and analyse multiple source materials (Primary Teacher Education)

6. Analyse and evaluate critically subject specific and multi-disciplinary literatures (Primary Teacher Education)

7. Reflect critically on and interpret French texts with reference to their social and cultural background and experience (French Studies)

8. The ability to make effective use of a variety of sources and to evaluate these critically (Film Studies)

9. Locate, retrieve and appraise material from a range of disciplines and perspectives in forming and supporting an argument (Student Designed Award)

10. Critically review the evidence supporting conclusions or recommendations, using validity, reliability and significance, and can investigate contradictory information (Student Designed Award)

11. An ability to use a wide range of musicological sources and be able to interpret and present findings in appropriate ways (Music)

12. The ability to analyse specific claims and the arguments for and against them (Philosophy)

13. Make discriminating use of a full range of library resources in order to identify appropriate source material, compile bibliographies, inform research and enhance presentations. (Religion and Theology)

14. Accurately cite primary and secondary sources within an approved system of scholarly reference (English)

We can see from these learning outcomes that accessing a variety of sources is important to being a critical consumer of research, and that these sources might take different forms. It is also clear that programme teams expect their graduates to be able to interpret sources within their contexts (see items 1, 2 and 7 in the list above). Critical consumers of research also need to be able to use academic protocols to show what they’ve done (item 14).

Does being a critical consumer of research look different in different disciplines? There were 13 PLOs from 10 programmes which were coded for both ‘critical consumer’ and ‘contextualised’. These are listed below:

1. Interpret experimental data in terms of current paradigms in the (human) biosciences (Biomedical Science, Human Biology)

2. Employ existing ideas and critical judgement when considering their own work and the work of others (Fine Art)

3. Identify and utilise trustworthy information sources, such as the ACM Digital Library to develop a coherent understanding of the issues in the domain (Computer Games and Animation)

4. Evaluate and exploit the idiosyncratic information sources characteristic of the game development community (Computer Games and Animation)

8

5. Synthesise and evaluate critically a range of primary and secondary historical sources, in accordance with their relevant historical contexts (History)

6. Critically evaluate examples of communicative behaviour for effectiveness in relation to context, purposes and audience (Communication, Media and Culture)

7. Competence in exploiting, for a wide variety of purposes, a broad range of authentic materials written or spoken in Spanish (Spanish)

8. Use of range of evidence to formulate appropriate and justified changes in policy and practice (Education Studies)

9. Evaluate and critique health care research in order to provide evidence-based practice. (Nursing Studies)

10. Locate, retrieve and appraise material from a range of disciplines and perspectives in forming and supporting an argument. (Student Designed Award)

11. Synthesise and evaluate the evidence from a range of sources to provide individual women with the information to make informed choices. (Midwifery)

12. Critically analyse the evidence to support decisions made as a lead professional in midwifery care, recognising the limits of professional knowledge and expertise (Midwifery)

13. Promote and contribute to the utilisation and application of research findings in the workplace (Health and Social Care)

These show that some teams have asserted the disciplinarity of research literacy through an explicit reference to the disciplinary context (items 1 and 4). Of more interest is where the distinctiveness of being a critical consumer of research in the discipline is actually expressed. The disciplinary differences here seem not to be in locating, reading or interpreting sources (which are all described in similar terms) but in how a critical consumer of the discipline’s research would put that knowledge to use, such as in making an argument (item 10) or informing practice (items 8, 9, 11, 12). This is starting to enter the territory of academic literacy in terms of behaving as a professional in the discipline community.

3.2.2 Designing and undertaking research, using appropriate methodology

The learning outcomes in 39 (43%) of the Programme Specifications explicitly referred to designing and undertaking research in the discipline. The attribute definition at the undergraduate level does clarify this should be included ‘where possible’. Of these 39 programmes, 16 of them made explicit mention of using appropriate methodology, although only three related this explicitly to the project e.g.

Design and implement research studies, selecting appropriate statistical methods to evaluate human responses to exercise (Sport and Exercise)

More often, learning outcomes made reference to the importance of methodology beyond the specific context of the project e.g.

A capacity to provide an ethnographic description and analyse it (Ethnography)

Describe, explain, evaluate and utilise a range of technical research methods including a wide range of class, field and laboratory based techniques (Geography)

Indeed there were 19 programmes which included PLOs coded for methodology that had not made a project an explicit feature in their PLOs (International Business Management; Business and Management; Business; Economics & Finance & International Business; Fine Art; Nursing Studies; Health and Social Care; History; International Business; Economics; English Language Communication; Electronic

9

Engineering; Mathematics; Multimedia Production; Philosophy; City and Regional Planning; Software Engineering; Social Work, and Education Studies). Research methodology appears to be important beyond the research project itself notably where research methodology was important in practice e.g.

Structure, design and apply tools of research, analysis, decision making and evaluation to a wide range of international scenarios and problems (International Business Management, Business and Management)

Demonstrate an ability to apply research methods to practice (Nursing Studies)

Or where the context for the application of the methodology was not specified

Utilise appropriate research and development methodologies within Fine Art contexts (Fine Art)

Have knowledge and understanding of the rationale and techniques for the selection, analysis and interpretation of data (English Language Communication)

3.2.3 Gap analysis

There were also 55 PLOs from 46 (51%) programmes which had been listed by course teams under the research literacy graduate attribute, but which were not coded against any of the three elements predefined through the SESE definition (critical consumer, project, methodology). The vast majority of these were PLOs where the ‘dispassionate outsider’ is likely to see only indirect relations with consuming or performing research. Although we can see reasons why these PLOs have been placed under the research literacy heading, our reading as dispassionate outsiders led us to actually code them for academic literacy. Here are three examples

Demonstrate an ability to relate theories of child development to observations of children in a variety of child care environments and settings (Early Childhood Studies)

Apply the knowledge, understanding and disciplinary skills of the economic and financial strategist to the analysis of international business behavior and the formulation of effective business strategy (Economics, Financial and International Business)

The capacity to analyse and understand diverse forms of discourse and group dynamics (Drama)

The final query took an overview of the items coded for research literacy and its sub elements in the Programme Specifications. This showed that in the 90 Programme Specifications there were 93 PLOs coded for critical consumer of research, 60 for project and 59 for methodology (see Table 6, Appendix 2 for more details).

3.2.4 Summary of research literacy

This analysis to explore the SESE definition of research literacy shows that course teams have explained being a critical consumer of research as one or more of:

o Accessing sources in a variety of formso Interpreting sources within their contexto Using academic referencing protocols

Where there are differences between disciplines in their definitions of critical consumer of research, these relate to how a student would put their consumption of research to use.

With regard to ‘designing and undertaking a small scale study, where possible, using appropriate methodology’, the analysis shows that:

o 43% of undergraduate programmes explicitly refer to students undertaking research such as through a small scale project

10

o Programme teams explained that knowledge and skills in utilising research methodology may have a broader application, beyond undertaking a project, particularly in relation to practice

The gap analysis showed thato 51% of Programme Specifications included PLOs which we saw as having

only indirect relations with consuming or performing research, and that would be more suitably placed under other attributes, mostly academic literacy.

A suggested rewording of Research Literacy, taking into account the preceding analysis might be

o ‘The ability to be a critical consumer of research, and also, where possible, to who can utilise the research methodologies of the discipline, and where possible, design and undertake at least a small-scale research project in the discipline, using appropriate methodology’

3.3 Critical self-awareness and personal literacy “Understanding how one learns, the ability to assess the work of oneself and others, and to identify one’s strengths and weaknesses. The ability to organise oneself and perform as an autonomous, effective and independent learner. The ability to relate to other people and function collaboratively in diverse groups, including the development of appropriate interpersonal skills, emotional intelligence and adaptive expertise.” SESE Section 4.1

This definition can be interpreted as having three component parts:

metacognition independence interpersonal skills

In addition, PLOs that articulated an expression of the attribute that went beyond the definition in the SESE were coded as ‘articulated’. Outcomes which expressed how personal literacy looked distinct to the discipline were coded as ‘contextualised’.

3.3.1 Metacognition

“Understanding how one learns, the ability to assess the work of oneself and others, and to identify one’s strengths and weaknesses

What does it mean to demonstrate metacognition at the undergraduate level? There were 8 PLOs in 8 programmes which were coded for both ‘metacognition’ and ‘articulated’. These are listed below:

1. Operate as a reflective practitioner by synthesising reflection on their personal experience and feedback from others in the professional environment in order to evaluate their own performance and recognise the need and opportunities for further personal and professional development. (Business and Marketing)

2. Demonstrate self-awareness through reflection on continuing personal and professional development including the ability to undertake self-evaluation and action planning. (International Business Management)

3. Demonstrate self-awareness by reflection on continuing personal and professional development including the ability to self-evaluate their own performance (Business and Management)

4. Seek and make use of feedback, and self-evaluate own work. (Business Management)

11

5. Identify and evaluate their strengths and weaknesses, including an awareness of their personal ethical positions (Geography)

6. Assess and reflect on the strengths and weaknesses of your work, and make use of feedback to improve your performance (History of Art)

7. Evaluate own performance, identify own strengths and weaknesses and respond appropriately to any identified learning needs (Student Designed Award)

8. Reflect on outcomes in order to evaluate the performance of oneself and others (Nutrition)

Four programme teams defined metacognition as coming about through reflection on experiences (items 1, 2, 3, 8). The other four programmes emphasised the importance of self-evaluation (items 4, 5, 6, 7). In one discipline metacognition also encompassed a self-awareness of one’s own ethical position (item 5).

Does metacognition look different in different disciplines? There were 14 PLOs from 9 programmes which were coded for both ‘metacognition’ and ‘contextualised’. These are listed below:

1. Demonstrate self-awareness by reflection on continuing personal and professional development including the ability to self-evaluate their performance (Business Management).

2. Seek and make use of feedback, and self-evaluate own work (Business Management).

3. Demonstrate the learning ability needed to undertake appropriate further training of a professional or academic nature (Business Management).

4. An ability to critically evaluate performance practice (their own and the work of others) (Drama)

5. An ability to evaluate constructively their own and other’s critical practice; ability to reflect in a critical and productive way on their practice; ability to transform their practice as a result of critical self-evaluation (Drama)

6. An ability to negotiate one’s own and others creative and intellectual goals in a challenging environment (Drama)

7. Undertake systematic critical self-reflection, within the planning, teaching and assessment cycle (Primary Teacher Education)

8. Develop personal reflective practice (Nursing Studies)9. Critically reflect on the factors that influence practice and apply these to

individual situations. (Nursing Studies)10. Develop and reflect on a set of professional skills based on an understanding of

the management issues involved in running a media business (Multimedia Production)

11. Work effectively as a reflective practitioner to enhance and update own practice (Midwifery);

12. Commit to continuing personal and professional development in response to changes in professional practice, service provision, social policy and life long learning initiatives (Occupational Therapy)

13. Demonstrate habits of critical reflection on their performance and take responsibility for modifying action in the light of this (Social Work)

14. Evaluate personal performance in applying theory to practice in the use of Sound Technology and Digital Music equipment (Sound Technology and Digital Music)

Here, once again, the disciplinarity is expressed by the context in which reflection is expected to take place, whether practice (items 7, 9, 11 and 13) or performance (items 4 and 5). Some extended the definition from identifying own strengths and weaknesses to planning and engaging in continuing professional development (items 1, 3 and 12)

12

as an expectation of their discipline.

3.3.2 Independence

The ability to organise oneself and perform as an autonomous, effective and independent learner.

What does it mean to be an independent learner at the undergraduate level? There were 6 PLOs in 9 programmes which were coded for both ‘independence and ‘articulated’. These are listed below:

1. Time planning and management skills (Anthropology Single)2. Demonstrate self-management in terms of meeting programme deadlines and

managing own programme of study to achieve academic and professional objectives (Accounting and Business; Accounting; Accounting and Information Management

3. Develop a high level of self-motivation and effective time management (English Language Communication)

4. Manage their own learning by clarifying personal values, setting personal objectives, managing time and tasks and evaluating their own performance (Publishing Media; Publishing)

5. Set goals and manage time and tasks effectively (Sociology) 6. Plan, justify and manage an individual programme of learning to meet own

identified learning needs (Student Designed Award)

These specified the activities which independent learners would be expected to be able to undertake: setting personal goals and objectives (items 4 and 5), and managing time and tasks (items 1, 2, 3, 4, 5). The Student Designed Award took a more holistic view of planning a programme of learning (item 6).

There were four PLOs from four programmes that were coded for ‘independence’ and ‘contextualised’:

1. Manage own learning, with minimum guidance, exercising initiative and personal responsibility (Business Management)

2. Self-management and organisational skills both at a personal level and in the area of project management (Automotive Engineering)

3. The ability to work independently, both critically and creatively (Music)4. Develop and reflect on a set of professional skills based on an understanding of

the management issues involved in running a media business (Media Technology)

Learning outcomes can usefully define the context in which the outcome of learning takes place. For some programmes independent learning was expected to have influence beyond the educational context such as in project management (item 2) or in creative practice (item 3). Similarly, programme outcomes can define the standard at which graduates would be expected to display independence in learning such as ‘with minimal guidance’ (item 1).

3.3.3 Interpersonal skills

The ability to relate to other people and function collaboratively in diverse groups, including the development of appropriate interpersonal skills, emotional intelligence and adaptive expertise.

What does it mean to demonstrate interpersonal skills at the undergraduate level?

13

There were 14 PLOs in 14 programmes which were coded for both ‘independence and ‘articulated’. These are listed below:

1. Ability to engage in constructive discussion in group situations and skill in group work. (Anthropology Single)

2. Work cooperatively and participate in multicultural and diverse teams, demonstrating personal initiative (Business)

3. Forge positive, effective working relationships with colleagues (Primary Teacher Education; Primary Teacher Education WBL)

4. Work collaboratively as a member of a team or group in a way which allows each individual’s talents to be utilised effectively (Primary Teacher Education; Primary Teacher Education WBL);

5. The ability to listen effectively to discussion and feedback and so to learn from and participate constructively these communicative processes (Film Studies)

6. The ability to work effectively in a team, displaying the abilities to listen, to contribute and to co-operate and thus to sustain and promote the efforts of the team to meet specific goals. (Film Studies)

7. Work effectively as part of a small research team and equally effectively as an independent researcher (Geography)

8. Apply self-awareness in evaluating their impact in team based work and utilise appropriate communication and problem resolution strategies. (Computer Games and Animation; Information Technology Management for Business)

9. Work with others constructively and supportively (History of Art)10. Listen actively so as to be able to participate in and learn from discussion, and

persuade rationally, through negotiation and defence of a position, while respecting other people's stances. (History)

11. Work productively in groups both as discussion leaders and in a supportive role. (History)

12. Demonstrate an understanding of the processes, both verbal and non-verbal, whereby people manage communication face-to-face and in the context of groups; (Communication, Media and Culture)

13. Take the initiative and lead others and participate as an active and engaged team member (Publishing; Publishing Media)

14. Be flexible and adaptable in the approach to and development of an issue, problem, or opportunity. (Architecture)

Some programme teams used their learning outcomes to explain interpersonal skills as: listening (items 5, 6, 10), discussion, argument and debate (items 1, 10, 11) or leadership (item 14). Some went on to explain why these skills were important. Clearly there are important to working in groups and teams (items 1, 2, 4, 6, 7, 8, 13). For History graduates it is important to be able to engage in a respectful discussion of differing views, for Business it is to work co-operatively in teams (Item 2), for teachers it is to develop effective working relationships with colleagues (item 3).

It is notable that only one in this list could be seen to represent ‘adaptive expertise’ (Item 14). This prompted a review of the PLOs coded as personal literacy to find other examples of adaptive expertise. This identified four examples across four programmes.

There were 16 PLOs from 15 programmes that were coded for both ‘interpersonal’ and ‘contextualised’. These were:

1. An ability to engage with cultures, populations and groups different from one's own, without foregoing a sense of personal judgement. - An awareness of cultural assumptions, including one's own and the ways in which these impact on an interpretation of others (Anthropology Single)

14

2. Interact effectively within a team / learning / professional group, recognise, support or be proactive in leadership, negotiate in a professional context and manage conflict (International Business Management)

3. Work creatively and productively in different kinds of teams (formal, informal, project-based, committee based etc.) (Economics, Finance and International Business)

4. Interact effectively within a team / learning / professional group; recognise, support or be proactive in leadership; negotiate in a professional context and manage conflict. (Business Management)

5. The ability to communicate, negotiate and network and teamwork in a multi-agency and multi-professional settings; creative problem-solving (Planning and Property Development)

6. Facilitate and support developing practitioners (Nursing Studies)7. Enhance and support self-development and the learning of others in the

workplace (Health and Social Care)8. The ability to participate effectively in group or team work, such as large-scale

ensemble performances (e.g. University choir) or a smaller-scale ensemble (e.g. string quartet, rock band or small vocal group) (Music)

9. Work effectively as part of a small research team and equally effectively as an independent researcher (Geography)

10. The ability to develop and use interpersonal communication, presentation and team working skills along with various other enterprise skills. (Computer Aided Mechanical Engineering)

11. Demonstrate psychomotor skills and critically reflect on the interpersonal skills required to meet to meet individual needs of mothers in different situations throughout pregnancy, labour and postnatal period (Midwifery)

12. Use leadership skills to supervise and manage others and contribute to planning, designing, delivering and improving future services. (Mental Health Nursing)

13. Work effectively in partnership with individuals, families, groups, communities and other health and social care providers to enable issues which affect the health and well-being of clients to be addressed effectively. (Occupational Therapy)

14. Work effectively as part of a multidisciplinary team (Physiotherapy)15. Demonstrate effectiveness as a problem solver, team worker and sensitive

mediator in early childhood contexts and settings (Early Childhood Studies)16. Demonstrate awareness of the multidisciplinary nature of early years work, and

the importance of respecting the views of other professionals (Early Childhood Studies)

The specific context in which graduates would be expected to employ interpersonal skills was also frequently defined in these 16 contextualised PLOs, such as in a multi-agency setting (item 5) or multidisciplinary (items 14 and 16) settings, a research team (item 9) in project (item 10) or committee based (item 3) teams, in ensemble performances (item 8) or in supervising others (items 6 and 12). Making these benefits clear to students is likely to be important.

3.3.4 Gap analysis for personal literacy

Overall, there is some cross over between the elements of this attribute, which is reflected in the wording of some learning outcomes. History, for example, explains that

15

critical self-awareness is necessary in order to become an independent learner in this PLO

Set and organise personal objectives, work to briefs and deadlines and evaluate performance through a process of critical self-reflection.

Similarly, Computer Science recognize that critical self awareness is needed in the development of interpersonal skills

Apply self-awareness in evaluating their impact in team based work and utilise appropriate communication and problem resolution strategies.

This is unproblematic as breaking down the attribute into these elements is only intended to aid discussion and understanding of the attribute. However, it does raise the question of whether the attribute should be title ‘critical self-awareness and personal literacy’ (my emphasis), or whether critical self-awareness is integrated across all subelements of personal literacy.

A first review of PLOs which had been listed by programme teams under the heading critical self-awareness and personal literacy but not coded against any of its subelements found that some related more explicitly to other attributes, particularly academic literacy and global citizenship. However, of more interest were a group concerned with communication skills (including presentation skills), which programme teams had thought worthy of an explicit mention, but which are not specified in our current SESE definition. Adding ‘communication’ as a sub element of interpersonal skills identified 32 explicit references to communication skills across 27 programmes.

The final query took an overview of the items coded for personal literacy and its sub elements in the Programme Specifications (after communication had been added). This showed that there were 69 PLOs coded for metacognition, 100 for independence and 117 to interpersonal skills (see Table 7, Appendix 2 for more details)

3.3.5 Summary of personal literacy

This analysis to explore the SESE definition of personal literacy shows that:

Reflection is seen as an important process in coming to understanding and improve your learning and practice

Disciplinary is often asserted through the specification of the context in which knowledge and skills and expected to be applied.

Communication is an important and valued interpersonal skills

The constructs ‘adaptive expertise’ and ‘emotional intelligence’ are not often used by programme teams

Critical self awareness is embedded throughout this literacy

A suggested rewording of Personal Literacy, taking into account the preceding analysis might be:

Critical self awareness and personal literacy“Understanding how one learns, the ability to assess the work of oneself and others, and to identify one’s strengths and weaknesses. The ability to organise oneself and perform as an autonomous, effective and independent learner. The ability to relate to other people and function collaboratively in diverse groups, including the development of appropriate communication and interpersonal skills, emotional intelligence and adaptive expertise.”

16

4. Summary and recommendations

4.1 Achievements Not all of the achievements were anticipated. Indeed some are not directly related to graduate attributes.

a The learning outcomes for each programme were considered, rewritten, improved and the numbers reduced.

a There is now a Programme Specification for every programme. These are more up to date, more student oriented, and more succinct. They have been foregrounded at the start of the Programme Handbook (rather than as an appendix), emphasising their external and student audiences.

b Conversations about the role and nature of the graduate attributes did happen in some places.

c Some programmes are taking advantage of the opportunity for course renewal and are attending to this in the next periodic review or as their PETAL theme.

d Ongoing development of the definitions of each of the graduate attributes.

4.2 Recommendations from the reflections on the implementation processSuggestions for further embedding of the graduate attributes:

b APQO to advise on the degree of consistency in format, style necessary in Programme Specifications.

c Training for chairs and members of validation and periodic review panels to include looking at the construction of programme level learning outcomes andthe graduate attributes.

d For the time being, while graduate attributes are still being embedded, programmes preparing for periodic review and new programmes preparing for validation be encouraged to submit draft proposals for their PLOs to their OCSLD faculty link for feedback as a critical friend.

e Encourage the PLSEs to share their knowledge and expertise of graduate attributes.

f Encourage Programme Leads to reflect on the process of team working for programme design in their Annual Review document. The 2012 template Section 1b asks for a commentary on the progress made/successes in the implementation of the changes needed to fully address the graduate attributes. (see http://www.brookes.ac.uk/asa/apqu/handbook/annual_review/home.html)

4.2 Recommendations from the analysis of programme learning outcomesIt is important to remember that that mapping exercise was intended to be the first step in a programme of curriculum development. It was expected that in many cases existing PLOs would only imperfectly map to the gradate attribute headings. The intention was that programme teams would attend to improving their PLOs subsequently through regular course review and quality assurance procedures. We hope that teams can use the gap analysis and tables of examples (Appendix 2) to reflect on how they might further develop their narratives for stakeholders about how graduate attributes are developed in their programmes.

It is probably premature to make a firm recommendation for changes to the wording of our graduate attributes until we have thoroughly analysed all five of them. Nevertheless at this early stage we note that there may be a case for some revisions to the wording of research literacy and personal literacy.

17

5. Appendices

Appendix 1 Methodology

This section presents details on how the taxonomies and commentaries were created.

Documentation collection and preparation

The undergraduate specifications were made available on the APQO website (https://www.brookes.ac.uk/about/directorates/asa/registry/apqo/programme-specifications/ug-specifications) on 8 January 2013.

The Programme Specification documents were imported into the nVivo qualitative data analysis software. Each document (source) was classified according to the following attributes and values.

Table 2: Source classification for documents imported into nVivo

Attributes Values

Faculty BUS, HSS, HLS, TDE

Level UG, PG, Foundation

Document Type Prog Spec, Mapping Doc, Narrative, Reflection

JACS 1 Social Studies, Business & Admin, Mass communications and documentation, Biological Sciences, Geographical Studies, Subjects allied to medicine, Education, Languages, Historical and philosophical studies, Law, Computer Science, Engineering and Technology, Mathematical sciences, Architecture, building and planning, Creative arts and design

The data set for the analysis presented in this interim report comprised 90 undergraduate Programme Specifications leading to BA/BSc awards, listed in Table 3.

18

Table 3: Programme Specifications used in this analysis

AB Animal Biology and Conservation LC LD BA English Lang CommunicationAG BSc Motorsport Technology LL LawAM BSc Mathematics LN Electronic EngineeringAN Anthropology combined honours MA BA BSc Mathematics combinedAR BA Architecture MD BA MusicAX BA Anthropology Single Honours MF BEng Motorsport EngineeringBK BA Business and Marketing Management MH BEng Mechanical EngineeringBT BA International Business Management ML BSc Mathematical SciencesBU BA Business and Management MN BSc Computer Aided Mechanical EngineeringBV BSc Biomedical Science MO BSc Multimedia ProductionDA BSc Accounting and Business MT BSc Media TechnologyDB BSc Business MY BSc MidwiferyDF BSc Economics, Finance and International Business

NB BA Mental Health Nursing

DG BSc Accounting NJ BSc Children's NursingDG BSc Accounting and Finance NK BSc Network ComputingDN BSc Accounting and Info Management NN BSc NutritionDP BA Business Management NQ Children's and mental health nursingDV BA Planning and Property Development NT BSc Adult NursingDW BA Drama NV BSC Adult and Mental Health NursingED15 BA Primary Teacher Education OS BOst OsteopathyED20 BA Primary Teacher Ed WBL PA BA Publishing MediaEF BEng Automotive Engineering PJ BA PublishingEM BSc Real Estate Management PK PQ BA PhilosophyEJ Environmental Sciences PO BA BSc PoliticsFC BA French Studies (Advanced) PS PX BSc PsychologyFE BA French Studies (Intermediate) PZ BSc PhysiotherapyFT&FM BA Film Studies QA BSc Quantity Surveying and Comm ManFX FD BA Fine ArtGG Geography QM BSc Construction Project ManGM BSc Computer games and animation RL BA BSc International RelationsHA HD BA History of Art RP BA City and Regional PlanningHB BSc Human Biology RX BA International Relations & PoliticsHC10 BSc Nursing Studies SD BSc Computer ScienceHC30 BA Student Designed Award SH BSc Software EngineeringHF BSc Health and Social Care SJ SU BSc sports coachingHI HT BA History SK BA Social WorkHO BSc International Hospitality Management SO BA BSc SociologyHX HQ BA Communication Media Culture ST BA BSc Statistics combinedIC BSc Information Technology Management for Business

SXA SXI BSc Spanish

ID BSc International Business SY BSc Sound Technology and Digital MusicIH BA Interior Architecture U736 BA Religion and TheologyIO Biological Sciences WE WD BA Education StudiesIL Mobile Computing X315 BA Early Childhood StudiesIX Economics, Political and IR XE EA BA EnglishJB Japanese Studies XS BSc sport and exercise

19

Coding

The nodes and subnodes for coding were created in advance, using the enquiry questions as a guide. Data were coded for each graduate attribute (node) and its elements (subnode). Coding nodes were set up to aggregate, so coding at e.g. research literacy automatically codes a level up to ‘UG attributes’. Coding was informed by, but not restricted to the placing of PLOs under the subheadings of the GAs, i.e. PLOs could be coded against multiple GAs, which may or may not include those that they had been listed under.

Table 4: Nodes and subnodes used to code the graduate attributesNode SubnodesAcademic LiteracyResearch Literacy Critical consumer of research

Undertake small scale researchUsing appropriate methodology

Digital and Information LiteracyGlobal Citizenship Global context: Knowledge and skills showing cross-

cultural awareness, valuing human diversity, the ability to work effectively, and responsibly, in a global context.

Values: Knowledge of global perspectives on how disciplinary knowledge is represented and understood within other cultures. Cross-cultural capability, beginning with an awareness of our own culture and perspectives.

Culture: Development of the confidence to question one’s own values and those of others responsibly and ethically, responsible citizenship, actively engaging with issues of equity and social justice, sustainability and the reduction of prejudice, stereotyping and discrimination

Critical self awareness and personal literacy

Metacognition: Understanding how one learns, the ability to assess the work of oneself and others, and to identify one’s strengths and weaknesses.

Independence: The ability to organise oneself and perform as an autonomous, effective and independent learner.

Interpersonal skills: The ability to relate to other people and function collaboratively in diverse groups, including the development of appropriate interpersonal skills, emotional intelligence and adaptive expertise.

Three additional codes were used to highlight extracts from programme documentation that could be used as exemplars:

● articulated: an example that has been well articulated, i.e. that provides an expression of the attribute that goes beyond the SESE definition.

● contextualised: an example of a GA has been contextualised for the discipline● teaching: an example that specifies the opportunities students will have to

develop that GA.

20

Enquiry questions, nVivo queries and analysis

There are two overarching aims of this evaluation which drove the enquiry questions and subsequent queries and analysis. First, to what extent are the graduate attributes visible in programme documentation? Second, to what extent have the graduate attributes been contextualised for the discipline? We are interested in questions such as ‘what does it mean to be a global citizen in the life sciences?’ or ‘what does it mean to be digitally literate in the creative arts?’. The remaining enquiry questions are preparing for the subsequent phases of the evaluation.

Table 5: Enquiry questions and how they led to queries run in nVivo

Enquiry question Query run in nVivo Analysis

What examples do we have of programme learning outcomes which articulate the graduate attributes well?

Which graduates attributes are easy and which are difficult to articulate?

Search for content coded for articulated and for each GA and its subnodes.

These are the examples used in the commentaries on each of the GAs.

Note where most hits occur.

How are each of the graduate attributes expressed within the context of the discipline?

What similarities and differences do we find between the disciplines in their interpretation of graduate attributes?

Matrix coding query with subnodes of the GA (columns) against JACS codes (rows). Repeat for each GA.

Search for content coded for contextualised and for each GA and its subnodes.

Select from hits to produce the tables of examples. Selection to include examples from a range of programmes.

Commentaries on each of the Gas.

Where is there evidence of a graduate attribute being taught within a programme?

Search for content coded for teaching for each GA.

Identify programmes which could be case studies in Years 2 and 3 of the evaluation.

How suitable is our existing model of digital literacy? How far up the model/triangle are students expected to progress?

Search for content coded for the Digital Literacy subnodes: access, skills, practices and attributes.

+ invivo coding of new elements of digital literacy.

Testing of the model (Beetham & Sharpe, 2009) developed from the JISC learner experience studies which informed the Brookes definition of digital literacy.

Are the SESE definitions of each of the graduate attributes sufficient?

Search for content coded for articulated and each element (subnode) of each GA.

+ invivo coding of new elements of each GA.

Create measurement instruments for each GA so that in later years we could ask/measure students’ achievements of the GAs directly.

21

Appendix 2: Tools for programme teams

Table 6. Gap analysis for research literacy

Critical consumer of

research

small scale research project Methodology

1 : AB Animal Biology and Conservation BSc 1 1 12 : AG BSc Motorsport Technology 0 0 03 : AM BSc Mathematics 0 0 04 : AN Anthropology combined honours 2 1 35 : AR BA Architecture 1 0 06 : AX BA Anthropology Single Honours 1 1 27 : BK BA Business and Marketing 0 1 08 : BT BA International Business Management 0 0 29 : BU BA Business and Management 0 0 210 : BV BSc Biomedical Science 2 5 011 : DA BSc Accounting and Business 1 1 112 : DB BSc Business 0 0 113 : DF BSc Economics, Finance and International Business 0 0 114 : DG BSc Accounting 1 1 015 : DG BSc Accounting and Finance 0 1 016 : DN BSc Accounting and Info Management 0 1 017 : DP BA Business Management 1 0 118 : DV BA Planning and Property Development 1 1 019 : DW BA Drama 3 1 020 : ED15 BA Primary Teacher Education 2 1 021 : ED20 BA Primary Teacher Ed WBL 2 1 022 : EF BEng Automotive Engineering 0 0 023 : EJ Environmental Sciences BSc 1 3 224 : EM BSc Real Estate Management 0 1 025 : FC BA French Studies 1 0 026 : FE BA French Studies Combined 2 0 027 : FT&FM BA Film Studies 1 0 128 : FX FD BA Fine Art 1 0 129 : GG Geography 0 2 230 : GM BSc Computer games and animation 2 1 031 : HA HD BA History of Art 1 2 032 : HB BSc Human Biology 1 1 133 : HC10 BSc Nursing Studies 2 0 134 : HC30 BA Student Designed Award 3 3 235 : HF BSc Health and Social Care 1 0 136 : HI HT BA History 2 0 137 : HO BSc International Hospitality Management 3 2 1

22

Critical consumer of

research

small scale research project Methodology

38 : HX HQ BA Communication Media Culture 1 0 039 : IC BSc Information Technology Management for Business 1 2 040 : ID BSc International Business 0 0 241 : IH BA Interior Architecture 0 0 042 : IL Mobile Computing 1 1 043 : IO Biological Sciences BSc 2 1 144 : IX Economics, Political and IR 1 0 145 : JB Japanese Studies 1 1 046 : LC LD BA English Lang Communication 0 0 247 : LL LW Law BA 1 1 248 : LN Electronic Engineering BSc 1 0 249 : MA BA BSc Mathematics combined 0 0 150 : MD BA Music 2 0 051 : MF BEng Motorsport Engineering 0 0 052 : MH BEng Mechanical Engineering 0 0 053 : ML BSc Mathematical Sciences 0 0 054 : MN BSc Computer Aided Mech Eng 0 0 055 : MO BSc Multimedia Production 0 0 156 : MT BSc Media Technology 1 0 057 : MY BSc Midwifery 6 0 058 : NB BA Mental Health Nursing 1 0 059 : NJ BSc Children's Nursing 1 0 060 : NK BSc Network Computing 1 0 061 : NN BSc Nutrition 2 2 262 : NQ Children's and mental health nursing 1 0 063 : NT BSc Adult Nursing 1 0 064 : NV BSC Adult and Mental Health Nursing 1 0 065 : OS BOst Osteopathy 1 2 066 : OT BSc Occupational Therapy 1 1 067 : PA BA Publishing Media 1 2 068 : PJ BA Publishing 0 1 069 : PK PQ BA Philosophy 2 0 170 : PO BA BSc Politics 2 0 171 : PS PX BSc Psychology 0 3 472 : PZ BSc Physiotherapy 1 2 073 : QA BSc Quantity Surveying and Comm Man 0 0 074 : QM BSc Construction Project Man 0 0 075 : RL BA BSc International Relations 2 0 176 : RP BA City and Regional Planning 0 0 177 : RX BA International Relations & Politics 2 1 178 : SD BSc Computer Science 1 0 079 : SH BSc Software Engineering 1 0 1

23

Critical consumer of

research

small scale research project Methodology

80 : SJ SU BSc Sports Coaching 2 1 081 : SK BA Social Work 1 0 182 : SO BA BSc Sociology 0 1 183 : ST BA BSc Statistics combined 0 0 084 : SXA SXI BSc Spanish 2 0 085 : SY BSc Sound Technology and Digital Music 0 1 186 : U736 BA Religion and Theology 3 0 187 : WE WD BA Education Studies 2 0 188 : X315 BA Early Childhood Studies 0 0 089 : XE EA BA English 3 1 090 : XS BSc Sport and Exercise 1 4 3

Totals 93 60 59

24

Table 7. Gap analysis for personal literacy metacogniti

on independenceinterpersonal

skills1 : AB Animal Biology and Conservation BSc 1 1 12 : AG BSc Motorsport Technology 0 3 33 : AM BSc Mathematics 0 3 14 : AN Anthropology combined honours 0 1 25 : AR BA Architecture 0 2 46 : AX BA Anthropology Single Honours 0 1 27 : BK BA Business and Marketing 1 0 18 : BT BA International Business Mgt 1 1 19 : BU BA Business and Mgt 1 1 110 : BV BSc Biomedical Science 0 1 111 : DA BSc Accounting and Business 1 1 212 : DB BSc Business 1 1 113 : DF BSc Economics, Finance and International Business 1 1 114 : DG BSc Accounting 1 1 015 : DG BSc Accounting and Finance 1 1 016 : DN BSc Accounting and Info Mgt 1 1 017 : DP BA Business Mgt 3 1 118 : DV BA Planning and Property Dev 0 1 119 : DW BA Drama 6 3 220 : ED15 BA Primary Teacher Education 2 2 221 : ED20 BA Primary Teacher Ed WBL 2 1 122 : EF BEng Automotive Engineering 0 3 123 : EJ Environmental Sciences BSc 1 1 124 : EM BSc Real Estate Management 1 2 025 : FC BA French Studies 0 2 126 : FE BA French Studies Comibined 0 2 127 : FT&FM BA Film Studies 1 1 528 : FX FD BA Fine Art 0 1 229 : GG Geography 2 0 130 : GM BSc Computer games and animation 1 0 131 : HA HD BA History of Art August 2012 1 1 232 : HB BSc Human Biology 1 1 233 : HC10 BSc(Hons) Nursing Studies 2 0 134 : HC30 BA Student Designed Award 4 1 235 : HF BSc Health and Social Care 2 1 436 : HI HT BA History 0 1 237 : HO BSc (Hons) International Hospitality Management 1 1 038 : HX HQ BA Communication Media Culture 3 1 139 : IC BSc Information Technology Management for Business 1 0 140 : ID BSc International Business 1 1 141 : IH BA Interior Architecture 0 2 342 : IL Mobile Computing 1 0 1

metacognition independence

interpersonal skills

25

43 : IO Biological Sciences BSc 0 2 144 : IX Economics, Political and IR 1 1 145 : JB Japenese Studies 0 2 146 : LC LD BA English Lang Communication 0 1 247 : LL LW Law BA 1 3 148 : LN Electronic Engineering BSc 0 2 149 : MA BA BSc Mathematics combined 0 3 150 : MD BA Music 1 1 151 : MF BEng Motorsport Engineering 0 1 152 : MH BEng Mechanical Engineering 0 2 153 : ML BSc Mathematical Sciences 0 2 254 : MN BSc Computer Aided Mech Eng 0 2 155 : MO BSc Multimedia Production 1 0 056 : MT BSc Media Technology 0 1 057 : MY BSc Midwifery 2 0 658 : NB BA Mental Health Nursing 0 0 159 : NJ BSc Children's Nursing 0 0 160 : NK BSc Network Computing 1 0 161 : NN BSc Nutrition 1 2 162 : NQ Children's and mental health nursing 0 0 163 : NT Adult Nursing BSc Hons 0 0 164 : NV BSC Adult and Mental Health Nursing 0 0 165 : OS BOst Osteopathy 1 0 066 : OT BSc Occupational Theraphy 1 0 167 : PA BA Publishing Media 0 2 168 : PJ BA Publishing 0 2 269 : PK PQ BA Philosophy 0 1 170 : PO BA BSc Politics 0 1 171 : PS PX BSc Psychology 0 1 172 : PZ BSc Physiotherapy 1 0 273 : QA BSc Quantity Surveying and Comm Man 0 2 174 : QM BSc Construction Project Man 0 2 175 : RL BA BSc International Relations 0 1 176 : RP BA City and Regional Planning 0 0 277 : RX BA International Relations & Politics 0 1 178 : SD BSc Computer Science 1 0 179 : SH BSc Software Engineering 1 1 080 : SJ SU BSc sports coaching 1 1 181 : SK BA SocialWork 1 0 082 : SO BA BSc Sociology 0 2 283 : ST BA BSc Statistics combined 0 1 284 : SXA SXI BSc Spanish 0 2 185 : SY BSc Sound Technology and Digital Music 1 0 086 : U736 BA Religion and Theology 2 1 2

metacognition independence

interpersonal skills

87 : WE WD BA Education Studies 2 1 088 : X315 BA Early Childhood Studies 1 1 2

26

89 : XE EA BA English 1 1 190 : XS BSc sport and exercise 1 1 2Totals 69 100 117

27

Table 8: Examples of how research literacy has been defined across programmes from a variety of disciplines

Elements of the SESE definition of research literacyDiscipline(JACS code 1)

The ability to be a critical consumer of research, and also ...

... where possible, to design and undertake at least a small scale research project in the discipline ...

... using appropriate methodology

Architecture, building and planning

Critically review precedents relevant to the function, organisation, and technological strategy of design projects (BA Architecture)

Obtain, analyse, evaluate and synthesise planning and property empirical data (BA Planning and Property Development)

Research skills in data and information sourcing and collection, analysis and presentation (BA City and Regional Planning)

Biological Sciences Interpret experimental data in terms of current paradigms in the human biosciences (BSc Human Biology)

Design and conduct studies to investigate the molecular activity of cells (BSc Biomedical Science)

Competently use appropriate field and laboratory methods and techniques for the study of human responses to exercise (BSc Sport and Exercise)

Business and Administrative Studies

Collect and critically analyse information to present reasoned arguments that demonstrate knowledge and understanding of the international hospitality industry (BSc International Hospitality Management)

As an individual and in groups, design and conduct effective self-directed research into accounting concepts and practices (BSc Accounting and Business)

Structure, design and apply tools of research, analysis, decision making and evaluation to a wide range of international scenarios and problems (BA Business and Management)

Computer Science Identify and utilise trustworthy information sources, such as the ACM Digital Library to develop a coherent understanding of issues in the domain (BSc Computer games and animation)

Apply the scientific method and report findings using accepted formalisms (BSc Information Technology and Management for Business)

Apply appropriate empirical methods, such as software metrics, to study the creation and use of software systems (BSc Software Engineering)

Creative Arts and Design

Use a wide range of musicological sources and interpret and present findings in appropriate ways (BA Music)

An ability to demonstrate creative and imaginative skills through the realisation of a project (BA Drama)

Utilise appropriate research and development methodologies within Fine Art contexts (BA Fine Art)

Education Critically assess and evaluate evidence in the context of research methodologies and data sources (BSc Sports Coaching)

Undertake empirical research within an area of personal choice (BA Primary Teacher Education)

Adopt a range of research perspectives and methodologies applicable to education (BA Education Studies)

28

Discipline The ability to be a critical consumer of research, and also ...

... where possible, to design and undertake at least a small scale research project in the discipline ...

... using appropriate methodology

Engineering and Technology

Propose and evaluate multimedia solutions by researching foundations, techniques, limitations, developments, trends and applications of those areas of computing related to multimedia technology, specifically networks and the processing of images and sound (BSc Media Technology)

Design and experiment with electronic test equipment and in simple electronics manufacture (BSc Sound Technology)

Experiment with and evaluate data processing techniques - compression, transmission and editing, particularly as applied to audio and image data (BSc Sound Technology and Digital Music)

Geographical Studies Create, synthesise, analyse and interpret different types of geographical evidence (BA/BSc Geography)

Conceive, design, execute, analyse and report back upon own independent research into geographical phenomena, both as an individual and as part of a team (BA/BSc Geography)

Describe, explain, evaluate and utilise a range of technical research methods including a wide range of class, field and laboratory based techniques. (BA/BSc Geography)

Historical and Philosophical

Make discriminating use of a full range of library resources in order to identify appropriate source material, compile bibliographies, inform research and enhance presentations (BA Religion and Theology)

Conduct basic research, drawing on a range of sources, and to present the results of this research in a clear, coherent and critical manner (BA Philosophy)

Select and deploy research methods and skills appropriate to a broad range of sources and media, including library collections, archival holdings, visual ad textual sources and statistical data (BA History)

Languages Reflect critically on and interpret French texts with reference to their social and cultural background and significance (BA French Studies)

Research, using scholarly information retrieval skills, involving the ability to gather, sift and organise material independently and critically (BA English)

Have knowledge and understanding of the rationale and techniques for the selection, analysis and interpretation of data (BA English Language Communication)

Law Identify, locate and retrieve standard legal materials in English law in paper and electronic format (BA Law)

Undertake independent research with minimal guidance (BA Law)

Demonstrate a good understanding of principles and techniques of research in English Law (BA Law)

Mass Communications and Documentation

Make effective use of a variety of sources and to evaluate these critically (BA Film Studies)

Undertake substantial individual supervised study demonstrating clearly defined aims and objectives, data collection, analysis and evaluation and reasoned conclusions (BA Publishing)

A grounded awareness of a broad range of research methods and professional practices and the discerning judgement to apply these appropriately to specific projects (BA Film Studies)

Mathematical Sciences

Apply acquired knowledge to novel situations in a variety of analytical contexts (BSc Mathematics)

Extend the subject knowledge base (BSc Mathematics)

Select and apply mathematical problem solving techniques and experience in analysing and evaluating these (BA/BSc Mathematics Com)

29

Discipline The ability to be a critical consumer of research, and also ...

... where possible, to design and undertake at least a small scale research project in the discipline ...

... using appropriate methodology

Social Studies Read and interpret texts (print, oral, film, multimedia) within their historical, social and theoretical contexts (Anthropology)

Formulate and investigate sociologically informed questions using a range of research strategies and methods of sociological enquiry (BA BSc Sociology)

Select and critically apply research methods (both quantitative and qualitative) appropriate to the study of economic and financial phenomena and be able to use such methods to develop creative solutions to a wide range of practical problems arising in the context of markets and organisations (BSc Economics, Finance and International Business)

Subjects allied to medicine

Evaluate and critique health care research in order to provide evidence-based practice (BSc Nursing Studies)

Undertake research in order to develop and inform practice (BOst Osteopathy)

Appraise research methodology including aspects of experimental design, safety, statistical interpretation of data, ethical considerations and means of reporting findings (BSc Nutrition)

30

Table 9: Examples of how critical self-awareness and personal literacy has been defined across programmes from a variety of disciplines (note that some of the PLOs listed below are shared by a number of programmes within the discipline)

Elements of the SESE definition of Critical self-awareness and personal literacy

Discipline(JACS code 1)

Metacognition Independence Interpersonal skills

Architecture, building and planning

Be self-aware and competent in self-management (BSc Real Estate Management)

Manage time and work to deadlines (BA Interior Architecture)

Communicate, negotiate and network and teamwork in a multi-agency and multi-professional settings (BA Planning and Property Development)

Biological Sciences Reflect on outcomes in order to evaluate the performance of oneself and others (BSc Sport and Exercise)

Demonstrate confidence and flexibility as independent learners to enhance employability and knowledge (BSc Human Biology)

Collaborate effectively with others to achieve common goals (BSc Psychology)

Business and Administrative Studies

Act as a self directed learner able to learn from reflection on experiences of study and employment (BSc Accounting and Finance)

With minimum guidance manage own learning, exercising initiative and personal responsibility (BA International Business Management)

Engage effectively within a team / learning / professional group, recognise, support or be proactive in leadership, negotiate in a professional context and manage conflict (BA Business and Management)

Computer Science Evaluate and reflect on the evolution of their strengths and weaknesses across the range of subject based competences involved in their chosen domain through the creation and implementation of a discipline based personal development plan (BSc Computer Games and Animation)

Apply self-awareness in evaluating their impact in team based work and utilise appropriate communication and problem resolution strategies. (BSc Computer Science)

Creative Arts and Design

Reflect in a critical and productive way on their practice and transform their practice as a result of such critical self-evaluation. (BA Drama)

Exercise initiative and personal responsibility in creative and academic practice (BA Drama)

Participate effectively in group or team work, such as large-scale ensemble performances (e.g. University choir) or a smaller-scale ensemble (e.g. string quartet, rock band or small vocal group). (BA Music)

Education Undertake systematic critical self-reflection, within the planning, teaching and assessment cycle. (BA Primary Teacher Education)

Learn both independently and co-operatively by developing an ability to set personal objectives, manage time and tasks (Early Childhood Studies)

Demonstrate awareness of the multidisciplinary nature of early years work, and the importance of respecting the views of other professionals (BA Early Childhood Studies)

Engineering and Technology

Evaluate personal performance in applying theory to practice in the use of Sound Technology and Digital Music equipment (BSc Sound Technology and Digital Music)

Employ organisational skills at both the personal level and in the areas of project and team management (BEng Motorsport Engineering)

Develop and use interpersonal communication, presentation and teamworking skills along with various other enterprise skills (BSc Computer Aided Mech Eng)

31

Discipline Metacognition Independence Interpersonal skillsGeographical Studies

Identify and evaluate their strengths and weaknesses, including an awareness of their various ethical positions (BA/BSc Geography)

Work effectively as part of a small research team and equally effectively as an independent researcher (BA/BSc Geography)

Work effectively as part of a small research team and equally effectively as an independent researcher (BA/BSc Geography)

Historical and Philosophical

Show independence in thought and critical self-awareness about one’s own beliefs, commitments and prejudices. (BA Religion and Theology)

Set and organise personal objectives, work to briefs and deadlines and evaluate performance through a process of critical self-reflection (BA History)

Listen actively, so as to be able to participate in and learn from discussion, and persuade rationally, through negotiation and defence of position, while respecting other people’s stances (BA History)

Languages Reflect critically upon their learning experiences in English Studies (BA English)

Work diligently, to fulfil briefs and meet deadlines, and to manage one’s time and tasks effectively (BSc Spanish)

Develop effective personal interaction, group-working and research skills appropriate to an undergraduate programme of study (BA English Language Communication)

Mass Communications and Documentation

Apply creative and production processes competently and reflect upon these in a critical manner that is underpinned by relevant theory (BA Film Studies)

Manage their own learning by clarifying personal values, setting personal objectives, managing time and tasks and evaluating their own performance (Publishing)

Work collaboratively and productively in a diverse team, taking leadership and support roles (Publishing)

Mathematical Sciences

The ability to select and apply mathematical problem solving techniques and experience in analysing and evaluating these (BSc Mathematics)

Effective self management (Mathematical Sciences and Statistics)

The ability to work in a team (Mathematical Sciences and Statistics)

Social Studies Demonstrate habits of critical reflection on their performance and take responsibility for modifying action in the light of this (BA Social Work)

Set goals and manage time and tasks effectively (BSc Sociology)

Work creatively and productively in different kinds of teams (formal, informal, project-based, committee based etc). (BSc Economics, Finance and International Business)

Subjects allied to medicine

Critically reflect on the factors that influence practice and apply these to individual situations (BSc Nursing)

Plan, justify and manage an individual programme of learning to meet own identified learning needs (BA Student Design Award)

Use leadership skills to supervise and manage others and contribute to the planning, designing, delivering and improving of future services (Mental Health and Children’s Nursing Programmes)

32