Soft SkillsSoft SkillsSystems / Teaching / AssessmentSystems / Teaching / Assessment

Kostas Kechagias

Presentation in 2nd MASS meeting

Work Package 4, Needs Analysis / Piloting

June 2010

ContentsContents

• Skills and Skills Frameworks• Teaching Skills and Competences

• Skills Assessment

• Experimental Methodology

• Conclusions/Homework

Skills and Skills Frameworks (1/13)Skills and Skills Frameworks (1/13)…towards definitions (DeSeCo)• great interest for identification of common skills• must be connected to success in life• the definition should address how they are used

Definition: skills or abilities individuals need in order to achieve success in life, within the context of their socio-cultural milieu, through adaptation to, shaping of, and selection of environments

• Often they are used interchangeably or in a vague sense

Skills and Skills Frameworks (2/13)Skills and Skills Frameworks (2/13)• a proliferation of efforts to develop lists of skills,

knowledge, and competencies necessary for success in the workplace and society

• “basic skills” are not sufficient for success in the workplace

• …“employability skills” • …“enabling skills,” “generic skills,” “core skills,”

“key competencies,” “ essential skills,” and “necessary skills.”

• methodology: start from a broad definition and then survey or observes workers, supervisors, and experts to determine what skills are common

Skills and Skills Frameworks (3/13)Skills and Skills Frameworks (3/13)SCANS• 1991, US president and Education Secretary,

America 2000• seven skill groups

• Influence: Organizational Effectiveness and leadership• Group effectiveness: Interpersonal skills, negotiation and

teamwork• Personal management: Self-esteem, goal setting/motivation,

and personal and career development• Adaptability: Creative thinking and problem solving• Communication: Listening and oral communication• Competence: Reading, writing and computation

• The foundation: Learning to learn.

Skills and Skills Frameworks (4/13)Skills and Skills Frameworks (4/13)Michigan• Center for Career and Technical Education,

Michigan State University• human relations skills

• Positive work attitude,• Teamwork• Listening• Honesty and Idengrity• Demonstrate initiative• Handling pressure & tension• Communication• …

• conceptual skills• technical skills

Skills and Skills Frameworks (5/13)Skills and Skills Frameworks (5/13)Scottish Qualifications Authority, 2008• Problem Solving • Communication• Numeracy• Information and Communication Technology • Working with Others

• Component: Working Co-operatively with Others • identifying, with support, your own role and the roles of other

people, and helping to achieve a shared goal to:

• analyzing the roles and behavior of others and adapting your own behavior to deal with the complexity of changing and challenging dynamics

Skills and Skills Frameworks (6/13)Skills and Skills Frameworks (6/13)DeSeCo (Definition and Selection of Competencies)• OECD, 1997• A competency is more than just knowledge and

skills. It involves the ability to meet complex demands, by drawing on and mobilising psychosocial resources (including skills and attitudes) in a particular context.

Skills and Skills Frameworks (7/13)Skills and Skills Frameworks (7/13)• Using Tools Interactively

• use language, symbols and text interactively • use knowledge and information interactively• use technology interactively

• Interacting in Heterogeneous Groups• relate well to others• cooperate• manage and resolve conflicts

• Acting Autonomously• act within the big picture• form and conduct life plans and personal projects• assert rights, interests, limits and needs

Skills and Skills Frameworks (8/13)Skills and Skills Frameworks (8/13)ALL (Adult Literacy and Life skills survey project)• OECD, 2005• employability skills derived from job and task

analysis and psychological theories of intelligence• scientifically convincing theory and an established

literature• accompanying tradition of measurement • could be learnt and, therefore, taught• had a direct impact on the social, health,

educational or economic life chances of individuals

Skills and Skills Frameworks (9/13)Skills and Skills Frameworks (9/13)European Qualification Framework• “Education and Training 2010” (2004)

• communication in the mother tongue• communication in foreign languages • mathematical competence and basic competences in

science and technology• digital competence • learning to learn • social and civic competences• sense of initiative and entrepreneurship • cultural awareness and expression

Skills and Skills Frameworks (10/13)Skills and Skills Frameworks (10/13)MASS Framework• we don’t have much information

Manners Ownership of tasks Attendance Motivation Professionalism Work output Conduct in workplace Timekeeping

Verbal Communication Organisation/ planning Team-working/ Respect Helping others Conscientiousness Ability to ask for help Adaptability/ Flexibility

• don’t seem to fit in any studied framework

Skills and Skills Frameworks (11/13)Skills and Skills Frameworks (11/13)• No grouping of skills is provided• Some skills may be fit in more than one

categories in other skill frameworks, e.g.• Motivation

• Taking on new challenges, working hard to achieve goals, thinking of new ways to do things

• Under SCANS contains elements from categories• Personal Qualities” (taking on new challenges) • “Thinking Skills” (thinking of new ways to do things)

• Seems to be closely related to “Proffesionalism”• Working to a high standard, being consistent in attitude (not allowing

emotions or personalities to influence you)

Skills and Skills Frameworks (12/13)Skills and Skills Frameworks (12/13)• Most frameworks are concerning higher education

graduates• One may argue that the target group is different,

so the skills to be considered are different• however, this should be established in the grounds of

empirical research of literature

• The skills should be defined in such a way as to minimize overlap

• Failure to do so may have impact to teaching, but mainly to assessment.

Skills and Skills Frameworks (13/13)Skills and Skills Frameworks (13/13)Questions:• What is the framework on which the MASS skills

are based?• What justifies the selection of the particular skills

as the most important ones?• If there is not a given one, referred from the

literature, what are the empirical studies that support the selection?

• Are the skills chosen in such a way as the interdependencies are minimized?

• Is the selection justified from the view point of the assessment?

Teaching Skills and Competences (1/9)Teaching Skills and Competences (1/9)Gonczi, 2003 (DeSeCo)• The best examples: active independent learning, especially in ways

that simulated contexts they might experience in later life

• those that integrated the key competencies into the teaching of other material, rather than attempting to teach them as stand-alone entities

• seemed to be more success in student outcomes when the key competencies were made explicit

• in many instances, it was not possible to separate out the key competencies in the process of teaching and learning.

• very few of the projects in vocational education specifically assessed the acquisition of key competencies directly. The most comprehensive attempt to assess them was via a portfolio of evidence which students collected for themselves.

Teaching Skills and Competences (2/9)Teaching Skills and Competences (2/9)

Welsh & Canning, 2003 (SQA)• The prospect of the isolation of Core Skills, as a

separately taught and assessed subject, would appear to be unpopular with candidates. From the literature, it can be seen that this is also unpopular with employers.

• Completely separate delivery and assessment often not popular with candidates as it’s ‘tantamount to a return to school’

Teaching Skills and Competences (3/9)Teaching Skills and Competences (3/9)

Shuman, et. all., 2005 (ABET)• The choice of team task depends on the

educational objectives• if the objective is to have students experience a

specific aspect of teaming such as brainstorming, specific group exercises new ideas can be used

• if the objective is for students to experience a broad range of team processes and behaviors, then more complex activities are suitable

Teaching Skills and Competences (4/9)Teaching Skills and Competences (4/9)Shuman, et. all., 2005 (ABET)• Two guiding principles should be followed in

choosing activities: fidelity and complexity• Fidelity: similarity of the training situation to the

students’ present and future working conditions. • The higher the fidelity, the more superior the transfer of

learning to the workplace • Complexity is defined by two subfactors: task

interdependence and cognitive effort. The more complex the activity, the more team skills are required by the participant.

Teaching Skills and Competences (5/9)Teaching Skills and Competences (5/9)Shuman, et. all., 2005 (ABET)• advantages and disadvantages:

• fidelity/high complexity activities: • most resemble real workplace” conditions, • typically are more difficult for the instructor to manage, • resource intensive, • time consuming for the student.

• lower in fidelity/ low complexity activities • typically more structured and easier to administer, • may be perceived as less relevant by the student, • the experience having less of a learning impact. Finally, • may not challenge the team nor provide the environment

necessary for intense interaction among team members

Teaching Skills and Competences (6/9)Teaching Skills and Competences (6/9)Teaching methods (Szul, 2002)• Coaching• Project Work (e.g. Newsletter)• Portfolio• Job-shadowing experiences• Videotapes/Observations• Interviews• Case studies• Role playing

Teaching Skills and Competences (7/9)Teaching Skills and Competences (7/9)MASS Teaching• seems to be standalone course

• justification needed• Interdisciplinary

• 18 weeks of approximately 4 hours/week• partners should be consulted for the feasibility

Teaching Skills and Competences (8/9)Teaching Skills and Competences (8/9)• The activities that the 1st learning byte contains

belong to the following categories:• Presentation: 4 times• Group discussion: 2 times• Directed Discussion (Class Discussion): 6 times• Quiz: 2 times• Observation: 1 time• Individual work: 1 time• Personal discussion with the teacher: 1 time

• Mainly teacher centered designed• “Hopefully, your grades will be improved”

Teaching Skills and Competences (9/9)Teaching Skills and Competences (9/9)• the design of the lessons seems a lot like the

designs of the vocational training• Question about their efficiency• Excellent quality of materials, within the concept

• instruction based not project based• question whether we train on the knowledge of

soft skills rather than soft skills themselves• In some places, as in formation of student groups,

more elaboration is needed

Assessment (1/14)Assessment (1/14)indicators and outcomes• indicators are the means by which we can

measure whether the outcomes have been achieved• achievements which may ‘indicate’ acquisition or

progress towards an outcome

• Not all indicators will be suitable for all target groups, and some will be target group specific.

Assessment (2/14)Assessment (2/14)• Methods for collection of information on soft

outcomes• Individual action planning, personal action planning

and goal setting• Reviews between trainers/assessors and clients to

record soft outcomes• Daily diary or personal journal• In-depth reflection during or after the course• Recorded observations of group or individual

activities• Presentation of material in a portfolio• Tests

Assessment (3/14)Assessment (3/14)• requirements from the tools

• reliability• Validity

• Requirements from the definitions• two ends

• so general that it becomes totally abstract and refers more to a system of values than to abilities

• less comprehensive in one’s ambitions and is content with assessing a few specific competencies in a necessarily limited context the

Assessment (4/14)Assessment (4/14)• Competencies do not exist in themselves; they can

only be demonstrated when an individual is acting in a specific context.

• pieces of research have shown that the subject’s performance is largely dependent on the contents of the task, on any previous training the subject may have had, on the type of strategy resorted to, and on the formal characteristics of the task

• the same tasks undertaken in different situations—such as work situations or job interview situations—can yield very different results

Assessment (5/14)Assessment (5/14)Teamwork (Baker, et al., 2005) (ALL Framework)• Focus towards identifying the specific

competency requirements of team members• “competencies” a cluster of related knowledge,

skills, and attitudes that affects a major part of one’s job • is correlated with performance on the job• can be measured against well-accepted standards• can be improved through training and development.

Assessment (6/14)Assessment (6/14)• three types of competencies that are central for

effective teamwork Cannon-Bowers et al. (1995) • team knowledge competencies,

• the principles and concepts that underlie a team’s effective task performance

• team skill competencies, • learned capacity to interact with other team members at

some minimal proficiency level• adaptability, situation awareness, performance

monitoring/feedback, leadership, interpersonal relations, coordination, communication, and decision making

• team attitude competencies• an internal state that influences a team member’s choices

or decisions to act in a particular way• Positive attitudes toward teamwork and mutual trust

among team members

Assessment (7/14)Assessment (7/14)ALL questionnaire for teamwork • Evaluation of teamwork requires observation• Target group: International, adults• Examines

• knowledge about team skills, • attitudes toward teamwork• historical experience with teamwork

Assessment (8/14)Assessment (8/14)

Assessment (9/14)Assessment (9/14)Zhuang, et. all., & Roberts, 2008 (ETS)• Questionnaire

• target at high school students (mean age 16.10) undertaking Ford PASS courses

• consists of reliable and qualitatively distinct factors• involves a variety of methods• has demonstrable validity evidence

Assessment (10/14)Assessment (10/14)• Contains

• Self-report ratings (57 items)• cooperation with others (15 items), • advocate and influence (12 items), • resolve • conflict/negotiate (17 items), • guiding others (13 items).

• Situational judgment tests (SJTs)• eight scenarios describing various situations and required

participants to evaluate the effectiveness of four reactions to each situation

• Teacher ratings

Assessment (11/14)Assessment (11/14)MASS assessment• one might form the impression of an underlying

assumption that each skill, can be assessed separately• Is it possible? Interrelated skills?• Some skills, as “motivation” is difficult to assess

outside a particular framework• Need for clarification of definitions?• Development of indicators?• Framework?

Assessment (12/14)Assessment (12/14)• the main assessment tool proposed is as self

reference questionnaire• students mark their “grades” against each skill, every

week• They can see the grades they put to themselves in

the previous weeks• “aim being that your scores will have improved” -

biased toward to a false improvement of grades?• What about the students that they do not (or they

thing they do not) improve? – motivation for more or drop out?

Assessment (13/14)Assessment (13/14)• three entities involved in the course, the students,

the teacher(s) and the materials• Student self evaluation• Student evaluation from the students (peer

evaluation)• Student evaluation from the teacher(s)• Teacher self evaluation• Teacher evaluation from the students• Material and course quality evaluation from the

students• Material and course quality evaluation from the

teachers.

Assessment (14/14)Assessment (14/14)• questionnaires are convenient, but others, like

portfolio or in depth interviews may be considered

• the partners should decide about the aims and the methodology of the evaluation

• If we decide to confine ourselves in a given framework, the uses of readymade and tested tools are also possible

• other aspects of influence • self-esteem and self-image

Experimental Methodology (1/1)Experimental Methodology (1/1)• The proposed methodology is the classical

experimental one• Two groups, the experimental and the control one• Initial Assessment• Intervention• Final Assessment

• What about long term results?

Conclusions/Homework (1/3)Conclusions/Homework (1/3)• Does MASS framework needs clarification and

justification?• Is the proposed schedule 18 weeks X 4 hours

suitable for the partners?• Each partner should report on this• Suggest modifications• Two alternatives

• All the same schedule – Comparable results• Different schedules – Test more than one variations of

program to see which works best.

Conclusions/Homework (2/3)Conclusions/Homework (2/3)• Teaching organization

• How many students• How many classes• What age• What educational background• Suggested period of teaching• …what else?

• Decision about assessment• Target of assessment• Ages of students• Assessment procedure: questionnaires only or something more?• Each partner studies and comments proposed questionnaires and

maybe contributes with more• …what else?

Conclusions/Homework (3/3)Conclusions/Homework (3/3)

Deadline?

What about end of June?

Thanks for your patience!!!