compendium of evaluation tools - crwg - gdrc€¦ · meeting workplace skills: the career...

Meeting Workplace Skills: the Career Development Contribution

Compendium of Evaluation Tools(Evidence Gathering)

February 2009

i

Compendium of Sample Evaluation (Evidence Gathering) Tools

Developed by the Canadian Research Working Group for

Evidence-Based Practice in Career Development (CRWGDRC) (Last Updated 2009-02-25)

What this Document Contains The following pages contain sample forms and check lists that could be used to evaluate a variety of different programs or interventions. It really is just a compendium of tools and data gathering (evidence documentation) forms that have been used in field test of the evaluation model that has been developed by the Canadian Research Working Group on Evidence-Based Practice in Career Development. There also are some example of the sort of data that can be produced from the forms. These data can present a strong case for the outcomes and impacts of services that clients receive. First there is some background information on evidence-based practice and outcome-focused intervention. This is followed by a series of forms that have been used in a work search program.

• Client processes, learning outcomes, and impact outcomes can all be documented on the first form (Tracking Check List for Clients…).

• If the program is manualized, the practitioner process can be tracked by putting the main agenda items into a check list to document the content covered and the facilitator skills used. (A sample is provided in the Work Search Skills: Program Audit Check List.)

• Client satisfaction can be documented on the Client Feedback Check List.

• Other possible client learning outcomes could be documented on the Check List of Potential Client learning Outcomes.

• If desired, the inputs could be tracked on the Sample Check List for Evaluating Inputs.

• Service standards could also be documented on a form such as the sample Service Standards Check List.

These are provided as samples of data that could be used to track the processes used and the outcomes achieved.

Third, there are several forms that have been used successfully to monitor a variety of interventions and training workshops. Some of these are from school settings, but most are generic enough that they could be used in a variety of contexts. There also are some examples of the data that result from the use of these forms.

ii

Background on Evidence-Based Practice and Outcome-Focused Intervention The idea behind evidence-based practice and outcome-focused intervention, is really very simple and can be thought of as involving seven steps: Step 1: Be clear about what you are promising. Step 2: Be clear about what service providers need to know, and be able to do, and the

appropriate personal attributes (KSAs, or Knowledge, Skills, and Attributes) to be able to do what's being promised.

Step 3: Create a way to document that the service providers are following the game plan and using the appropriate KSAs

Step 4: Create a way of documenting how well the client outcomes are being achieved. This involves obtaining some tangible indication of how the client has changed, i.e., the change in knowledge, skills, or personal attributes, plus, ideally, the impact of that learning.

Step 5: Gather together all the documentation, both the service provider stuff (from Step 3) and the client stuff (from Step 4)

Step 6: Look for links between what the service providers are doing (from Step 3) and what client outcomes are achieved (from Step 4) and write your final report.

Step 7: Brag to whom ever you can about the effective job you are doing and especially show them the link between your competent staff and the client outcomes they help people achieve.

If I were a funder, my bottom line reasoning would be something like this: “You think you have a program that will help clients attain ‘X’. I'm willing to fund such a program. What evidence will you show me that my investment in your agency to deliver the agreed upon program has been a worthwhile investment?” In most cases, client flow data will not be enough. In most cases, documentation of all the skills and competencies of the service providers (Step 3) also will not be enough. A clearly articulated Step 1 will help convince me to fund the program, because I can see what you are promising the clients to achieve. (BUT it won't help me be willing to fund it a second time, if you can't show me that the client goals you promised have been met, i.e., Step 4.) A clearly articulated Step 2 also will help convince me to fund the program because I can see that the agency staff have the competencies to deliver the program as described. (Again, it would not help me be willing to fund the program a second time if you could not show that the client goals have been met.) If at the end of the program, all you can show me is that the agency staff went about their business in an appropriate way and used appropriate skills in delivering the program, and in fact the program was expertly delivered, I won't be too very much impressed. I'll really be wanting ALSO to have some documentation showing that the client outcomes have been reached. That's what Step 4 is all about. And the bottom line is, if I'm a funder and you can't show me any evidence that the client goals have been reached, then I'd be hard pressed to fund your agency again, especially if I’m a funder who has a board of directors to answer to, more requests for funding than I can possible support, and pressure to fund only those requests that are making a difference in the lives of clients. Step 4 deals with getting evidence that the programs (interventions, etc) are making a difference in the lives of clients. In the many many talks, classes, and workshops given by members of the CRWGDRC on evidence-based practice, it is common to get resistance regarding some part of the overall approach. The usual point of resistance is that practitioners don't want to collect data that places themselves as integral players in the game plan that results in client success (Step 3). Sometimes, people object to gathering data that indicate the client outcomes have been

iii

achieved. To these sorts of objections we can only ask, "What's the good of gathering any other kind of data, if you don't have evidence that the client goals have been met?" The bottom line for funders is "show me that the client outcomes have been achieved" OR "show me the evidence that your program has had the kind of impact on your clients that you promised me when I funded you." The bottom line for agency managers is “my agency and my staff have been integral players in producing the client outcomes. The clients could not have done it by themselves.” About client outcomes, the driving force behind evidence-based practice is to have some evidence (not necessarily numbers, or test results, or placement percentages, but some sort of identifiable evidence [proof would be too strong of a word, but they want something more than "trust me I know it's working"]) some evidence that the programs (or interventions, or treatments, or workshops, etc) are working (Step 4). Step 4 is the heart of evidence-based practice. If people end up believing that they are doing evidence-based practice, and they can't show any documentation that the promised client outcomes have been achieved, then they will be in for a sad experience when it comes to having their funding renewed. Regarding Step 4, it would be a real shame to scrap Step 4 and to leave the agency with no data (evidence) to support their beliefs that they have made a worthwhile impact on the clients they served. The heart of Step 4 is getting some sort of support for practitioners' beliefs that they are having a worthwhile impact on the clients they serve. It's one of the reasons why practitioners, once they get into the habit of documenting their successes, don't want to stray away from an evidence-based practice approach. It not only provides more convincing arguments to funders, but it helps practitioners feel good about the work they are doing because they now have something tangible that indicates to them (as well as their clients and their funders) that they are doing worthwhile work.

A good companion to the information in this compendium of tools is the Framework for Evaluating the Effectiveness of Career Development Interventions available from the CRWGDRC website http://www.crwg-gdrc.ca.

iv

Retrospective Assessment One difficulty with using self-assessments to determine changes in skill or knowledge resulting from workshops on counselling interventions is that people don’t know what they don’t know. To illustrate, people taking a course on interpersonal communication might be asked to rate their communication skills at the beginning of the course and again at the end of the course. At the beginning many people think that their communication skills are reasonably good and they rate themselves quite high. During the course, as they get to know more about what constitutes good communication, they realize that their knowledge about interpersonal communication, as well as their skills for communicating effectively, are not as good as the initially thought. At the end of the course they are asked to rate themselves again, and often the ratings are lower than they were at the beginning, even though they have learned a lot and have a more adequate level of skills. This is because their measuring stick has changed, as they developed greater knowledge about interpersonal communication. Thus the post-test scores end up being lower than the pre-test scores, even though positive change has occurred. Retrospective assessment is often used to address this problem, as it creates a consistent measuring stick for both pre and post assessments. One common procedure for retrospective assessment has been called Post-Pre Assessment. This process is used ONLY at the end of a course or program. It asks people to use their current level of knowledge to create a common measuring stick for pre-course and post-course assessments. For example, when a course (or workshop, or program, etc.) is finished participants can be asked: Knowing what you know now about interpersonal communication, how would you rate yourself before the workshop, and how would you rate yourself now? The self-assessment is done only at the end of the course or workshop, but it asks people to self-assess their pre-course competencies (hence the name “post-pre assessment), followed by their post-course competencies, using the same measuring stick, i.e., “Knowing what you know now…” In the examples that follow, some sample forms are presented, along with the results of a workshop or course evaluation, to illustrate the types of conclusions that can be drawn. These data provide strong support for the learning changes that have occurred as a result of the workshop.

v

Evaluation as a Decision-Making Process Frequently, evaluation is approached from a judgmental perspective: An evaluator passes judgement on the topic under examination. We suggest instead to approach evaluation from a decision-making perspective, where an evaluator examines the evidence and decides on the level of acceptability on the topic under examination. Consider as an example the self-evaluations (or supervisor evaluations) of topics such as: knowledge, skill, level of competence, degree of preparation, degree of engagement, etc. In such situations, we have found it useful, and have found that we get high inter-rated agreement, as well as intra-rater agreement across time, when we use the following decision-making approach. In other words when several raters look at the same instance of performance, there is high agreement between different observers on what should be the appropriate rating. AND when the same rater looks at the same instance of performance on different occasions, there is high agreement on the rating from Time 1 and Time 2. Approaching evaluation as a decision-making endeavour involves a two-step process. Step A, decide if the competency (degree of preparation, level of engagement, level of knowledge, etc.) is acceptable (employable at a beginning level in your agency), or unacceptable (unlikely to result in continued employment at your agency).

Unacceptable Acceptable

Then, in Step B, decide:

If the level of competence is unacceptable, is it:

• Really quite poor (= 0) or • Almost OK (= 1)

If the level of competence is acceptable, is it:

• Just barely OK (= 2) [but still acceptable, otherwise it would be 1] • Really very good (= 4), or • Somewhere in between (= 3)

[better than minimally acceptable, but not yet excellent]


0 1 2 3 4

Many of the forms and checklists in this Compendium of Sample Evaluation (Evidence Gathering) Tools adopt the approach outlined above. We have found in our field trials that approaching evaluation as a decision-making process provides reliable data in a wide variety of situations.

Sample Evaluation (Evidence Gathering) Tools p. 1

Sample Tracking Check List for Clients in Work Search Program

(to be completed for each participant)

Rating (see scale below) Expectations for Client 0 1 2 3 4

Attendance Job Leads Network Telephone Practice Cover Letter Resumé Portfolios Application Forms Self-presentation skills Job interview skills Video interview practice

Client Process refers to how well the client

engaged in the training process for each element

Negotiate employment contract

Telephone skills • Network calls

• job leads Informational interview skills Self-presentation skills Job interview skills

Client Learning: Skills refers to how well the client mastered the skills taught in

this program

Negotiate appropriate employment contract

Network chart calling cards Sample application forms Resumé Cover letter Number of job interviews obtained Job offer No Yes

Client Outcomes refers to the products that clients produce during the

Job Search program. These should be part of

each client's file

Job accepted No Yes

Please use a two-step process. (A) decide if the skill use is Acceptable or Unacceptable, then (B) assign the appropriate rating:

0 = performance is very poor 1 = performance is poor 2 = performance meets minimum acceptable standard 3 = performance is good 4 = performance is outstanding


Sample Work Search Skills: Program Audit Check List

This form is for use by facilitators, supervisors, or external evaluators to gather data on how closely the Work Search Program is being followed. For each program element listed below, indicate whether of not that element was covered (No, Uncertain, Yes), and the quality demonstrated by the facilitator. To assess the quality, please use the scale described at the bottom of page two. If some elements were not addressed by the facilitator, please indicate the reason(s) for departing from the facilitator guide.

Completed Rating Program Element N ? Y 0, 1, 2, 3, 4

Group building Self-exploration

• values

• future aspirations • sources of personal

satisfaction

• interests • abilities • list other items in self-

profile

Tracking down job leads • Using print sources • Using internet sources • Telephone skills • Conducting cold calls

Cover Letter

Resumé

Portfolios

Application Forms Self-presentation skills • Explained skills • Practice skills

Interview skills • Explained skills • Practice interviewing

skills

Negotiate employment contract

Used skills Rating Facilitator Skills

(General Use) N ? Y 0, 1, 2, 3, 4 Clear instructions Listening to participants Skill demonstrations Positive Approach

• tally facilitator comments positive corrective negative

Engaging all group members • tally # of facilitator

interactions with each participant

Sharing air time Match skills to relevant program element

Other groups skills




Client Involvement Check List (to be completed by service provider)

Please check each way you have involved your client in the counselling process.

As a reliability check, your clients also may be asked to complete this check list.

Client Name:

Set learning objectives together in the first interview yes no

Document progress towards learning objectives yes no

Provide client with written service standards yes no

Client’s role in evaluation is reviewed as part of the initial interview yes no

Clients receive orientation/training for any data collection requested of them yes no

Client’s completion of evaluation activities is acknowledged in agency reviews and/or through thank you letters yes no

Before completing an intervention, client’s progress is reviewed with them in the context of their learning objectives yes no

Clients are assured of confidentiality regarding their evaluation activities

yes no

Other yes no

Other yes no


Rating (see scale below)

Sample Client Feedback Check List (Some agencies may wish to duplicate this page and

use it at both the beginning and end of a program.) 0 1 2 3 4

At the beginning of the program, some client needs were identified that this program was intended to meet. Please list the needs and indicate how well each of these needs have been met.

•

•

•

•

In addition to the above, some needs were identified during the program as being particularly important to you. Please list these needs and indicate how well each of these needs have been met.

•

•

•

•

You have an occupational goal Yes No

specify goal

Rate how optimistic you are that your goal will be achieved

You have an action plan Yes No

specify main elements of action plan

Rate your chances of completing your action plan



Client is self-sufficient in the following areas (please list)

•

•


Sample Client Feedback Check List (Continued) (To be completed by each client at the finish of the program) Rating

(see scale below)

0 1 2 3 4

My counsellor has: • outlined staff roles & responsibilities

• outlined expectations of client

• completed assessment of client needs

• outlined available programs

• explained how programs address client needs

• negotiated with me my goals for the program

Generally speaking, how competent do you think the staff are? • receptionist

• counsellor

• program coordinator

• group facilitator

Generally speaking, please rate each of the following: • accessibility

• timeliness

• responsiveness to client needs

• overall satisfaction with service

At the beginning, how knowledgeable were you about the programs and services available to you?

NOW, how knowledgeable are you about the programs and services available to you?




Sample Check List of Potential Client Learning Outcomes Please rate each item using the scale described at the bottom of this page.

(If a particular item is not relevant for your client, draw a line through the rating boxes for that item)

Rating (see scale below) 0 1 2 3 4

1. Personal management outcomes

a. Build and maintain a positive personal image

b. Interact positively and effectively with others

c. Change and grow throughout one’s life

2. Learning and work exploration outcomes

a. Participate in life-long learning supportive of life/work goals

b. Locate and effectively use life/work information

c. Understand the relationship between work and society/economy

3. Life/work building outcomes a. Secure, create and maintain work

b. Make life/work-enhancing decisions

c. Link decision-making to life/work

d. Maintain balanced life/work rules

e. Understand the changing nature of life/work roles

f. Understand, engage in and manage one’s own life/work process




Sample Check List for Evaluating Inputs Please rate each item using the scale described at the bottom of this page.

Rating (see scale below) Service Standards 0 1 2 3 4

Number of FTE staff available for career counselling

Proportion of staff time available for career counselling

Staff time use: (record % then rate performance on each item)

• work with clients

• consultation

• client advocacy

• networking

• coordination

• collaboration

• supervision

• professional development

Client flow: (record # clients, then rate performance on each item)

• decision making

• skills enhancement

• job search

• employment maintenance

• Total # client (some clients may work on more than 1 dimension)

Agency has met objectives: (list objectives below)

•

•

•

Please use a two-step process. (A) decide if the outcome use is Acceptable or Unacceptable (B) assign the appropriate rating:



Sample Service Standards Check List Please rate each item using the scale described at the bottom of this page.

Rating (see scale below) Service Standards 0 1 2 3 4

Evaluation of Service: Overall • accessibility

• timeliness

• responsiveness

• general client satisfaction

Evaluation of Service: Specific

• General performance expectations

• Specific program considerations (please list)

• Client intervention processes: decision making

skills enhancement

work search

employment maintenance

• Overall client self-sufficiency

• Other

Please use a two-step process. (A) decide if the outcome use is Acceptable or Unacceptable (B) assign the appropriate rating:



Skills Check List for A Career Education Unit for Junior High School or Senior High School

Item Date completed

Evaluation

1. Self-profile (Students develop a self-profile) • values • future aspirations • sources of personal satisfaction • interests • abilities • list other items in self-profile 2. Career Information Research (students become aware of a wide variety of career alternatives) • Student used informational interviewing to find information on

at least 3 jobs

• Student used career files to get information on at least 6 jobs 3. Career decision-making (Students demonstrate the ability to use a decision making process) • named steps in process • used process to make 1 decision (name the decision) 4. Career plan (Students will develop a realistic career plan) • Develop a realistic career plan (containing general

occupational goal, routes/prerequisites to achieve the goal, possible branching career paths related to general goal)

• Develop a back-up plan (containing the same elements as the primary plan)

5. Complete job sample application form

6. Interviewing Skills • student can list basic job interview skills • student demonstrates job interview skills (during practice

interviews)

NOTE. Each of the specific learning outcomes attached to the activities included in the Career Education unit would be listed along with the procedure for assessing them

Please use a two-step process to evaluate how well the skill or outcome for the student, has been met. (A) decide on whether the outcome is acceptable or unacceptable, then (B) assign the appropriate rating:

(0) unacceptable, (1) not really acceptable, but almost there, (2) minimally acceptable (but still OK otherwise it would be 1 or 2), (4) exceptional, and (3) somewhere between minimally acceptable and exceptional.

Graphically, the scale looks like this: ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞

0 1 2 3 4

Acceptable Unacceptable


Goal Attainment Scaling This method provides an easy to use way of quantifying client change that can be aggregated across clients, different client problems, and across time.

0

+2

+1

-2

-1

1. Start by asking the client for the expected outcome of counselling. Get an observable indicator of the expected outcome, or at least an observable correlate of the outcome. Call this “0”.

2. Ask the client to identify an outcome that would be plausible, but that would far exceed his or her expected outcome. Get an observable indicator of the expected outcome, or at least an observable correlate of the outcome. Call this “+2”.

3. Ask the client to identify an outcome that would be in between the expected outcome and the one that would far exceed expectations. Get an observable indicator of the expected outcome, or at least an observable correlate of the outcome. Call this “+1”.

4. Ask the client to identify an outcome that would be very disappointing. Get an observable indicator of the expected outcome, or at least an observable correlate of the outcome. Call this “-2”.

5. Ask the client to identify an outcome that would be in between the expected outcome and the one that would very disappointing. With many clients, this item is “stay the same as I am now”. Get an observable indicator of the expected outcome, or at least an observable correlate of the outcome. Call this “-1”.

Negotiate the anchors with the client at the beginning of counseling, then use the GAS as a measure of change. Sometimes, clients cannot imagine much change taking place, so they have quite low expectations for the +1 and +2. In these cases, clients may reach +2 before the termination of counselling. In such cases, it is appropriate to negotiate a new scale, where the original “+2” might become the new “0” or even “-1”. In these situations, it might be possible for a client to achieve the equivalent of “+3” or “+4” as a measure of change, even though the scale only goes up to “+2”. Often, clients have several goals they want to work on in counselling. In such cases, it is useful to have a separate goal attainment scale for each goal, each with negotiated observable indicators for the five anchor points. Thus, for example, a counsellor and client might have a goal attainment scale for pain control, anger control, general feeling of well-being, and self-confidence, if those were four areas that they were working on together. Sometimes, even when there are multiple goals, it is still useful to include a general goal attainment scale that applies to the overall goal of counselling. The general goal attainment scale might be phrased in terms of number of goals met, or the extent to which the most troublesome goals are attained. The key factor is that even in the general case, the five anchor points need to represent a negotiated common perspective between counsellor and client. The summary sheet on the following page is a useful way to complete the above process. When competed, it can photo copied, with one copy returned to the client and the other copy going into the client’s file.


Goal Attainment Scaling

Summary Sheet

+2

Far better than expected outcome

+1

In between 0 and +2

0

Expected Outcome

–1

In between 0 and –2

–2

Far worse than expected outcome


Job Interview Skills Practice Log Sheet

Use the following form to keep track of practice job interviews you have done in the course, or job interviews you have attended in the field.

Use the following scales to chart your reactions.

I thought the interviewer was 1=rotten 2=OK 3=really friendly

Your skills 1=terrible 2=not too good 3=OK 4=mostly good 5=excellent

How you felt the interview went 1=terrible.................5=OK.................10=great

Client Name

Date Interviewer Name Interviewer rating

My skills How I felt

This form could be used by clients to self-monitor and self-evaluate their skill practice. It also could be used by the counsellor to evaluate the client performance during the training program.


Anger Management Practice Record Name

Situation

Date:

Skill Use How well did it work

Count to 10 1 2 | 3 4 5

Walk away

Talk it out

Deflection

Get help

Describe what happened

I am sad O.K. Happy about how this worked

Next time I will

No Yes

Now I can usually do this OK

I still need more work on this skill

In evaluating the outcomes, please use a two-step process. (A) decide on whether the outcome is acceptable or unacceptable, then (B) assign the appropriate rating:

(0) if the outcome is just terrible, (1) if the outcome really is not acceptable, but almost there, (2) if the outcome is barely acceptable (but still OK otherwise it would be 0 or 1), (4) if the outcome is outstanding, and (3) if it is somewhere between minimally acceptable and outstanding.

Graphically, the scale looks like this:


Skill Practice Record Sheet Name

Skill Date: From To

Each time you practice your skill, record the result in the box below. Use one box for each practice time.

Who did I try this with? When?

What happened


Next time I will


What happened


Next time I will


What happened


Next time I will


What happened


Next time I will

No Yes

Now I can usually do this skill

I still need more work on this skill This form has been used in a variety of settings to monitor skill practice at home or other field settings. It can be used for a variety of skills, such as: saying nice things to other people, thinking self-supporting thoughts, looking at people when speaking to them, organizing work materials, or taking a deep breath and letting it out slowly instead of getting mad, etc.


Self-Monitoring Affective State (and Other Conditions)

Motivation Level Monitoring Guide

Place a mark on the chart below to show how motivated you feel to do your work (attend classes, look for work, etc.).

Use the five-scale below.

5 = very motivated-it's the most important thing for me to do today

4 = very motivated, but something might come up to interfere

3 = quite a bit motivated, I think I will end up doing some of it

2 = I sort of care and I might get around to doing some work today

1 = If I run out of things to do, I'll try looking in some of my books

0 = I couldn't care less if I ever did my school work

5

4

3

2

1

0

06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 01 02 03 04 05 (time of day)

Often counsellors and teachers have trouble gathering data on changes in affective and attitudinal variables such as motivation, stress, self-confidence, etc. The above system has worked very effectively to get some self-monitored data from clients on these sorts of variables. The numbers in the bottom row of above grid indicate time of day. The numbers in the left column indicate the rating for motivation level. Clients introspect for a few moments to self-assess their level, then they place an "x" at the appropriate place on the grid. This is done every hour. Usually, the grid can be produced on both sides of an 8.5X14 sheet of paper. This leaves room for 4 grids on one side and 3 grids plus the instructions on the other side. The sheet can be folded in quarters to make it easy to carry. The same procedure could be used go measure a variety of affective or attitudinal variables, such as: stress level, self-esteem, commitment to a program, how much a person likes school, etc. For each new variable, an appropriate set of descriptors for 1, 2, 3, 4, and 5, would need to be developed, but the grid could remain the same. Some samples of the descriptors that could be used for self-monitoring various affective states. Students are encouraged to develop their own descriptors and to explore creative ways to use this type of self-monitoring grid.


Sample Instructions for Self-Monitoring Guides

Stress Monitoring Chart It is important to monitor the intensity of your stress levels for at least two reasons:

1. Research has shown that this will help reduce the psychological side effects associated with your stress and often will help to reduce stress level.

2. It is useful in helping to determine the effects of your treatment program.

The following five point scale is useful in helping people monitor the severity of their headache.

0 - No stress 1 - Low level, only enters awareness when you think about it 2 - Aware of stress level most of the time, but still able to stay focused on job 3 - Very stressful, beginning to interfere with what you are doing 4 - Severe stress, creates major interference with what you are doing 5 - The most stress you ever feel

To monitor your stress level, mark the appropriate number on the graph at each hour and join the points together.

List at the right of the graph any medication you take, and the time it was taken, and any situations that affected your stress level and when they occurred.

Placing a coloured dot on your watch will help you remember to do this.

Self-Confidence Monitoring Chart It is important to monitor the intensity of your self-confidence level for at least two reasons:

1. Research has shown that this will help you feel more in control of how confident you feel.


The following five point scale is useful in helping people monitor their level of self-confidence.

0 - No confidence at all in your ability to handle the situation 1 - Low confidence, but I might get lucky and it will turn out OK 2 - Sort of confident, I think I might be able to handle it, but probably not 3 - Pretty confident, I think I have a chance of this working out OK 4 - Very confident, I'll probably be able to pull this off 5 - Completely confident about what you am doing

To monitor your confidence level, mark the appropriate number on the graph at each hour and join the points together.

List at the right of the graph any medication you take, and the time it was taken, and any situations that affected your self-confidence and when they occurred. Placing a coloured dot on your watch will help you remember to do this.


Job Satisfaction Monitoring Chart It is important to monitor your job satisfaction for at least two reasons:

1. Research has shown that this will help place job satisfaction in a more objective perspective.


The following five point scale is useful in helping people monitor their level of job satisfaction.

0 - No satisfaction at all with your job 1 - Somewhat satisfied, there are some things about your job that are OK 2 - Satisfied enough – as jobs go, this one is as good as any 3 - Mostly satisfied, occasionally there are things about your job that you actually like 4 - Very satisfied, but there are occasional things bother you 5 - Completely satisfied with your job, feeling a sense of self-fulfillment

To monitor your satisfaction level, mark the appropriate number on the graph at each hour and join the points together.

List at the right of the graph any medication you take, and the time it was taken, and any situations that affected your job satisfaction and when they occurred. Placing a coloured dot on your watch will help you remember to do this.

This tool was developed initially for use with headache patients in my private practice. It turned out to be a useful source of many indicators of treatment outcome. Comparing the charts across time, clients and I could both see changes in the number of headache episodes, i.e., the number of level 4 or 5 headaches. We also tracked the average level of headache intensity and the number of headache-free hours per day. We cold see the effects of medication and clients began to be better at seeing the situations that triggered their headaches. I also asked client to mark on their headache charts when they did their home practice at relaxation, which provided us both with an indication of client adherence to the program. I combined this monitoring chart with the relaxation monitoring chart shown later in this reading file. The relaxation monitoring chart gave us some evidence that the client was in fact developing greater skill at relaxation. Putting all of this together, I was able to demonstrated a nice link between the work we were doing together and the outcome: My case notes provided evidence that I was following a fairly typical protocol for teaching relaxation, the relaxation monitoring chart provided evidence that the client was also following the program and learning relaxation skills, and the headache monitoring chart provided evidence that headache activity was reducing. Clients and I could both say with some assurance that headaches were becoming less of a problem, and it was due to the client learning relaxation skills as a result of following the relaxation training program that I was orchestrating. The instructions I used for the headache monitoring chart are printed below and the actual grid is reproduced on the next page. The who tool was printed on an 8.5 by 14 inch sheet of paper, the instructions and room for 3 days worth of data on 1 side of the sheet and 4 days worth of charts on the other side. Thus, 1 sheet of paper provided data for a whole week.


Headache Progress Chart It is important to monitor the intensity of your headaches for at lest two reasons:

1. Research has shown that this will help to reduce the psychological side effects that often accompany a headache.


The following five point scale is useful in helping people monitor the severity of their headache.

0 - No headache 1 - Low level, only enters awareness when you think about it 2 - Aware of headache most of the time, but it can be ignored at times 3 - Painful headache, but still able to continue job 4 - Severe headache, difficult to concentrate with demanding tasks 5 - Intense incapacitating headache

To monitor your headache level, mark the appropriate number on the graph at each hour

and join the points together. List at the right of the graph any medication you take, and the time it was taken. Placing a coloured dot on your watch will help you remember to do this. Name:


Intensity 5 Date Medication Time

4

3

2

1

0 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5

a.m. noon p.m. midnight a.m.


4

3

2

1

0 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5



4

3

2

1

0 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5



Relaxation Home Practice Log Sheet

Name__________________________

Date/Time

Indicator Start Finish Difference

pulse rate (beats per minute)

breathing rate (beats per minute)

finger temperature (degrees)

Feelings, body sensations, etc.

Thoughts, perceptions, imagery, etc.

This sheet has been used as part of relaxation training programs, to help participants track their home practice and become more aware of the feelings and thoughts associated with deep relaxation, as well as to give counsellors information on how much home practice clients are doing. Clients are taught to self-monitor heart-rate, breathing rate, and given a small thermometer to measure finger temperature. The clients then take these three measures before and after home practice, thereby providing objective data on what is happening to their bodies while they practice the relaxation exercise.


Sample workshop Evaluation Youth Transition Workshop

Feedback

The purpose of the Youth Transition Workshop is to provide current information about career planning processes and resources that can help parents, relatives, educators, counselors, and youth workers have more meaningful interactions with youth about their transition planning decisions. This feedback form will help us evaluate the workshop and identify ways in which we can improve it. First there are some General Questions

In answering these general questions about the workshop, we would like you to compare yourself now and when before the workshop began. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now in each of the following questions using the evaluation framework below?

Very low Somewhat low

Neither high nor low

Somewhat high

Very high

VL SL ? SH VH

Start of course Now

VL SL ? SH VH VL SL ? SH VH

Knowledge of how current social, demographic, economic, and workplace trends impact opportunities and how workers can take advantage of those opportunities

Understanding of the career planning process in general

Understanding of myths that impede effective transition planning decisions for youth

Knowledge of career development tools and resources

Knowledge of ways to develop and demonstrate employability skills

Knowledge of work search strategies and ways to support work search

Competence to engage in career conversations with youth

Competence to be a guide for youth to be more effective in their career/life planning

Excitement about the field of career development


Building Successful Partnerships and Collaborations Sample Workshop Evaluation

Workshop Objectives The objectives of the workshop are:

1. to clarify the characteristics of partnerships (and collaborations); 2. to identify the factors that contribute to forming partnerships (and collaboration); and, 3. to develop a framework for understanding group process

First Some General Questions In answering these general questions, we would like you to compare yourself now and before the workshop. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now?

In responding to the questions, please use a two-step process. (A) decide on whether the characteristic in question is acceptable or unacceptable, then (B) assign the appropriate rating:

(0) unacceptable, (1) not really acceptable, but almost there,

(2) minimally acceptable (but still OK otherwise it would be 1 or 2), (4) exceptional, and

(3) somewhere between minimally acceptable and exceptional. Graphically, the scale looks like this: ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞

Regarding the Primary Objectives, and knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now?:

Before Workshop

Now

1. Clear understanding of the characteristics of successful partnerships and collaborations

2. Knowledge about the factors that contribute to (or interfere with) forming partnerships & collaborations

3. Knowledge regarding a developmental framework for analyzing group process

4. Repertoire of practical tools and approaches for facilitating the development of partnerships

Generally speaking,

5. how useful did you find the workshop? 6. how would you rate the workshop facilitation? 7. how would you rate the workshop facilities (room, etc.)? 8. how would you rate the food?


Now Some Specific Questions Building Successful Partnerships and Collaborations is designed to provide participants with background knowledge and a framework for strategic planning. A range of topics is covered and a range of activities to help personalize the knowledge or apply it to participant work settings. Hopefully, there will be something in this workshop for all participants: some of the ideas or activities may be useful to most participants, while others may be useful only to a few. We are interested in finding out your reaction to the ideas and activities contained in this workshop. Therefore, we would appreciate it if you would give us your impression of each of the main components of the workshop. The items follow the order in which they occurred.

Please indicate the extent to which you personally found the following components of the workshop useful.

9. Classified Adds-Opener

10. Characteristics of successful partnerships

11. Factors contributing to successful collaborations

12. Working Alliance

13. Group leader skill framework

14. Factors involved in planning group process

15. Stages of group development

16. Group leader approaches, tailored to group development stage

17. How partnerships develop

18. Case studies

19. How useful were the participant handout materials? Identify one useful thing from this workshop that you want to remember. Please write any additional comments about the workshop here.


Building Successful Partnerships and Collaborations Workshop Evaluation

Sample Evaluation Results Workshop Objectives The objectives of the workshop are:

1. to clarify the characteristics of partnerships (and collaborations); 2. to identify the factors that contribute to forming partnerships (and collaboration); and, 3. to develop a framework for understanding group process

First Some General Questions In answering these general questions, we would like you to compare yourself now and before the workshop. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now? In responding to the questions, please use a two-step process. (A) decide on whether the characteristic in question is acceptable or unacceptable, then (B) assign the appropriate rating:


Graphically, the scale looks like this: ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞

Regarding the Primary Objectives, and knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now?

Before Workshop

Ave

Now

Ave

1. Clear understanding of the characteristics of successful partnerships and collaborations -- 4 7 5 1 2.2 -- -- -- 7 10 3.6

2. Knowledge about the factors that contribute to (or interfere with) forming partnerships & collaborations -- 3 8 5 1 2.2 -- -- -- 6 11 3.6

3. Knowledge regarding a developmental framework for analyzing group process 2 4 10 1 -- 1.6 -- -- -- 9 8 3.5

4. Repertoire of practical tools and approaches for facilitating the development of partnerships 2 3 6 6 -- 1.9 -- -- -- 9 8 3.5

Detailed description of results • Of the 68 ratings (4 questions times 17 people), 3 increased by 4 points, 7 increased by 3 points,

20 increased by 2 points, 33 increased by 1 point, and 5 remained the same. There were 18 ratings that changed from unacceptable to acceptable. All “now” ratings are in the acceptable range.

• The self assessments of all participants showed increased scores on half or more of the 4 items above and none of the ratings went down.

• The pre-workshop assessments of 4 on #1 & #2 came from 2 different people, and of course those scores did not increase after the workshop. There were two additional people whose ratings on #1 or #2 were 3 before and after the workshop.


Generally speaking, Ave

5. how useful did you find the workshop? -- -- -- 3 14 3.8 6. how would you rate the workshop facilitation? -- -- -- 2 15 3.9 7. how would you rate the workshop facilities (room, etc.)? -- 2 3 8 4 2.8 8. how would you rate the food? -- -- 3 6 8 3.3

Now Some Specific Questions Building Successful Partnerships and Collaborations is designed to provide participants with background knowledge and a framework for strategic planning. A range of topics is covered and a range of activities to help personalize the knowledge or apply it to participant work settings. Hopefully, there will be something in this workshop for all participants: some of the ideas or activities may be useful to most participants, while others may be useful only to a few. We are interested in finding out your reaction to the ideas and activities contained in this workshop. Therefore, we would appreciate it if you would give us your impression of each of the main components of the workshop. The items follow the order in which they occurred.

Please indicate the extent to which you personally found the following components of the workshop useful. Ave

9. Classified Adds-Opener -- 1 1 7 6 3.2

10. Characteristics of successful partnerships -- -- -- 2 14 3.9

11. Factors contributing to successful collaborations -- -- -- 4 12 3.8

12. Working Alliance -- -- 1 5 10 3.6

13. Group leader skill framework -- -- 1 7 8 3.4

14. Factors involved in planning group process -- -- -- 5 11 3.9

15. Stages of group development -- -- -- 7 9 3.6

16. Group leader approaches, tailored to group development stage -- -- -- 6 10 3.6

17. How partnerships develop -- -- -- 4 12 3.8

18. Case studies -- -- 3 6 7 3.3

19. How useful were the participant handout materials? -- -- -- 4 12 3.8


Creating Healthy Lifestyles An Intervention Guide For Building A Healthy Lifestyle

Program Evaluation First Some General Questions

In answering these general questions about the program, we would like you to compare yourself now and when the program began. Knowing what you know now, how would you rate yourself 3 months ago, when the program began, and how would you rate yourself now?

First compare your knowledge about lifestyle, stress, and fitness.

very low somewhat low

neither high or low

somewhat high

very high

VL SL ? SH VH

Start of program Now VL SL ? SH VH VL SL ? SH VH Knowledge of stress and stress control Knowledge of how to manage personal change Knowledge of nutrition and nutrition control

Now compare your level of stress, fitness, etc. Start of program Now VL SL ? SH VH VL SL ? SH VH Level of stress Level of nutrition (high=healthy) Level of fitness Confidence in ability to manage personal change

To what extent would you say that any changes depicted above were a function of participating in our program, and to what extent were they a function of other factors (e.g., job, personal circumstances, etc.)

mostly other factors

somewhat other factors

uncertain somewhat this program

mostly this program

What for you were the 3 best features of this program

1.

2.

3.

What for you were the 3 worst features of the program

1.

2.

3.


Now Some Specific Questions Stress Mastery is a multi-faceted program, offering a wide variety of approaches, interventions, and tools for helping people alter lifestyle habits, improve nutrition, reduce stress, exercise more regularly, become more fit, and develop a more balanced lifestyle. Our philosophy has been to provide people with a variety resources and encourage them to choose the ones that fit best for them. Hopefully, there will be something in Stress Medicine for everyone: some of the resources may be useful to most participants, while others may be useful only to a few. Some of the ideas may have been helpful in assisting you in making the types of changes you want to make in your life, while other sections might be useful only in terms of the information provided, and some parts of the guidebook might not be useful at all to some people. We are interested in finding out your reaction to the ideas, tools, and strategies contained in this guidebook. Therefore, we would appreciate it if you would give us your impression of each of the main components in our program. The items follow the order in the participant guidebook. We are asking that you tell us:

• whether you found the information useful, and • the extent to which you actually used the ideas and tools we provided.

Please use the two rating scales below.

usefulness of information/resource I did I used this information/resource Not useful useful information not use and found it

should be omitted

but retain

uncertain

very useful

extremely useful

this interfered with

progress

did not help

much

somewhat helpful

very helpful

extremely helpful

NUO NUR ? VU EU DNU IWP NMH SH VH EH

NUO NUR ? VU EU DNU IWP NMH SH VH EH

Ch. 1: Ready for Change Stages of change Lifestyle planner (lifestyle pie chart)

Ch. 2: Maintaining Balance Stress model model for stress control Goal setting/self-management Action planner

Ch. 3: Taking Charge of Nutrition Nutrition and stress Nutrition planner

Ch. 4: Habit Management Steps for habit management Awareness worksheet (triggers, self-talk, etc.) Reinforcement and maintenance

Ch. 5: Eating on the Run Healthy restaurant eating Menu planning Shopping tips


Ch. 6: Pulling It All Together

Review and reflection Changes and strategies Healthy weight

Ch. 7: Becoming your Own Best Friend Being aware of self-talk Changing self-talk Self-coaching Building self-esteem

Ch. 8: Food Labels: What They Really Mean Ingredients lists Nutritional claims

Ch. 9: Relaxation: Skill for the Future Background information Relaxation practice exercise

Ch. 10: Healthy Living Healthy Living: Wellness (vs. illness) General information (self-talk, water, sleep) Specific information: alcohol, smoking, drugs) Specific information: Stress & illness Circle diagram depicting symptoms & disease Stress symptom check list

Ch. 11: Active Living Healthy living: programs Healthy weight: programs Fit for anything: programs

Ch. 12: Next Steps Reward and maintenance Guidelines for maintenance

Additional Resources Reading list Internet sites Self-monitoring stress

General Program Components Log Book Weekly exercise tips Weekly home practice activities

Please write any additional comments about the program here.


Creating Healthy Lifestyles An Intervention Guide For Building A Healthy Lifestyle

Sample Evaluation Results Participant Self-Assessments Of Change

Knowledge of stress and stress control

Response category Start Finish very low 17% 0% somewhat low 30% 0% neither high nor low 26% 0% somewhat high 21% 70% very high 4% 30%

Knowledge of how to manage personal change


Knowledge of nutrition and nutrition control


Level of fitness


Level of stress


Ability to manage personal change


Level of nutrition


To what extent were the above changes a result of program participation?

Response category % mostly other factors 0% somewhat other factors 0% uncertain 9% somewhat this program 26% mostly this program 57% missing data 8%

Evaluation (Evidence Gathering) Tools p. 30

What The Participants Say

“ This program made me realize that change is possible for me to make, and that even a

small effort can bring significant benefits. When I joined the program, I thought that only a superhuman effort would result in any positive benefits, and that thought itself was a barrier to making a behavioural change. Now I know that “I can do it” and that knowledge itself is half the battle.”

“ I’ve made some positive lifestyle changes during the 12 weeks of this course, but I will make more changes over the next 6 months. The true success of this course to me will be reached in the next 1–>5 years.”

“ I would like to see a 6 month, 1 year, and 2 year follow-up program, maybe incorporating a refresher class (fitness and education), or perhaps even a short 3-4 week refresher program, even if we had to pay for it. This would help us get back on track if we’ve fallen by the wayside and it would give additional support to those who continue with their lifestyle changes.”

“ This is a fantastic program. Even though I had to miss a few classes, I still got a lot out of the reading material. The biggest benefit for me is that I am starting to see myself as being a more active person, rather than a “couch potato”. I appreciated the focus of the program being on life change and wise choices, rather than on diet and weight loss. I used to go to 7-11 for a slurpie and a chocolate bar, now I have a club soda and veggie sticks!”

This research was made possible through the generous support of many partners:

The Calgary Health Trust, Shoppers Drug Mart, Petro-Canada, Gimble Eye Centre, and University of Calgary.

Creating Healthy Lifestyles, is delivered by the LifeLong Wellness Research Institute

510, 444 – 5th Avenue S.W., Calgary, Alberta T2P 2T8 Tel: 264-1610, Fax: 262-7076


Creating Healthy Lifestyles Six Month Follow-Up Feedback Form

To what extent are you still implementing some of the components of the program?

not really

a little bit a lot

1. generally being more active 2. getting regular exercise 3. using the information on stress control 4. using the nutrition information 5. using the knowledge on how to manage personal change

Knowing what you know now, how would you rate yourself when the program began, when the program finished, and how would you rate yourself now?

Please use the scale below.

very low somewhat low

neither high or low

somewhat high

very high

VL SL ? SH VH

Start of program End of program Now VL SL ? SH VH VL SL ? SH VH VL SL ? SH VH

Level of stress Level of nutrition (high=healthy) Level of fitness Confidence in ability to manage personal change

To what extent would you say that the changes depicted above were a result of participating in our program, and to what extent were they a result of other factors (e.g., job, personal circumstances, etc.)




mostly this program

We would like some feedback on the amount of information the participant manual contains. What would you say about the amount of information provided? too much information, just

summarize it about the right amount of

information not enough information, I’d

like more

(Please turn over)


Please comment in the space below, on any components of the program you found particularly useful or any aspects of the program that detracted from its effectiveness.

Please comment also on what impact your participation in Stress Mastery has had in your life.

Can we use your comments in research reports, provided your identity is not revealed?

yes no

If yes, please complete the form below. I, authorize Dr. Bryan Hiebert and his research team to use any of the above information, provided my identity is not revealed.

(Signature) (date)

Please fax this form to Dr. Bryan Hiebert at 282-9244 or mail it to Dr. Bryan Hiebert, Division of Applied Psychology, University of Calgary, Calgary, AB, T2N 1N4

Orientation to Counselling and Guidance in TVET


Before Workshop

Workshop Evaluation Workshop Objectives The objectives of the workshop are:

1. to provide an overview of basic career development theory, 2. to identify factors that contribute to people’s career development, 3. to provide an overview of basic skills used in career-life planning 4. to provide a framework for integrating career-life planning into TVET curriculum.

First Some General Questions In answering these general questions, please compare yourself now and before the workshop. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now?



Graphically, the scale looks like this: ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞

Regarding the Primary Objectives, and

knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now?:

After Workshop

1. Clear understanding of basic career development theory

2. Knowledge about the factors that contribute to (or interfere with) people’s career development

3. Knowledge regarding basic skills used in career-life planning

4. Awareness of the importance of career-life planning in TVET

5. Repertoire of practical tools and approaches for facilitating the career development

Generally speaking,

6. how useful did you find the workshop? 7. how would you rate the workshop facilitation? 8. how would you rate the workshop facilities (room, etc.)? 9. how would you rate the food?

0 1 2 3 4


0 1 2 3 4



0 1 2 3 4


0 1 2 3 4


Now we would like your assessment of the learning you experienced in this workshop.

For each component of the workshop listed below listed below, please assess how useful that component was for you, using the 5-point scale below:

(0) Negligible, (1) Inadequate, (2) useful (3) very useful (4) Outstanding.

Graphically, the scale looks like this: ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞


Workshop Topic 0 1 2 3 4

1. General Model: Road Map 2. Exploring the Context 3. Factors Influencing Career Plans 4. Personal Career Line 5. Clarifying the Roles: Definitions 6. The Service Grid 7. Impact of Economic Realities on Work Life 8. Assets and Resources 9. Competency Assessment

10. Skill Framework 11. Skill Practice 12. Implementation: Accessing Services 13. Infrastructure 14. Action planning

What for you were the 3 best features of this workshop?

1.

2.

3.

What for you were the 3 features of this workshop that were less relevant for you?

1.

2.

3.

Please provide any additional comments about the workshop in the space below.

0 1 2 3 4



Orientation to Counselling and Guidance in TVET Workshop Evaluation: Summary of Responses

Workshop Objectives The objectives of the workshop are:

1. to provide an overview of basic career development theory, 2. to identify factors that contribute to people’s career development, 3. to provide an overview of basic skills used in career-life planning 4. to provide a framework for integrating career-life planning into TVET curriculum.

First Some General Questions

In answering these general questions, we would like you to compare yourself now and before the workshop. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now?




Regarding the Primary Objectives, and knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now?

Before Workshop

Ave

After Workshop

Ave

1. Clear understanding of basic career development theory 4 15 12 1 2 1.5 -- -- 1 14 19 3.5

2. Knowledge about the factors that contribute to (or interfere with) people’s career development 1 13 13 7 -- 1.8 -- -- 1 17 16 3.4

3. Knowledge regarding basic skills used in career-life planning 6 13 7 8 -- 1.5 -- -- 1 14 19 3.5

4. Awareness of the importance of career-life planning in TVET 3 11 12 4 3 1.8 -- -- -- 12 21 3.6

5. Repertoire of practical tools and approaches for facilitating the career development 5 18 7 4 -- 1.3 -- -- 1 20 13 3.4

Detailed description of results • Of the 170 ratings (5 questions times 34 people), 6 increased by 4 points, 41 increased by 3 points,

65 increased by 2 points, 47 increased by 1 point, and 7 remained the same. There were 89 ratings that changed from unacceptable to acceptable. All of the “after workshop” ratings are in the acceptable range and only 4 of those are ratings of minimally acceptable.

• Excepting those with a rating of 4, and 2 people with ratings of 3, on “before the workshop,” the self-assessments of all participants, showed increased scores on two or more of the 5 items above and none of the ratings went down.

• The people whose ratings did not change had high ratings before the workshop and there was little room for their scores to increase.

• Taken all together, the self-ratings pertaining to the workshop objectives indicate that participants experienced major gains in competence as a result of their participation in this workshop.


0 1 2 3 4

0 1 2 3 4



0 1 2 3 4


Generally speaking,

Ave 6. how useful did you find the workshop? -- -- -- 5 29 3.9 7. how would you rate the workshop facilitation? -- -- -- 9 25 3.7 8. how would you rate the workshop facilities (room, etc.)? -- -- -- 10 21 3.7 9. how would you rate the food? -- -- 5 14 14 3.6

Now we would like your assessment of the learning you experienced in this workshop. For each component of the workshop listed below listed below, please assess how useful that component was for you, using the 5-point scale below:

(0) Negligible, (1) Inadequate, (2) useful (3) very useful (4) Outstanding.

Graphically, the scale looks like this: ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞ ➞

Unacceptable Acceptable Workshop Topic

0 1 2 3 4 Ave

10. General Model: Road Map -- 2 4 15 13 3.1 11. Exploring the Context -- -- 3 13 18 3.4 12. Factors Influencing Career Plans -- -- 5 11 18 3.4 13. Personal Career Line -- -- 2 9 23 3.6 14. Clarifying the Roles: Definitions -- -- 2 9 23 3.6 15. The Service Grid -- -- 4 14 15 3.3 16. Impact of Economic Realities on Work Life -- 1 3 13 17 3.6 17. Assets and Resources -- -- -- 19 15 3.4 18. Competency Assessment -- -- 4 13 17 3.7 19. Skill Framework -- -- 3 8 23 3.6 20. Skill Practice -- 1 1 9 23 3.6 21. Implementation: Accessing Services -- -- 3 12 19 3.5 22. Infrastructure -- -- 1 13 19 3.4 23. Action planning -- -- 1 5 28 3.8

Detailed description of results • There was some variation in how useful participants found each of the activities in the workshop,

but the vast majority of participants rated all of the topics as 3 or 4 with regards to usefulness. • More than 80% of participants gave ratings of 3 or 4 for the usefulness of all workshop activities. • The above data, coupled with the open ended responses summarized below, indicate that the

participants found the group discussion particularly useful and the experiential learning format was a good fit for the vast majority of participants.

0 1 2 3 4


0 1 2 3 4



15. What for you, what were the 3 best features of this workshop? Similar comments are grouped together under a common heading. Numbers in parentheses indicate the number of people who made similar comments. No number after a comment indicates that only one person made that remark.

Facilitation • The method of conducting the workshop (4) • The group discussions were a great chance to explore other peoples

experiences in this field (3) • The discussions were very good (2) • Practical, clearness of presentations, authenticity of instructor

General comments regarding content

• Getting better definitions & a more clear understanding of career concepts (6) • Discovering some useful ways of evaluating our services (5) • Using many practical activities, not only theoretical, helped put theory into

practice (4) • Good foundation of the topics regarding vocational guidance concepts (2) • Comprehensiveness of topics covered (2) • The lectures gave us new skills and ideas to implement in our schools (2) • Practicing the skills used in career-life planning (2) • Integrating career-life planning • Well organized, useful content • Learning new ideas about careers • The theories were placed in our local context • I now know the basic concepts of career development

Comments on specific activities

• Action planning (11) • Communication skills (8) • Different services for different needs (5) • Post-pre evaluation (4) • Our personal career line (2) • Influences on career goals (2) • Wrap up – thorough review • Road map • Developing effective TVET programs • How to plan things by moving backwards, from outcomes to objectives to

methods

General • No comment (4) • The time of the workshop (adjusting the agenda times • A great chance to met an expert in this field


Before Workshop

Career Transitions Program Final Evaluation Form

Program Goals The goals of the program are that participants will:

1. formulate a clear understanding of what they want in their career (career vision) 2. articulate the things they want or feel they need to do in their life (life vision) 3. describe a framework for the process of building a career 4. acquire the skills needed to explore career alternatives 5. acquire an adequate repertoire of job search skills

First Some General Questions

In answering these general questions, we would like you to compare yourself now and before the program. Knowing what you know now, how would you rate yourself before the program, and how would you rate yourself now?


(0) unacceptable, (1) not really acceptable, but almost there, (2) minimally acceptable (but still OK otherwise it would be 1 or 2), (4) exceptional, or (3) somewhere between minimally acceptable and exceptional.


Regarding the Program Goals, and knowing what you know now, how would you rate yourself before the program, and how would you rate yourself now?

Before Workshop

Now

1. Clearness of your vision of what you want in your career

2. Knowledge from a thorough assessment of your qualities, skills, interests, and experiences

3. Knowledge from in-depth research on possible career options

4. Knowledge of a framework for the process of building a career

5. Confidence that you have sufficient qualifications, knowledge, skills and abilities to get your career started

6. Confidence that work opportunities actually exist that align with what you want in your career

7. Certainty in having a definite career goal that you wish to pursue

8. Optimism about what lies ahead in terms of your career


Regarding the Specific Activities you completed, and knowing what you know now, how would you rate yourself before the program, and how would you rate yourself now?

Before Workshop

Now

9. The adequacy of your career portfolio

10. The adequacy of your research portfolio, i.e., information about career options that align with their career vision

11. The adequacy of your professional communication skills for connecting with people in order to get direct, first-hand information about career options

12. The adequacy of your knowledge of resources for generating career options and identifying people and organizations for career research

13. To what extent would you say that any changes depicted above were the result of

completing the Career Transitions Program, and to what extent were they a function of other factors in your life?




mostly this program

14. Are you currently working? Yes No 15. If� you answered yes to the above question,

to what extent is this work related to your career path? �

16. When you are actively engaged in career/work search, how

confident are you that your activities are effective? �

17. Generally speaking, how effective are your written job search tools (e.g., resumes, cover letters, business cards, etc.) are? �

18. How effective are your verbal communication for career situations such as calls, meetings, and interviews? �

19. How effective are your strategies for maintaining momentum and motivation in quest to implement your career path? �

Eval

uatio

n (E

vide

nce

Gat

herin

g) T

ools

p.

40

Car

eer T

rans

ition

s Pr

ogra

m

Prog

ram

Eng

agem

ent:

Part

icip

ant F

orm

M

odul

e 1,

Uni

t 1

In w

orki

ng th

roug

h M

odul

e 1,

to w

hat e

xten

t ha

ve y

ou c

ompl

eted

the

follo

win

g ac

tiviti

es?

How

use

ful d

id y

ou fi

ne th

e ex

erci

ses?

In

resp

ondi

ng, p

leas

e us

e a

two-

step

pro

cess

. (A

) dec

ide

on w

heth

er th

e de

gree

of u

sefu

lnes

s w

as a

ccep

tabl

e or

una

ccep

tabl

e, th

en

(B) a

ssig

n th

e ap

prop

riate

ratin

g:

(0) u

nacc

epta

ble,

(1

) not

real

ly a

ccep

tabl

e, b

ut a

lmos

t the

re,

(2) m

inim

ally

acc

epta

ble

(b

ut s

till O

K o

ther

wis

e it

wou

ld b

e 1

or 2

), (4

) exc

eptio

nal,

or

(3) s

omew

here

bet

wee

n

min

imal

ly a

ccep

tabl

e an

d ex

cept

iona

l.

Act

ivity

D

idn’

t do

it

Sort

of

did

it D

id it

th

orou

ghly

Uni

t 1 E

xerc

ises

• Th

e B

ig P

ictu

re w

orks

heet

• Li

st o

f De-

mot

ivat

ors

•

List

of P

ossi

ble

Car

eer

Opt

ions

• C

ore

Mot

ivat

ors

wor

kshe

et

•

Rea

lity

Che

ck B

rain

stor

m

•

Car

eer-

build

ing

Fram

ewor

k

(car

eer v

isio

n an

d cu

rren

t situ

atio

n)

Did

you

par

ticip

ate

in th

e gr

oup

for M

odul

e 1,

Uni

t 1:

Y

es

N

o

Eval

uatio

n (E

vide

nce

Gat

herin

g) T

ools

p.

41

For p

artic

ipan

ts w

ho a

nsw

er Y

es to

the

abov

e qu

estio

n, p

ose

the

follo

win

g ad

ditio

nal q

uest

ions

. H

ow u

sefu

l was

the

grou

p ex

perie

nce

to y

ou?

In re

spon

ding

, ple

ase

use

a tw

o-st

ep p

roce

ss.

(A) d

ecid

e on

whe

ther

the

degr

ee o

f use

fuln

ess

was

acc

epta

ble

or u

nacc

epta

ble,

then

(B

) ass

ign

the

appr

opria

te ra

ting:

(0

) una

ccep

tabl

e,

(1) n

ot re

ally

acc

epta

ble,

but

alm

ost t

here

, (2

) min

imal

ly a

ccep

tabl

e (b

ut s

till O

K o

ther

wis

e it

wou

ld b

e 1

or 2

), (4

) exc

eptio

nal,

or

(3) s

omew

here

bet

wee

n m

inim

ally

acc

epta

ble

and

exce

ptio

nal.

To

wha

t ext

ent d

id p

artic

ipat

ing

in th

e gr

oup

he

lp to

mak

e th

e w

orks

hop

resp

onse

s m

ore

clea

r?

gave

you

mor

e m

otiv

atio

n to

com

plet

e th

e w

orks

heet

s?

help

you

to c

larif

y w

hat i

s th

e ne

xt s

tep

you

wan

t to

take

in y

our c

aree

r?

help

to in

crea

se y

our s

elf-c

onfid

ence

?

he

lp y

ou c

ompl

ete

the

prog

ram

in a

mor

e th

orou

gh w

ay?

To w

hat e

xten

t was

the

grou

p a

sour

ce o

f sup

port

for y

ou?


Career Transitions Program Program Engagement: Counsellor Form

(Completed by the Career Coach after each interview. Note, this information is already being collected. The form has been revised lightly to make it more parallel to the participant engagement form)

To what extent did you review the following topics in the participant interview?

Activity Didn’t do it

Sort of did it

Did it thoroughly

Interview #1: Date __________________ Discuss general reactions to the program. Any “ah-ha’s”? Any frustrations?

Unit 1 Exercises • The Big Picture worksheet

• List of De-motivators

• List of Possible Career Options

• Core Motivators worksheet

• Reality Check Brainstorm

• Career-building Framework (career vision and current situation)

Discussion of next steps General Notes:


Action Planner

Action Plan for

(write in name of lifestyle area you want to work on)

General Goal (write in goal statement)

Action steps Date completed Evidence Finished

(check)

1.

2.

3.

4.

Special learning barriers to overcome Plan to overcome the barriers

Plan for sustaining motivation Reward for completing goal

compendium of evaluation tools - crwg - gdrc€¦ · meeting workplace skills: the career...

Documents