1 author: rogahn, mary ellen a manager coaching training

1

Author: Rogahn, Mary Ellen Title: A Manager Coaching Training Course at XYZ: An Evaluation of

Effectiveness and Skill Transfer The accompanying research report is submitted to the University of Wisconsin-Stout, Graduate School in partial completion of the requirements for the Graduate Degree/ Major: MS Training and Development

Research Advisor: Dr. Jeanette Kersten

Submission Term/Year: Summer, 2013

Number of Pages: 65

Style Manual Used: American Psychological Association, 6th edition

I understand that this research report must be officially approved by the Graduate School and that an electronic copy of the approved version will be made available through the University Library website

I attest that the research report is my original work (that any copyrightable materials have been used with the permission of the original authors), and as such, it is automatically protected by the laws, rules, and regulations of the U.S. Copyright Office.

My research advisor has approved the content and quality of this paper.

STUDENT:

NAME: Mary Ellen Rogahn DATE: July 19, 2013

ADVISOR:

NAME: Dr. Jeanette Kersten DATE: July 19, 2013

----------------------------------------------------------------------------------------------------------------------

This section to be completed by the Graduate School

This final research report has been approved by the Graduate School.

Director, Office of Graduate Studies: DATE:

2

Rogahn, Mary Ellen. A Manager Coaching Training Course at XYZ: An Evaluation of

Effectiveness and Skill Transfer

Abstract

Improving employee performance is an area of constant scrutiny among

organizations large and small. Business coaching is becoming an increasingly important

strategy that organizations are using to increase overall employee performance. Company

XYZ decidedly uses coaching as a strategy to drive improved employee performance and

implements a global roll-out of a coaching skills training program. This study is an

examination of the effectiveness of that program and will assist Company XYZ in

identifying areas of focus to sustain the coaching skills and coaching behaviors among

their global manager population.

3

Acknowledgments

Sincere thanks to my advisor, family and friends who have supported my study and

success as a graduate student. I appreciate and value all of you:

To my advisor, Dr. Jeanette Kersten: You have been a constant source of validation and

motivation. I appreciate your advice, guidance and availability throughout the past year. To my

children, Mykenzie and Madisyn: I appreciate your support of my work and hope I have provided

you with an inspiration for setting a goal and seeing it through. To my parents: You are the role

models of hard work and dedication. Finally, to my friends: You provided support and

encouragement that gave me the motivation to press on; thank you.

4

Table of Contents

.................................................................................................................................................... Page

Abstract............................................................................................................................................2

List of Tables .............................................................................................................................. 6

Chapter I: Introduction.....................................................................................................................8

Statement of the Problem ................................................................................................. 9

Purpose of the Study ...................................................................................................... 10

Assumptions of the Study .............................................................................................. 10

Definition of Terms ....................................................................................................... 11

Limitations of the Study ................................................................................................ 11

Methodology ................................................................................................................. 12

Chapter II: Literature Review .................................................................................................... 14

Key Authors, Contributors & Models used in Business Coaching .................................. 14

Standards and Competencies ......................................................................................... 18

Assessments used in Business Coaching ........................................................................ 19

Training Course Evaluation ........................................................................................... 20

Summary ....................................................................................................................... 23

Chapter III: Methodology .......................................................................................................... 24

Subject Selection and Description .................................................................................. 24

Instrument Selection ...................................................................................................... 24

Data Collection Procedures ............................................................................................ 25

Data Analysis ................................................................................................................ 25

Limitations .................................................................................................................... 26

5

Summary ....................................................................................................................... 26

Chapter IV: Results ................................................................................................................... 27

Item Analysis ............................................................................................................... 27

Qualitative Analysis ..................................................................................................... 39

Summary ...................................................................................................................... 44

Chapter V: Summary, Conclusions and Recommendations........................................................ 46

Summary of the Study .................................................................................................. 46

Limitations ................................................................................................................... 47

Factors Impacting Survey Effectiveness........................................................................ 48

Conclusions of Item Analysis ....................................................................................... 50

Conclusions and Recommendations .............................................................................. 52

References ................................................................................................................................ 54

Appendix A: Permission for Study – Company XYZ ................................................................ 57

Appendix B: UW- Stout Implied Consent Statement for Research Involving Human Subjects .. 58

Appendix C: Permission to Use Survey Tool ............................................................................. 60

Appendix D: Invitation to Survey Participants & Implied Consent ............................................ 61

Appendix E: Coaching Conversations Skill Retention Survey ................................................... 62

6

List of Tables

........................................................................................................ …………………...……Page

Table 4.1: Question # 1 – The class environment helped me learn ................................................ 28

Table 4.2: Question # 2 – There were no major distractions that interfered with my learning ........ 28

Table 4.3: Question # 3 – The program material will be helpful for my success in the future ........ 29

Table 4.4: Question # 4 – I immediately applied my learning in my job........................................ 30

Table 4.5: Question # 5 – I was engaged with what was going on during the program .................. 31

Table 4.6: Question # 6 – The activities and exercises aided in my learning ................................. 31

Table 4.7: Question # 7 – I was given adequate opportunity to demonstrate what I was

learning……………………………………….…..……………………………….…...…32

Table 4.8: Question # 8 – The program met my expectations ........................................................ 33

Table 4.9: Question # 9 – I am clear on how to apply what I learned on the job ............................ 34

Table 4.10: Question # 10 – I would recommend this program to my coworkers .......................... 35

Table 4.11: Question # 11 – How confident are you that you will be able to apply what you have

learned back on the job?……....…………….…..…………………………………..……36

Table 4.12: Question # 12 – If you chose somewhat confident or not at all confident, please circle

all that apply. My confidence is not high because………….....……………….….…….37

Table 4.13: Question # 13 – How committed are you to applying what you learned in your

work?.......................................................................………………………….….………38

Table 4.14: Question # 14 – If you chose somewhat committed or not at all committed to

Question 13 please circle all that apply. My commitment is not high because…...……38

Table 4.15: Question # 15 – What barriers do you anticipate that might prevent you from applying

what you learned?...................................................………………………….….………40

7

Table 4.16: Question # 16 – What might help you overcome those barriers?................................41

Table 4.17: Question # 17 – What outcomes are you hoping to achieve as a result of your

coaching efforts?………………………….…………………………………….………41

Table 4.18: Question # 18 – What support do you need?...............................................................43

Table 4.19: Question # 19 – What evidence have you observed that indicates coaching is having a

positive impact on business results?...…….…………………………………….….…..44

8

Chapter I: Introduction

Companies spend billions of dollars annually to improve employee performance, and

organizations often struggle with quantifying the return on those dollars (Miller, 2012). The

decisions of what to invest in and what activities will yield the highest and best use of a

company’s development dollars remain an area of constant scrutiny. As leaders at all levels

struggle with increased complexity and challenging business environments, organizations have a

renewed focus on investing in activities and development methods to motivate, engage and retain

employees. This is especially true among global organizations (Miller, 2012). Business coaching

is one approach organizations are using to improve their return on investment and it has become

an important strategy for corporate leadership development and change management practices

(Bennett & Bush, 2009).

The coaching industry has become a $2 billion annual global market (International Coach

Federation, 2012). International Coach Federation (ICF), an organization that supports the

industry of professional coaching, has increased membership from 30,000 in 2006 to almost

47,500 by the end of 2011 – providing clear evidence of the increased demand (International

Coach Federation, 2012). In the past decade, the management literature of Harvard Press, T&D

Magazine, OD Practitioner and other popular publications have become filled with articles on the

benefits of coaching and mentoring on both personal and business performance productivity.

Companies that have used professional coaching for business reasons have seen a median return

on their investment of seven times their initial investment (International Coach Federation, 2009).

Returns are expressed in higher productivity, improved quality, increased individual and

organizational effectiveness and increased retention.

9

Historically, business coaching has been a process reserved for senior leaders, but recent

emphasis in comprehensive talent development strategies has shifted this activity to a global

necessity for all levels of an organization (Zeus & Skiffington, 2001). This shift is the catalyst for

the development of coaching skills among managers tasked with developing talent in

organizations. A 2011 study by Bersin and Associates reported that the inability of a manager to

effectively coach was the number one performance management challenge organizations face

(Pace, 2011). In order for management to successfully coach in an organization, it is important to

determine the significance and purpose of coaching. “The aim of coaching is to engage in a

conversation focused on discovery and actions that will help a person achieve their desired

outcome and become self-directed in their learning and development” (Bennett & Bush, 2009, p

2). As employees engage in their own development, managers shift to an enabling style of

managing and increase the success they have in dealing with difficult operational and

performance issues (Van Velsor, McCauley, & Ruderman, 2010).

Statement of the Problem

From 2010-2012, Company XYZ invested in a manager coaching training curriculum to

train 7,200 global managers in the skills of coaching. The program, part of a comprehensive

talent management strategy, was a skill-based curriculum designed to build coaching

competencies for the global manager population. Company XYZ decided to integrate this skill-

based intervention based on the results of an annual employee satisfaction survey specifically

targeting manager effectiveness from the employee’s perspective. 30,000 employees provided

feedback on 3,000 managers, representing almost half of the global manager population.

Collectively, the feedback clearly indicated employees were significantly dissatisfied with

manager performance as it related to ongoing coaching and providing feedback on job growth and

10

career development. As a result of the feedback, senior management immediately requested to

close the gap and develop a broader skill set among the global management population to include

core coaching skills for any manager with at least one direct report. The program specifically

targeted 65 percent-75 percent of the classroom experience as learning application to impact

learning retention and improve learning transfer to real-time job application. To date, 75 percent

of managers have completed the course and no measurement of learning application has been

collected. Without a measurement of learning application, Company XYZ cannot identify the

overall effectiveness of their investment nor can they identify appropriate sustainability programs

to aid ongoing skill development.

Purpose of the Study

This study will evaluate the effectiveness of a global manager coaching program in meeting

business goals of increasing the coaching application in employee manager interactions and

overall application of coaching skills on the job. A survey will be developed to specifically

evaluate on the job application of skills learned in the classroom experience. The study will target

executive band leaders who completed the global manager coaching program in the last 18

months. Participation will be voluntary. This study acknowledges that organizations have many

interventions available to improve skills performance; therefore, isolating the impact of the skills

applied or not applied will aid the company in assigning additional resources to solve the business

need and improve the return on investment of the global manager coaching program.

Assumptions of the Study

This study assumes that:

1. The researcher will receive an adequate response rate

2. The findings can be generalized beyond this population

11

3. The results will show improved productivity for individuals and business

4. The coaching skills and practices will be globally transferable

Definition of Terms

Coaching. “Coaching is a means of developing people to enable more effective

performance and fulfillment of potential” (Zeus & Skiffington, 2001).

Skills Coaching. “Skills coaching is coaching that is specifically focused on the

employee’s current tasks” (Zeus & Skiffington, 2001).

Performance Coaching. “Performance coaching is coaching that is specifically focused on

the employee’s performance” (Zeus & Skiffington, 2001).

Developmental Coaching. “Developmental coaching is coaching that is specifically

focused on employee’s future career or a new leadership role” (Zeus & Skiffington, 2001).

Leader Development. “Leader development is the expansion of a person’s capacity to be

effective in leadership roles and processes” (Corporate Executive Board Company, 2012).

Leadership Development. Leadership development is the expansion of a collective

capacity of organizational members to engage effectively in leadership roles and processes in

order to produce direction, alignment and commitment” (Van Velsor, McCauley, & Ruderman,

2010).

Limitations of the Study

While the survey will provide further insights into the effectiveness of a global manager

coaching program in meeting business goals, there are several limitations the researcher must take

into consideration to evaluate and interpret the findings. The following are a few of these

limitations:

1. The sample is limited to executive band leaders at Company XYZ.

12

2. Participation is voluntary and leaders may elect to disregard the survey impacting the

amount of responses and quality of the overall findings.

3. Findings from this survey at Company XYZ are predicted to change over time with the

addition of sustainability efforts, which will vary by region.

4. The study has excluded the population below executive band, which may be considered for

further study.

Methodology

A survey will be developed to measure effectiveness of a global manager coaching program.

The survey will be sent to executive band leaders who have completed the training in the last 18

months and participation will be voluntary. The survey will be developed using a template from

the Kirkpatrick Partners to ensure the use of a validated instrument (Kirkpatrick, 2006). Data

collection will be through Company XYZ’s internal survey tool to promote compliance on the

request. Invitation for response will be sent via email to expedite and encourage responses.

Results of the survey will be analyzed using Kirkpatrick’s four level evaluation methodology,

which consists of reaction, learning, behavior and business results (Kirkpatrick & Kirpatrick,

2006).

“Successful executive coaching facilitates growth, positive change and improved

performance” (Hauser, 2009, p 8). Company XYZ continues to invest in the growth and

development of their global manager population. The global manager-coaching program is both a

financial and a productivity investment. Without measuring the extent to which leaders engage in

coaching activities and behaviors, Company XYZ will not be able to clearly define opportunities

for the sustainability of those activities and behaviors. This evaluation will assist in identifying

current application of skills as well as the potential need for skill reinforcement support. Chapter

13

I presented the business case and purpose of the study. Chapter II will provide a review of the

literature associated with this study. Chapter III will provide an in-depth review of methodology

and methods used in this study. Chapter IV will present the results and Chapter V will provide a

discussion of the results and recommendations for further application and research.

14

Chapter II: Literature Review

The literature review for this study consists of four interrelated areas that are foundational

to the research and deployment of a global manager coaching program. The researcher focuses on

areas that support continued education in coaching as well as opportunities to sustain the skills of

coaching among participants who have attended the global manager coaching program. The first

review examines key contributors to the field of coaching and models used in business coaching.

The second review explores the standards & competencies used in business coaching. The third

review examines the use of assessments and tools in business coaching. Finally, the fourth review

examines the process of evaluation specific to classroom learning and learning transfer with focus

on Kirkpatrick’s four levels of evaluation.

Key Authors, Contributors & Models used in Business Coaching

Business coaching has been rooted in the field of behavioral psychology and aligned with

Frederick Hertzberg’s theory that individuals are motivated by five key factors inherent in their

work: challenge, growth, sense of contribution, recognition and responsibility (Hertzberg, 1987).

Coaching can impact employee engagement by stimulating thought and action in each of the five

key areas. Business coaching has evolved in the past ten years from an intervention that placed

toxic behavior at the top to an integrated talent management process that developed the

capabilities of high-potential employees (Coutu & Kauffman, 2009). Organizations use coaching

to supplement training, mentoring and on-the-job assignments and to meet emerging leadership

development needs (Velsor, McCauley & Ruderman, 2010). These emerging needs include

developing high-potentials, preparing leaders for future roles and helping strong performers

develop new skills to meet business opportunities.

15

The rationale of using a business coach to improve productivity for both the individual

and the organization has grown increasingly stronger in the past few years (International Coach

Foundation, 2012). This has been validated by third-party research conducted by organizations

such as the Corporate Executive Board (CEB), the International Coach Federation (ICF) and the

Center for Creative Leadership (CCL). Coaching has been determined to help people and groups

become self-directed in their learning and development (Bennett & Bush, 2009). “Coaches at

their best are change agents who understand the value of creating a resilient workforce capable of

adapting to a constantly-changing environment” (Locke, 2008). When a coach is able to adapt to

the surroundings they are dealing with, positive results ensue. For example, “research shows that

companies that utilize a coaching approach to learning experience 50% less turnover, 56% greater

customer loyalty, and 27% more profitability” (Berard, 2005). Additional benefits are seen in the

relationship between the manager and the employee. One study reports that “eighty-five percent

of participants gave credit to the coaching process in stating a noticed improvement in their

manager” (Wenson, 2010). Many professionals have capitalized on the opportunity to meet the

growing business need to provide coaching and several have risen as thought leaders contributing

to the expertise of the field.

Sir John Whitmore wrote the book Coaching for Performance, which has served as a

platform to the GROW model for coaching. GROW, an acronym for Goal, Reality, Options

(or Obstacles), Will (or Way Forward) is a sequencing process used to format questions to

advance the dialog (Whitmore, 2002). The GROW model provides a framework for a coach to

advance the dialogue into actionable steps (Whitmore, 2002). Whitmore’s work began in the

field of athletics in the 1960s and most recently expanded in the areas of motivation, meaning,

16

purpose and barriers in a coaching relationship (Whitmore 2002). Now in the public domain,

the GROW model has served as a primary guide to a coaching dialogue for many practitioners.

Frederic Hudson, author of The Handbook of Coaching as well as career- and life-

planning consultant for over 30 years, has been considered an expert in adult development

(Hudson, 1999). Together with Pamela McLean, author of The Completely Revised Handbook of

Coaching (2012), he operates The Hudson Institute of Santa Barbara; an organization recognized

globally as a premier leadership and coaching training institution. The core premise of the

institute is that effectiveness of a coach begins with Self-as-Coach, the self-awareness of the

coach (Hudson, 1999). Through the examination of self, the possibility exists to highlight

something in the self of the coach that can either promote or undermine a client’s abilities to

achieve the proposed changes. Coach training has continued expansion into the theory and

practice of coaching, including extensive reading and several practicums. Most complete this

training within one year. The institute’s most recent work, Spot Coaching Approach™, has been

defined as a three-stage coaching process that builds coaching competency for managers or

leaders through everyday coaching conversations (Hudson, 1999).

Zeus and Skiffington (2003) advanced the profession of coaching by mapping the

coaching process to fundamental behavioral principles of change and transition. The authors’

underlying premise is that the role of the coach is to support and guide the coachee through a

transition, citing that groups who have been trained to manage change make better leaders,

perform better and have greater morale and health (Zeus & Skiffington, 2003). The authors’ work

has led to the creation of a premier coaching school and has become an industry standard for

materials used throughout the coaching industry.

Since 1970, the Center for Creative Leadership (CCL) has been devoted to the

17

research and education of leadership development. Their research in leadership has concluded

that leadership development is an ongoing process and that growth for individuals is accelerated

when individuals are assessed, challenged and supported. The Assess, Challenge, Support model

forms the basis of the CCL Coaching Framework (Van Velsor, McCauley & Ruderman, 2010).

According to the CCL, leadership coaching is a practice in which the coach and the person being

coached – the coachee – collaborate to assess and understand the coachee and his or her

developmental needs, challenge current constraints while exploring new possibilities and ensure

accountability and support for reaching goals and sustaining development (Van Velsor, McCauley

& Ruderman, 2010).

Researcher Elouise Leonard-Cross set out to explore whether developmental coaching

delivers business results and found a correlation between coaching and an individual’s self-

efficacy levels (Leonard-Cross, 2010). Self-efficacy, described as an individual’s personal

judgment of their own ability to succeed, is seen as a key variable altered by developmental

coaching (Leonard-Cross, 2010). The capacity for self-efficacy supports a critical component in

effective coaching, building awareness and responsibility (Whitmore, 2002).

Researchers Heslin, Vandewalle and Latham (2006) examine “Implicit Person Theories”

(IPTs), which suggest that the malleability of personal attributes (e.g., personality and ability)

influence a manager’s willingness to help others improve performance. This in turn impacts the

coaching relationship as well as the frequency in which the manager engages in coaching activity.

This provides some rationale for the research from a Bersin and Associates report that found that

only 11 percent of senior leaders coach their employees and only 15 percent “very frequently”

discuss the importance of coaching and development (Pace, 2011). However, companies are

attempting to improve these statistics. “The research from 214 organizations spanning a variety

18

of geographies and industries shows that more companies are strategically integrating their talent

management processes to lead, manage, develop, reward and assess employees using a coaching

and development model” (Pace, 2011). Incorporating talent management processes in this

manner affects the personal attributes of managers and allows for better coaching.

Standards & Competencies for Business Coaching

The industry’s governing bodies that set standards, core competencies and ethics for

coaches and training schools continues to evolve. To date, no one organization has universal

authority, but the industry does have a few organizations, such as the European Mentoring and

Coaching Council (EMCC) and the International Coach Federation (ICF), that have made great

strides to move in this more formally-recognized direction. Journals devoted to organizational

coaching, such as the International Journal of Evidence-based Coaching and Mentoring, the

Coaching Psychologist and the International Journal of Coaching in Organizations, emerged at

the turn of the century. The International Coach Federation’s membership grew from around

30,000 in 2006 to almost 47,500 by the end of 2011, thus providing clear evidence of the increase

in the growth of the coaching profession (International Coach Federation, 2012). ICF has

established 11 core coaching competencies identified to support greater understanding of skills

and approaches used within today's coaching profession (International Coach Federation, 2012).

ICF’s core competencies are grouped into four clusters based on common ways of looking at

them. The four clusters representing the 11 competencies are: setting the foundation, co-creating

the relationship, communicating effectively and facilitating learning and results (International

Coach Federation, 2012). The competencies have been used as the foundation for the ICF

credentialing process examination. In addition to course work, documented hours provide

evidence of competency and funnel into a hierarchy of credentials practitioners can assume and

19

market in the respective practices. As the industry of business coaching expands, the movement

has created barriers to entry through standards and competencies that will continue to evolve.

Assessments used in Business Coaching

The Center for Creative Leadership (CCL) has researched the field of leadership

development for over 40 years. A core premise of this research is that leadership development is

an ongoing process and that assessment is a critical component in support of that process (Van

Velsor, McCauley & Ruderman, 2010). According to the CCL, a three-stage process of

assessment, challenge and support is critical because it provides individuals with an

understanding of the level of current performance potential development needs (Van Velsor,

McCauley & Ruderman, 2010). The use of an assessment examines the gap between the current

performance or reality and the desired performance. The gap then becomes the basis for the work

in the coaching relationship.

Assessments have been widely used in support of coaching relationships. Assessments

provide data to both the client and the coach. This data can be collected by conversations with the

client; however, psychometrics or assessments provide a shortcut for collecting information about

the client and establish a common language for the coach and the client (Rogers, 2004).

Assessments are formal or informal and have been a collaboration of self and other individuals.

Examples of formal assessments include 360-degree feedback, performance appraisals,

organizational surveys, customer data and psychometrics in conjunction with external consultants

(Sherpa Coaching, 2013). Informal assessments include observations, solicited and unsolicited

feedback from the boss and peers and self-assessments. Self-assessment also includes

psychological inventories, analysis of decisions and mistakes (Rogers, 2004). Some assessments

require a certification process prior to use while some do not. The costs of using assessments

20

range from free-of-charge to thousands of dollars. The use of an assessment provides information

about the client that is not presently known and assists in the development of the client (Rogers,

2004). Key questions a coach would ask relating to the use of an assessment are: What will they

do with the new information? How can this information become a way for the client to gain

insights that were not available to them before the assessment? In the end, an assessment is only

as effective as the coaching discussion that followed it (Rogers, 2004).

Training Course Evaluation

According to the Corporate Executive Board (CEB), learning programs with the highest

application rates improved learner performance by 51 percent and business results by 28 percent

(Corporate Executive Board Company, 2012). The Corporate Leadership Council (CLC), part of

the CEB, researched and found that while 77 percent of learning and development functions

prioritized the improvement of learning measurement, in this study only 24 percent were

determined effective at measuring overall performance and only 18 percent were effective at

communicating the impact (Corporate Executive Board Company, 2012). Given the prioritization

of improving learning measurement and the lack of effectiveness in executing learning

measurement, the CLC suggest a three-step process to build a practical and sustainable

measurement approach.

Step 1: Identify appropriate measurement methodology to demonstrate business impact.

Step 2: Boost line involvement by involving stakeholders and aligning expectations.

Step 3: Deliver insights effectively by demonstrating the impact of the learning and

development function (Corporate Leadership Council, 2012).

Evaluation of the training program occurred at the end of the training program but the

work involved in creating evaluations that measure learning, behavior change and business results

21

started at the beginning of the instructional design process (Kirkpatrick & Kirkpatrick, 2006).

Donald Kirkpatrick began the work of clarifying and defining evaluation in 1959 when he

authored four articles titled Techniques for Evaluating Training Programs. Since then, the

Kirkpatrick process has been considered foundational to evaluating training programs.

Kirpatrick’s work recognized four distinct levels in evaluation which he concluded are all

necessary in an overall course evaluation process:

Level 1 – Reaction, a measurement of satisfaction of the learning experience.

Level 2 – Learning, a measurement of the extent to which learning occurs.

Level 3 – Behavior, a measurement of the extent to which behavior changes as a result of

the training.

Level 4 – Results, a measurement of the extent to which business results are met/impacted

by the training (Kirkpatrick & Kirkpatrick, 2006).

Movement from level two to level three evaluations is focused on evidence of behavior

change from classroom to workplace. Evaluators ask: What activities followed the classroom

experience to help reinforce the learning? Did the participant engage in the activities to reinforce

the learning? The Kirkpatrick model provides a platform to show evidence to support each level

of evaluation. Jim Kirkpatrick continued his father’s legacy in training evaluation and advocating

the business case for sound evaluation. According to Kirkpatrick, “level three is the forgotten

evaluation level, yet it is the key to maximizing training and development effectiveness. After all,

what is training and learning unless it’s applied?” (Kirkpatrick, 2007). The evaluation process is

about collecting a chain of evidence that proves the training is effective. The final level, level

four, is about return on expectations (ROE) – expectations of the business stakeholders who

commission the training. The business partnership is the cornerstone of the four levels of

22

evaluation since it is the expectations of business stakeholders, clearly defined at the beginning of

the process, that substantiate the ROE (Kirkpatrick, 2007). Once the desired business impacts are

defined, the next step is to identify the process improvements that enable them (Levenson, 2009).

A 2009 study illustrated these linkages of coaching and business impact and examined 12

coaching engagements. Conclusions from the study supported that coaching should have an

impact on business results if it improved decision making and/or execution (Levenson, 2009).

“To show business impact, it is necessary to demonstrate an impact on either strategic or financial

performance, but not necessarily both” (Levenson, 2009).

The Success Case Method, developed by Robert O. Brinkerhoff, provides analysis of

extreme groups within a training program and communicates those results through case study and

storytelling. The essential purpose of a Success Case study is to find out how well a training

program is working by identifying what is helping or hindering the learning application

(Brinkerhoff & Dressler, 2002). The methodology for the Success Case Method follows a two-

step process. Step one is to survey a sample of participants who are both successful and

unsuccessful in learning application to assess the extent to which they apply their training to

impact business. Step two is to interview participants to determine both the nature and business

value of their learning application and identify and explain the contextual factors that helped or

hindered their respective results (Corporate Leadership Council, 2012). Information gathered

through steps one and two were shared with stakeholders in terms of evaluation of training results

(Corporate Leadership Council, 2012).

A strategic focus on evaluation demonstrates value, deepens partnerships and promotes

collaboration with multiple stakeholders (Derven, 2012). Several professional and membership

organizations, such as the American Society of Training and Development (ASTD), the Return on

23

Investment (ROI) and the Corporate Executive Board, have been available to assist with

evaluation efforts by providing guidelines, models, training and certification in evaluation

methodology. For the purposes of this research, the Kirkpatrick model will be used.

Summary

The review of literature examines four areas of importance for the success of a global

manager coaching program. The core premise of a coaching relationship was that the learner or

coachee owns their development. Coaching is a learning process that promotes awareness and

accountability for the person being coached. Measuring effectiveness is an important step in

determining the required actions that will support a classroom learning investment by identifying

the potential areas of sustainability investment that reveal the greatest impact. Chapter III will

discuss the approach to surveying the participants that participated in the global manager

coaching program to examine learning transfer and identify sustainability opportunities.

24

Chapter III: Methodology

The purpose of this study was to evaluate the effectiveness of Company XYZ’s global

manager coaching program. Specifically, the target of the study was the behavioral change impact

from the training, level three of Kirkpatrick’s four-level evaluation model (Kirkpatrick &

Kirkpatrick, 2006). The study utilized Company XYZ’s internal electronic survey tool to collect

individual data from participants who have completed the course in the last 18 months. The

results of the study provided direction in the identification and development of sustainability

plans for all participants.

Subject Selection and Description

All Company XYZ executive band leaders of the organization who have completed the

course in the past 18 months were asked to complete the survey (see Appendix E). At the time of

this study, there were an estimated 358 executive band leaders who completed the course.

Participation in this evaluation was voluntary. The senior human resource representative from

Company XYZ granted permission to administer to the target population (see Appendix A).

Informed consent was provided in the invitation to the survey (see Appendix B). The invitation to

the survey was administered via email (see Appendix D).

Instrument Selection

The survey selected for this study was the amended L1 Hybrid course evaluation from

Kirkpatrick partners (see Appendix E). The permission for use was obtained by direct contact

with the developer (see Appendix C). The developer also approved the conversion of the paper

instrument to an online instrument.

The evaluation consisted of 20 questions. The evaluation was converted to an online version

using Survey Central, Company XYZ’s online survey tool, with permission from the senior

25

human resource manager for Company XYZ.

Survey Central is available at no cost and, as an internal online tool, offers a less expensive

method and increases the potential for a higher response rate. The online tool allowed participants

to complete the survey at their convenience and provide researcher-timely results.

Data Collection Procedures

All Executive Band leaders at Company XYZ that have completed the course were

identified for inclusion in the study. The researcher prepared an email invitation to include the

survey link and an overview of the study. The email invitation included information about the

study, including how the data will be used, the anticipated length of time for completing the

evaluation, and option to decline response to the evaluation. Information about who to contact

with questions and the anonymous online survey link was also included (see Appendix D). No

personal identifying information was collected and the data was immediately analyzed. The

online tool collected individual data from executive leaders representing seven global regions as

designated by Company XYZ. Participants were asked to respond to multiple-choice statements

with the option to add comment. After reading each statement, respondents were asked to

indicate the extent to which they agree with the statement, from strongly agree to strongly

disagree.

Once submitted, participants were not able to withdraw, as data was co-mingled with other

responses anonymously. Participants received information about informed consent at the

beginning of the survey.

Data Analysis

The data source was individual survey data compiled through an online version of the

survey instrument. Survey data was compiled and response rates were calculated for the entire

26

survey and for individual questions. The results of this study will be used to assist Company

XYZ in developing plans for sustainability of coaching skills taught in the classroom. Analysis

will focus on the engagement in coaching activities following the classroom learning experience.

Limitations

While this study is extremely useful and successful in many areas, there are a few

limitations. These include:

1. The sample is limited to Executive Band leaders at Company XYZ.





4. The study has excluded the population below Executive Band, which may be considered

for further study.

Summary

This chapter describes an introduction to the methodology, instrument selection, sampling,

data collection, data analysis and limitations. Chapter IV will provide results of the study,

followed by a discussion, including limitations, conclusions and recommendations based on the

study in Chapter V.

27

Chapter IV: Results

This study is designed to assist Company XYZ assess skill retention from a global manager

coaching skill development program. In understanding the current coaching skills retention of

executive band leaders, opportunities for improvement can be identified. Opportunities for

improvement will be generalized to include the full manager population.

The survey examines responses received through the online survey tool. Two hundred

seventeen leaders were identified and invited to participate in the survey. Fifty-seven individuals

responded to the survey, resulting in a response rate of 26 percent. Survey participants were

given a link to the online survey tool in the email invitation of the survey. Respondents were

asked to respond to 20 questions relating to the skills instructed and retained following their

classroom experience. After reading each statement, participants were asked to indicate the

extent of agreement ranging from strongly agree to strongly disagree using a four-point Likert

scale. A copy of the survey is included in Appendix D.

The survey was organized into five general categories and one demographic indicator. The

categories of assessment are: Learning Environment, Relevance, Delivery, Overall and Energy for

Change.

Item Analysis

Question 1: The class environment helped me learn.

In response to the question of the learning environment’s impact on learning, 58 percent of

the participants agreed and 37 percent strongly agreed. Five percent of the population strongly

disagreed. Overall, 95 percent of respondents felt the class environment helped them learn. The

response mean was 3.06.

28

Table 4.1

Question #1 – The class environment helped me learn

# Answer Response %

1 Strongly Disagree 3 5%

2 Disagree 0 0%

3 Agree 33 58%

4 Strongly Agree 21 37%

Total 57 100%

A response of 95 percent in agreement indicates strong support that the learning

environment helped participants learn. The five percent of the population that indicated strong

disagreement leads the researcher to question more specifically what aspects of the learning

environment negatively impacted the ability to learn. . A qualitative response option was not

provided, so further analysis is not available.

Question 2: There were no major distractions that interfered with my learning.

In response to the question of whether or not there were distractions that interfered with

learning, 12 percent of respondents indicated they strongly disagreed. Eighty-eight of the

respondents were in agreement that no major distractions interfered with the learning. The

majority of respondents indicated no major distractions interfered with the learning. The response

mean was 3.20.

Table 4.2

Question #2 – There were no major distractions that interfered with my learning

# Answer Response %

29


2 Disagree 0 0%

3 Agree 29 51%


Total 57 100%

The majority of respondents (51 percent) maintained that no distractions interfered with

their learning. Eliminating or reducing the amount of distractions was a critical consideration in

the program design to improve participants’ abilities to focus on classroom learning. In the case

of the seven individuals choosing strongly disagree; the researcher can conclude that distractions

occurred, which negatively impacted the capacity to learn.

Question 3: The program material will be helpful for my success in the future.

In response to the question of material relevance, 98 percent agreed that the program

content would be helpful for success in the future. Twelve percent of the respondents strongly

agreed that the program material would be helpful to future success in improving employee

performance. One participant disagreed that the program material was helpful to their future

success. The response mean was 3.10.

Table 4.3

Question #3 – The program material will be helpful for my success in the future

# Answer Response %


2 Disagree 1 2%

3 Agree 49 86%


30

Total 57 100%

The researcher can conclude that coaching skills are valued as a leadership practice by a

majority of executives. A further conclusion is that the majority confirms that the learning will be

helpful in their future success. In the case of the one respondent in disagreement, a qualitative

response option was not available, so possible reasons for disagreement are left to speculation.

Question 4: I immediately applied my learning in my job.

In response to the question of application to on the job behaviors, 95 percent agreed they

could immediately apply the learning in their jobs. Forty-four percent strongly agreed that they

would be able to immediately apply this learning. Five percent of respondents disagreed that they

would be able to immediately apply the learning in their jobs. The mean response was 3.9.

Table 4.4

Question #4 – I immediately applied my learning in my job

# Answer Response %


2 Disagree 3 5%

3 Agree 29 51%


Total 57 100%

A majority of respondents, 95 percent, felt immediate application of classroom learning

was possible. Immediate application of learning is a critical consideration in the program design

because the evidence in support of the training indicates that managers were not providing

coaching for their employees. Respondents confirmed that the skills of manager coaching are

relevant to on-the-job experiences and expectations.

31

Question 5: I was engaged with what was going on during the program.

The question of engagement in the learning experience provided 65 percent agreement.

Thirty-five percent of respondents strongly agreed. One hundred percent of the participants

agreed that they were engaged in the classroom. The mean response was 3.70.

Table 4.5

Question # 5 – I was engaged with what was going on during the program

# Answer Response %


2 Disagree 0 0%

3 Agree 37 65%


Total 57 100%

All respondents agreed they were engaged in the classroom experience. This question

supports a conclusion that the design of the program facilitated engagement.

Question 6: The activities and exercises aided my learning.

In response to the question of whether or not the activities and exercises aided learning, 65

percent agreed. Thirty-two percent strongly agreed that the activities and exercises aided

learning. Three percent of respondents disagreed that the activities and exercises aided learning.

The mean response was 3.33.

Table 4.6

Question #6 – the activities and exercises aided my learning

# Answer Response %

32


2 Disagree 2 3%

3 Agree 37 65%


Total 57 100%

The majority of respondents, 97 percent, agreed that the activities and exercises aided in

their learning. A qualitative response option was not provided for this question. This would have

been beneficial to gain further insight on the three percent of respondents who disagreed.

However, overall, this question supports the finding that the activities and exercises supported

classroom learning.

Question 7: I was given adequate opportunity to demonstrate what I was learning.

In response to the question of opportunity to demonstrate learning, 97 percent agreed they

were provided adequate opportunity to demonstrate the learning. Of the 97 percent who

responded in agreement, 28 percent of respondents strongly agreed. Three percent of respondents

disagreed they had adequate opportunity to demonstrate what was learned. The mean response

was 3.24.

Table 4.7

Question #7 – I was given adequate opportunity to demonstrate what I was learning

# Answer Response %


2 Disagree 2 3%

3 Agree 39 69%

33


Total 57 100%

A majority, 97 percent, was in agreement that the classroom experience provided the

opportunity to demonstrate what was learned. This supports the effectiveness of the program

design in meeting company XYZ’s criteria to engage learners through the demonstration of skills.

Respondents were given the opportunity to provide a qualitative response to elaborate on their

disagreement, but did not provide qualitative information. Opportunities for further research will

be discussed in Chapter V.

Question 8: The program met my expectations.

In regard to meeting expectations, 62 percent agreed that the program met their

expectations and 26 percent strongly agreed the program met their expectations. However, 12

percent of respondents disagreed that the program met their expectations. An option for a

qualitative response was provided. The mean response was 3.14.

Table 4.8

Question #8 – The program met my expectations

# Answer Response %


2 Disagree 7 12%

3 Agree 35 62%


Total 57 100%

34

The overall program met the expectations of 88 percent of the participants. Seven

respondents disagreed that the program met their expectations. Respondents were given the option

to provide a qualitative response to elaborate on the reasons for their disagreement but chose not

to. An 88 percent agreement supports a conclusion that course objectives were aligned with

participant objectives.

Question 9: I am clear on how to apply what I learned on the job.

In response to the question of clarity in the application of learning to on-the-job behaviors,

97 percent agreed that they were clear on how to apply what they learned in the classroom to their

jobs. Thirty-seven percent of respondents strongly agreed that they were clear on how they would

apply learning to their job. Four percent of respondents disagreed on knowing how they would

apply what they learned. The mean response was 3.33.

Table 4.9

Question #9 – I am clear on how to apply what I learned on the job

# Answer Response %


2 Disagree 2 3%

3 Agree 34 60%


Total 57 100%

Ninety-seven percent of respondents are clear on how to transfer what they learned in the

classroom to their on-the-job behaviors. Clarity on application of learning should contribute to

transfer of learning in that participants can form linkages of skills and application. Linking skills

35

learned to on-the job-application provides the opportunity for transfer of learning to on-the-job

work habits.

Question 10: I would recommend this program to my co-workers.

Stakeholders of the program were interested in knowing if the program would be

recommended to peers. Company XYZ places a high value on internal recommendations of

training programs and views recommendations as a critical component to the ongoing success of a

program. The mean response was 3.31.

Table 4.10

Question #10 – I would recommend this program to my co-workers

# Answer Response %


2 Disagree 3 5%

3 Agree 33 58%


Total 57 100%

A majority, 95 percent of participants, would recommend this program to their co-

workers. Of the majority, 37 percent strongly agree that they will recommend the program to

their co-workers. Only five percent of respondents would not recommend this program.

Question 11: How confident are you that you will be able to apply what you have

learned back on the job?

Confidence in the application of skills was seen as an important variable to company XYZ’s

success. Company XYZ views confidence in the application of skills as crucial to supporting

36

learning retention. 100 percent of respondents are at least somewhat confident they will be able

to apply the learning’s on the job. The mean response was 1.80.

Table 4.11

Question #11 – How confident are you that you will be able to apply what you have learned back

on the job?

# Answer Response %

1 Extremely Confident 18 32%

2 Confident 32 56%

3 Somewhat Confident 7 12%

4 Not at all Confident 0 0%

Total 57 100%

Of the respondents, 32 percent are extremely confident they will be able to apply what

they learned back on the job. Fifty-six percent are confident and 12 percent are somewhat

confident. One hundred percent of respondents feel some confidence that they will be able to

apply the skills they learned in the classroom on the job.

Question 12: If you chose somewhat confident or not at all confident, please circle all

that apply. My confidence is not high because…

Participants that responded somewhat confident to Question 11 were asked to elaborate on their

response to indicate possible reasons for the lack of confidence. Seven respondents indicated they

were only somewhat confident. Lack of confidence was quantified as not having the support or

resources, having other higher priorities, or not having a clear picture of what is expected. One

respondent indicated that they lacked the necessary knowledge and skills.

37

Table 4.12

Question #12 – If you chose somewhat confident or not at all confident, please circle all that

apply. My confidence is not high because….

# Answer Response %

1 I do not have the human support to do it 1 14%

2 I do not have the necessary resources to do it

2 29%

3 I have other higher priorities 2 29%

4 I do not have a clear picture of what is expected of me

1 14%

5 I do not have the necessary knowledge and skills

1 14%

Total 7 100%

Of respondents, 12 percent were somewhat confident they would be able to apply the

skills on the job. Of those 12 percent, all were able to identify contributing factors for their lack

of confidence. Factors identified that would impact their confidence, such as amount of support,

resources, priority, expectations and knowledge and skills could be considered for identification

of sustainability opportunities.

Question 13: How committed are you to applying what you learned in your work?

In the question regarding commitment to application of what was learned to the job, 100

percent were at least somewhat committed. Of respondents, 32 percent are extremely committed

to applying what they learned in their work. Sixty-five percent are committed and 3 percent are

somewhat committed to application of learning to work. The mean response was 1.72.

38

Table 4.13

Question #13 – How committed are you to applying what you learned in your work?

# Answer Response %

1 Extremely Committed 18 32%

2 Confident 37 65%

3 Somewhat Committed 2 3%

4 Not at all Committed 0 0%

Total 57 100%

A majority, 97 percent, are firmly committed to apply the learning, which suggests skill

development may continue throughout the job application. Supporting the commitment of

application should be considered for sustainability efforts.

Question 14: If you chose somewhat committed or not at all committed, please circle

all that apply. My commitment is not high because…

For the two respondents choosing somewhat committed in Question 13, both supported

their belief with a qualitative response. Reasons for low commitment were lack of recognition,

lack of resources and other higher priorities. One respondent chose multiple reasons.

Table 4.14

Question #14 – If you chose somewhat committed or not at all committed to Question 13, please

circle all that apply. My commitment is not high because…

# Answer Response %

1 I do not get rewarded or recognized for coaching

1 33%

2 I do not have the necessary resources to 1 33%

39

do it

3 I have other higher priorities 1 33%

4 I do not have a clear picture of what is expected of me

0 0%

4 I do not have the necessary knowledge and skills

0 0%

Total 3 100%

In this study, commitment to application was not a concern among respondents. Of the

two participants indicating a slight concern, three reasons were cited for selecting somewhat

committed to the application of learning to their jobs. Recognition, resource support and

prioritization could be possible focus areas for sustainability efforts. The survey response rate of

26 percent limits the conclusions that can be drawn to provide concrete components of a

sustainability effort.

Qualitative Analysis

The category of Energy for Change identified five questions (Questions 15-19) that

provided respondents with the option for qualitative responses. The questions did not require a

response to advance in the survey. The majority of respondents chose not to provide a qualitative

response. A thematic analysis is provided for all qualitative responses.

Question 15: What barriers do you anticipate that might prevent you from applying

what you learned?

Question 15 asked respondents to identify the barriers that will prevent them from

applying what they learned. Nine barriers were identified. Twenty-five participants left the

question blank. Time, capacity and coaching maturity were the most frequently-cited responses

among the nine barriers.

40

Table 4.15

Question # 15 – What barriers do you anticipate that might prevent you from applying what you

learned?

Blank 25

Capacity 1

Coaching maturity 8

Competing business priorities 2

Do it consciously 1

Lack of reinforcement 1

Time 6

Time & capacity 9

Time, capacity & maturity of coaches 2

Time & coaching maturity 2

Grand total 57

Of the total respondents, 25 did not respond, representing 44 percent of the total

participants. The researcher can speculate on possible reasons for a blank response as:

respondents did not believe barriers exist, experienced survey fatigue or other conflicting

priorities. Time and capacity were the key barriers identified that will impact the respondents’

abilities to apply their learning on the job.

Question 16: What might help you overcome those barriers?

Question 16 asked respondents to identify what would help them the overcome barriers

noted in Question 15. Six responses were identified as possible solutions to overcome the

barriers. A total of 22 respondents chose not to respond.

41

Table 4.16

Question # 16 – What might help you overcome those barriers?

Blank 22

Practice 12

Prioritize 15

Prioritize & practice 5

Support 1

Telepresence 1

Time & credibility 1

Grand total 57

Key themes of the need to practice and prioritize emerged in the responses to support

prioritization of the coaching activity and practicing of skills learned.

Question 17: What outcomes are you hoping to achieve as a result of your coaching

efforts?

On the question of outcomes respondents were hoping to achieve as a result of their

coaching efforts, 52 responded qualitatively. Empowered and engaged employees, noted by 23

respondents, emerged as the most frequently-cited responses. Relationship development with

teams and coaching maturity also surfaced as common themes.

Table 4.17

Question # 17 – What outcomes are you hoping to achieve as a result of your coaching efforts?

Better teamwork 1

Blank 5

42

Coaching maturity 8

Coaching maturity & empowered & engaged employees 5

Continual learning & leadership team development 1

Empowered & engaged employees 23

Empowered & engaged employees & engaged leadership 1

Improved morale 1

More insight about people, culture and businesses, so I can contribute more from my function and leadership position. I find coaching is more fruitful than just expressing yourself and your conversation is deeper than artificial one. Coaching can also help

1

Relationship development with team 3

Relationship development with team & empowered & engaged employees 6

Relationship development with team & stronger pipeline 1

Stronger pipelines 1

Grand total 57

The majority of respondents identified positive outcomes to their coaching efforts. The

most frequent responses were empowered, engaged employees and building relationships with

their teams. Coaching maturity was also noted by 13 of the participants.

Question 18: What support do you need?

Question 18 asked what support respondents would need. The majority, 30 respondents,

left the question blank. Refresher training was the primary request for the majority of those that

responded. Improving systems and processing, reinforcement and reminders, senior leaders as

coaches, space to learn and stronger leadership were also identified as support opportunities.

43

Table 4.18

Question # 18 – What support do you need?

Better systems & processes 1

Blank 30

Practice 5

Refresher training 12

Reinforcement 1

Reminders 3

Senior leadership as coaches 1

Space to learn 1

Stronger leadership 3

Grand total 57

Although a majority of participants left the question of necessary support blank, several

had suggestions on how to provide the adequate support to enhance learning. The highest

demand for support lies in refresher training, an element that proves important in coaching.

Question 19: What evidence have you observed that indicates coaching is having a

positive impact on business results?

On the question of observed evidence that coaching is having a positive impact on

business results, 17 respondents did not respond. A total of 14 identified improved employee

performance while 7 participants identified improved coaching skills. Ten respondents indicated

it was too early to identify a business impact. Other impacts noted were alignment to strategy and

reduced frustration in the organization, performance management process, company opinion

survey results, improved morale, increased accountability and improved listening skills.

44

Table 4.19

Question # 19 – What evidence have you observed that indicates coaching is having positive impact on business results?

Alignment to strategy & reduced frustration in organization 1

Blank 17

Performance management process 1

Company opinion survey results 4

Improved coaching skills 7

Improved employee performance 14

Improved morale 1

Increased accountability 1

Leaders are developing "listening skill" which encourages employees to express their opinions, introduce new ways of thinking, not afraid to challenge in conversations, etc. that is leading a positive cultural change and creation of innovations

1

Too new, nothing yet 10

Grand total 57

Improved employee performance is a key theme identified by participants to provide

evidence of a positive impact of the coaching skills on business results. A total of 10 respondents

indicated coaching activities were too new to identify a result. Of the total respondents, 17 chose

not to respond.

Summary

This survey instrument was a standard survey design adapted from the Kirkpatrick

website. Several questions were similar in nature and may be directly related to the blank

responses. The perceived duplication of questions may also have contributed to survey fatigue

and prompting respondents to move quickly though the survey. The option to advance to the next

45

question without providing a qualitative response may have negatively impacted the results and

could also be considered a design flaw. Given the targeted population, executive band leaders,

the researcher concludes that a more targeted approach, such as focus groups or personal

interviews, may have provided a higher response rate as well as more descriptive and prescriptive

responses from this population. Discussion of the results and recommendations for future

research will be discussed in chapter V.

46

Chapter V: Summary, Conclusions and Recommendations

The purpose of this study was to evaluate the learning experience and the impact of

learning on coaching applications as well as identify potential areas of focus for sustainability.

Company XYZ made a significant investment in resources and capital for a skill-based learning

experience for global managers. The aim of the investment was to improve employee

performance through the application of coaching. Coaching has been determined to help the

person or group become self-directed in their own learning and development (Bennett & Bush,

2009). The research suggests that the best coaches are those who are able to adapt to the ever-

changing environment and its pupils (Locke, 2008). When a coach is able to become a change

agent, their coachees will in turn be able to adapt to their surroundings for their learning and

development. Company XYZ was interested in capitalizing on the business results coaching

could provide. This study was conducted to determine what it is that the participants require in a

coaching environment, and how best to help those participants transfer what they learned into the

career world. Chapter V will summarize the study, the methodology and the results and present

conclusions and recommendations for future actions.

Summary of the Study

The literature review for this study consists of four interrelated areas that are foundational

to the research and deployment of a global manager coaching program. The four areas are: Key

authors, contributors and models used in business coaching, Standards & competencies for

business coaching, Assessments used in business coaching and Training course evaluation. These

areas of concentration were chosen to inform and educate in the field of coaching and to identify

potential areas of focus for the sustainability of coaching skills.

47

A survey was created and administered to evaluate the learning experience. The survey

methodology was a four-step process consisting of the following steps: subject selection,

instrument selection, data collection procedures and data analysis. Company XYZ’s in-house

survey capability was used to administer the survey. The survey sample was executive band

leaders who completed the training and voluntarily participated in the evaluation. The survey

design was a template from the Kirkpatrick Partners™ which was modified slightly to focus on

the assessment of the participants’ learning experiences and capacity for retention of the skills that

were taught. The results of the survey will be used to develop plans for sustainability of coaching

skills taught in the classroom.

Limitations

While the survey provided further insights into the effectiveness of a global manager coaching

program in meeting business goals, there are several limitations the researcher must take into

consideration to evaluate and interpret the findings. The following are the list of limitations:

1. The sample is limited to executive band leaders at Company XYZ.





4. The study has excluded the population below executive band, which may be considered for

further study.

The limitations remained consistent throughout the study. The anticipated limitations held

through and no new limitations were identified. For example, the sample is limited to executive

band leaders at Company XYZ. Further research would need to be conducted in order to

48

determine the results of coaching on other samples of employees. Second, participation is

voluntary and leaders may elect to disregard the survey, impacting the amount of responses and

quality of the overall findings. This generally allows the responses that do come in to be accurate

and honest, but responses may also be limited. Creating a mandatory participation would increase

the number of leader responses, but may also affect the integrity of the answers and significantly

impact the overall reactions. Furthermore, findings from this survey at Company XYZ are

predicted to change over time with the addition of sustainability efforts, which will vary by

region. Just as coaches must become change agents, so must the surveys and findings, since the

coachees are always changing as well. Lastly, the study has excluded the population below the

executive band, which may be considered for further study. Similar to the first limitation, this

needs to be taken into account, realizing that the results may not apply to every employee of

Company XYZ.

Factors Impacting Survey Effectiveness

Several factors could be considered as compromising factors to the overall effectiveness

of the survey. The following list represents the researcher’s analysis of possible mitigating

factors.

1) The general survey approach tends to be less effective for executive band audience.

The decision to create an email and survey response for this group, while expeditious,

proved ineffective given the lack of responses in total and depth of response.

Although approximately 358 band leaders completed the course in the past 18 months

and were asked to complete the survey, only 57 band leaders participated.

Additionally, in several incidences, many of those 57 respondents did not choose to

49

respond or elaborate on their responses, subsequently compromising the depth of data

the researcher was seeking.

2) Executive band leaders experience a very demanding work schedule. It is possible

that lack of response rate was compromised by the schedule prioritization for

executive bandleaders. When responses are not mandatory, these leaders have copious

other tasks they need to complete before they can use their time on a voluntary survey.

3) Company XYZ utilizes surveys for a variety of organizational needs. It is possible

that the executive band leaders chosen for the survey were experiencing survey

fatigue, which would be evident in the lack of responses and the quality of responses.

Furthermore, several of the qualitative responses that were left unanswered could have

been caused by survey fatigue. When the survey did not require a response to proceed,

band leaders most likely wanted to get through the survey as quickly as possible,

sacrificing responses that could have been useful.

4) In many cases, executive band leaders use their administrative assistants to manage

their email accounts. If an administrator deemed this as less important, it is possible

the executive band leader never even saw the request for survey.

To mitigate these data collection challenges for future survey needs of this population, the

researcher recommends a focus group approach or individual one-on-one interviews. These

tactics would likely provide a richer data set. Additionally, a senior level manager request for

response will likely improve response rate and quality of responses.

50

Conclusions of Item Analysis

The item analysis, representing each survey question and its results individually, provides

data to support the following conclusions and recommendations for the teaching’s future

sustainability. These conclusions and recommendations follow the participants’ responses

1) Providing opportunities to practice skills will build confidence for managers

applying coaching in their manager activities. Although all leaders indicated

that they were at least somewhat confident that they will be able to apply what

they have learned back on the job, most agree that the extra practice will support

their confidence.

2) Providing managers support and resources will impact skill mastery. Not

having the human support or the necessary resources to apply what the leaders

have learned back on the job are key factors that influence their confidence in

the application of learning. Therefore, providing the support and resources aids

in confidence and overall mastery.

3) Establishing coaching as a business priority with clear expectations for coaching

activity will impact the frequency of coaching conversations. Several

participants indicated barriers that may prevent them from applying what they

learned on the job. Included in these barriers are lack of time and competing

business priorities. If coaching were more of an emphasized priority, the time

and commitment would be made to engage in more coaching conversations.

4) Building on knowledge and skills of coaching will impact skill mastery for

managers. A few responses indicated that participants do not think they have

the necessary knowledge and skills to apply what they learned back on the job.

51

Providing the extra training and practice, reminders and refreshers of the

training will allow the managers to build on their knowledge and skills of

coaching, so they do feel that they have enough of a grasp on the skills to apply

it to their jobs.

5) Supporting the commitment of application of coaching skills through reward

and recognition will impact the frequency of coaching skills used by managers.

The leaders who answered that they only felt somewhat committed to applying

what they learned in their work responded that they are not committed since

they do not get rewarded or recognized for coaching. Again, these executive

band managers have extremely busy schedules, and when coaching is not

something that is recognized, they may participate in other behaviors that do get

rewarded. Therefore, if the commitment of application of coaching skills is

rewarded and recognized, more managers will likely participate in those skills.

6) Linking the skills leaders learned to on-the job-application will increase learning

retention. Numerous managers identified evidence of positive impacts of

coaching skills on business results. Examples include improved employee

performance, improved morale and improved accountability. Incorporating

specific incidences of these examples will allow the leaders to be able to

understand the coaching skills better, and therefore better be able to replicate

them later in their careers.

7) Co-worker recommendation supports positive mindsets at the start of the

learning experience. It is important to maintain positive attitudes towards

coaching in order to sustain commitment and confidence.

52

Conclusions and Recommendations

The first action in support of training effectiveness is a solid partnership between training

departments and managers to maximize the investment and link the training to meeting business

goals. According to the CEB, learning programs with the highest application rates improved

learner performance by 51 percent and business results by 28 percent (Corporate Executive Board

Company, 2012). A component of this training did indeed promote application but the researcher

feels more accountability would promote greater application. Manager interactions with

employees are also a strong contributor to the overall effectiveness of a training program.

According to Brinkerhoff & Gill (1994), training effectiveness is impacted by manager

involvement in 4 key areas:

1) Influencing the attitude of the trainee coming to the training event,

2) Approving the time and money spent for the training,

3) Providing or not providing time to practice the new skills back on the job,

4) Providing or not providing reinforcement of the knowledge and skills (pg. 19).

In support of this, the researcher recommends an expanded expectation of managers of all

employees participating in the course. Building on Brinkerhoff, a recommendation of a post-

course debrief to identify immediate opportunities for reinforcement and discuss application

opportunities will promote continued application of skills (Brinkerhoff and Gill, 1994).

Additionally, the researcher recommends a series of follow-up coaching sessions where the

manager can demonstrate the skills taught and model for the employee a coaching relationship.

Leaders teaching leaders creates success for leaders and the organization and provides additional

practice so the leaders can increase confidence.

53

In addition to the organizational and managerial support, the researcher recommends

supporting the commitment of the application of coaching skills through reward and recognition.

If managers know they are measured and will be rewarded for applying coaching skills, the

organization will undoubtedly see more committed coaching activity. Reward and recognition

may also impact the barrier sited as not having the time or capacity to coach.

Several respondents indicated coaching maturity was impacting their capacity to be

successful in their coaching relationships. To accelerate maturity, the researcher recommends a

refresher or skill-reinforcement training. Providing opportunities to practice and get feedback

will build confidence and demonstrate coaching maturity.

54

References

Bennett, J., & Bush, M. W. (2009). Coaching in organizations: Current trends & future

opportunities. OD Practitioner, 41(1), 2-7.

Berard, B. (2005). Coaching and mentoring: Enhancing education. Chief Learning Officer, 4(11),

31-35.

Brinkerhoff, R. O., & Dressler, D. E. (2002, July). Using evaluation to build organizational

performance and learning capability: A strategy and a method. Performance Improvement,

41(6), 14-21.

Brinkerhoff, R.O., & Gill, S. J. (1994). The learning alliance. San Francisco, CA: Jossey-Bass,

Inc.

Corporate Executive Board Company. (2012). Measuring for impact: A practical guide to

improving learning measurement. Retrieved from: www.ldr.executiveboard.com

Corporate Leadership Council. (2002) Reframing the measurement debate: Moving beyond

program analysis in the learning function. Washington DC: Corporate Executive Board.

Coutu, D., & Kauffman, C. (2009, January). What can coaches do for you? Harvard Business

Review. HBR Research Report, 91-97.

Derven, M. (2012, November). Building a strategic approach to learning evaluation. T & D, 54-

57.

Hauser, L. (2009). Evidence-based coaching: A case study. OD Practitioner, 41(1), 8-13.

Hertzberg, F. (1987). One more time: How do you motivate employees? Harvard Business

Review, 65(5), 109-120.

Heslin, P. A., Vandewalle, D., & Latham, G. P. (2006). Keen to help? Managers' implicit person

theories and their subsequent employee coaching. Personnel Psychology, 59, 871-902.

55

Hudson, F. (1999). The handbook of coaching: A comprehensive resource guide for managers,

executives, consultants and human resource professionals (1st ed.). San Francisco: Jossey-

Bass.

International Coach Federation. (2012). Global coaching study. Lexington, KY:

PricewaterhouseCoopers LLP. Retrieved from ICF: http://www.coachfederation.org/.

International Coach Federation. (2009). Global coaching client study. Lexington, Kentucky:

Association Resource Centre, Inc. and PricewaterhouseCoopers LLP. Retrieved from:

www.coachfederation.org/

Kirkpatrick, D., & Kirkpatrick, J. C. (2006). Evaluating training programs: The four levels (3rd

ed.). San Francisco: Barrett-Koehler.

Kirkpatrick, J. (2007). The hidden power of Kirkpatrick’s four levels. T & D, 61(8), 34-37.

Leonard-Cross, E. (2010). Developmental coaching: Business benefit – Fact or fad: An evaluative

study to explore the impact of coaching in the workplace. International Coaching

Psychology Review, 5(1), 36-47.

Levenson, A. (2009). Measuring and maximizing the business impact of executive coaching.

Consulting Psychology Journal: Practice and Research, 61(2), 103-121. doi:

11010.1037/a0015438

Locke, A. (2008). Developmental coaching: Bridge to organizational success. Creative Nursing,

14(3), 103.

McLean, P. (2012). The completely revised handbook of coaching: A comprehensive resource

guide for managers, executives, consultants and human resource professionals (1st ed.).

San Francisco: Jossey-Bass, Inc.

http://www.coachfederation.org/

56

Miller, L. (2012). ASTD state of industry report: Organizations continue to invest in workplace

learning. T & D, 42-48.

Pace, A. (2011, October). A new era of performance management. T & D, 12.

Rogers, J. (2004). Coaching skills: A handbook. New York: Open University Press.

Sherpa Coaching (2013). The executive coaching survey. Cincinnati, OH: Sherpa Executive

Coaching.

Van Velsor, E., McCauley, C. D., & Ruderman, M. N. (2010). The Center for Creative

Leadership: Handbook of leadership development (3rd ed.). San Francisco, CA: Jossey-

Bass, Inc.

Wenson, J. E. (2010). After-coaching leadership skills and their impact on direct reports:

Recommendations for organizations. Human Resource Development International, 13(5),

607-616.

Whitmore, J. (2002). Coaching for performance: Growing people, performance and purpose (3rd

ed.). Boston: Nicholas Brealey Publishing.

Zeus, P., & Skiffington, S. (2001). The complete guide to coaching at work. New York: McGraw-

Hill.

Zeus, P., & Skiffington, S. (2003). Behavioral coaching. North Ryde, Australia: McGraw-Hill.

57

Appendix A: Permission for Study – Company XYZ

From: Martin, Sue (XYZ Company)

Subject: RE: Coaching Retention Survey Date: January 28, 2013 4:34:13 PM CST To: Rogahn, MaryEllen (XYZ Company)

Hi MaryEllen- per our discussion, I’d support targeting the EB audience who completed training for this survey. With just a few edits to the questions, the data will be very helpful in determining the overall effectiveness of our coaching training program.

Thank you.

Sue

From: Rogahn, MaryEllen (Company XYZ) Sent: Sunday, January 27, 2013 3:27 PM To: Martin, Sue (Company XYZ)

Subject: Coaching Retention Survey

Hi Sue:

As I mentioned, to complete my master’s thesis I have proposed the measurement of skill retention from our coaching conversations course. I’ve attached a survey I created in collaboration with the Kirkpatrick Partners which constitutes validated survey. My first thought was to measure this for the HR population but another thought would be to survey the EB population that has completed the training which might be useful as we map out sustainability approaches.

I have not added demographic questions as my first thought was just HR but if you think it would be useful to survey EB’s I think it would be good to ask a question for regional location.

Survey Access Link: http://supportcentral.ge.com/survey/sup_surveycenter_user_default.asp?prod_id=17778&doc_id=3746836

Best Regards,

Mary Ellen

Mary Ellen Rogahn

262-544-3927 Direct ~ 262.510.5152 cell

Learning Design & Technology CoE

http://supportcentral.ge.com/survey/sup_surveycenter_user_default.asp?prod_id=17778&doc_id=3746836


58

Appendix B: UW-Stout Implied Consent Statement for Research Involving Human Subjects

Consent to Participate In UW-Stout Approved Research

Title: Evaluation of a global manager coaching training program

Research Sponsor:

Dr. Jeanette Kersten, EdD

Program Director – MS Training & HRD

Chair-People, Process, Culture

College of Management

248 Jarvis Tech Wing

University of Wisconsin, Stout

Menomonie WI 54751-0790

715-232-5229

Investigator:

Mary Ellen Rogahn

262-510-5152

478 Candlewick Court

Delafield, WI 53018

Description:

The aim of this study is to evaluate the effectiveness the global manager coaching program. Specifically, the target of the study is behavioral change and to assess the impact of the global manager coaching program.

Risks and Benefits:

Survey results will be used to identify opportunities we can explore to support continued development in the skills learned.

Special Populations:

NA

Time Commitment and Payment:

Survey contains 18 questions and it is estimated to take approximately 15 minutes to complete. Payment= NA

Confidentiality:

Individual identity will not be included on any analysis documents. The aim of this survey is to assess the impact of the training and opportunities for reinforcement of the coaching program.

Right to Withdraw:

59

Your participation in this study is entirely voluntary. You may choose not to participate without any adverse consequences to you. You have the right to stop the survey at any time. However, should you choose to participate and later wish to withdraw from the study, there is no way to identify your anonymous document after it has been turned into the investigator. If you are participating in an anonymous online survey, once you submit your response, the data cannot be linked to you and cannot be withdrawn.

IRB Approval:

This study has been reviewed and approved by The University of Wisconsin-Stout's Institutional Review Board (IRB). The IRB has determined that this study meets the ethical obligations required by federal law and University policies. If you have questions or concerns regarding this study please contact the Investigator or Advisor. If you have any questions, concerns, or reports regarding your rights as a research subject, please contact the IRB Administrator.

Investigator: Mary Ellen Rogahn

262-510-5152

IRB Administrator

Sue Foxwell, Research Services

152 Vocational Rehabilitation Bldg.

UW-Stout

Menomonie, WI 54751

715.232.2477

[email protected]

Advisor: Dr. Jeanette Kersten EdD

715-232-5229

[email protected]

Statement of Consent:

“By completing the following survey you agree to participate in the project entitled, Global Manager Training Program.

mailto:[email protected]

60

Appendix C: Permission to Use Survey Tool

Dear Mary Ellen,

So sorry for the delay. This got overlooked.

It is no problem for you to use the form within your work, properly cited as you suggest.

We also love to see any research on training evaluation that you are allowed to share.

Best regards,

Wendy

Wendy Kayser Kirkpatrick

President - Kirkpatrick Partners

PLEASE NOTE OUR NEW MAILING ADDRESS AND PHONE:

68 West Shore Drive, Newnan, GA 30263

Direct line: (770) 683-2846 | Customer service: (443) 856-4500

Thank you for supporting The One and Only Kirkpatrick company!

From: Mary Ellen Rogahn [mailto:[email protected]] Sent: Monday, December 17, 2012 4:27 PM To: [email protected] Subject: SPAM-LOW: Use of evaluation form

To whom to may concern:

I am a graduate student at University of Wisconsin - Stout seeking permission to use your " Hybrid course evaluation form" in conjunction with a research paper I am completing for the graduate requirements of University of Wisconsin-Stout. I am researching the effectiveness of a global manager coaching program and would like permission to modify the evaluation to address specific course learning targets. Kirkpatrick evaluation model will be cited and reference throughout. Please indicate what is needed to secure this request.

Mary Ellen Rogahn

[email protected]



61

Appendix D: Invitation to Survey Participants & Implied Consent

Dear Executive Band Leader:

Continuing our journey to build a coaching culture we’d like your feedback on your application of the coaching skills taught in the Coaching Conversations program.

The evaluation survey should take no more than 15 minutes. I welcome follow up if you have further comment.

This study is part of academic research, the outcome will contribute to a master’s thesis I am writing evaluating the effectiveness of our global manager coaching program. I am confident the results of this study will strengthen our existing curriculum and provide ideas for long-term sustainability for our coaching skills.

Since the survey is part of academic research I have attached the implied consent document for your review. Completion of the survey will constitute your consent; no further action is required on your part.

Click here to activate the survey, Survey Access Link: http://supportcentral.ge.com/survey/sup_surveycenter_user_default.asp?prod_id=17778&doc_id=3746836

I appreciate your thoughtful responses. I will share analysis once complete and work with your teams to support the continued use of coaching skills.

BR,

Mary Ellen

Mary Ellen Rogahn

262-544-3927 Direct ~ 262.510.5152 cell

Learning Design & Technology CoE

XYZ Corporate



62

Appendix E: Coaching Conversations Skill Retention Survey

To measure retention of skills learned in the global manager-coaching program

Learning Environment:

Q1. The class environment helped me to learn.

Strongly Disagree Disagree Agree Strongly Agree

Explain/Comment: (maximum 1000 characters)

Q2. There we no major distractions that interfered with my learning.



Relevance:

Q3. The program material will be helpful for my success in the future.



Q4. I immediately applied my learning in my job.



Delivery:

Q5. I was engaged with what was going on during the program.



63

Q6. The activities and exercises aided in my learning.



Q7. I was given adequate opportunity to demonstrate what I was learning.



Overall:

Q8. The program met my expectations.



Q9. I am clear on how to apply what I learned on the job.



Q10. I would recommend this program to my co-workers.



Energy for Change:

Q11. How confident are you that you will be able to apply what you have learned on the job?

Not at all Confident Somewhat Confident Confident Extremely Confident


64

Q12. If you chose somewhat confident or not at all confident, please explain your response.

Q13. How committed are you to applying what you learned in your work?

Not at all Committed Somewhat Committed Extremely Committed

Q14. If you chose somewhat committed or not at all committed, please comment on the reasons why.

Q15. What barriers do you anticipate that might prevent you from applying what you learned?


Q16. What might help you overcome those barriers?


Q17. What outcomes are you hoping to achieve as a result of your coaching efforts?


Q18. What support do you need?


Q19. What evidence have you observed that indicates coaching is having a positive impact on business results?


65

Demographics:

Q20. Please indicate the region you currently represent

o Europe o North America & Canada o Latin America o EAGM o APAC o India o China o Other


1 author: rogahn, mary ellen a manager coaching training

Documents