teacher education accreditation council - epp.byu.eduepp.byu.edu/standards-pdfs/2013/3-teac audit...

Teacher Education Accreditation Council (TEAC)

©TEAC ♦ One Dupont Circle ♦ Suite 320 ♦ Washington, DC ♦ 20036 ♦ 202/466-7236 ♦www.teac.org 1

Teacher Education Accreditation Council

CONFIDENTIAL DOCUMENT

Do NOT Quote or Cite Without Permission

TEAC Audit Report Based on the Inquiry Brief Proposal of the

Brigham Young University School Leadership Program

October 2-4, 2012

First draft of audit report sent to program faculty on: March 15, 2013 Final audit report accepted by program faculty on: April 8, 2013

Audit team Members: Jon F. Wergin, Professor of Educational Studies, Antioch University, Yellow Springs,

OH Jerusalem D. Rivera-Wilson, Senior Faculty Associate and Director of Clinical

Training and Field Experiences, SUNY University at Albany, Albany, NY Garrick Peterson, Principal, Lakeridge Junior High, Alpine School District, Provo, UT

(local practitioner) Sydnee Dickson, Teaching and Learning Director, Utah State Office of Education, Salt

Lake City, UT (state representative) Brief authors: Julie M. Hite E. Vance Randall Shannon K. Dulaney



I. Introduction

Summary of the Case Brigham Young University

School Leadership Program

Preparing teachers and educational leaders has been part of Brigham Young University’s mission since its founding in 1876. The Department of Educational Leadership & Foundations (EDLF) is one of five departments in the University’s McKay School of Education. Other departments include Communication Disorders, Counseling Psychology and Special Education, Instructional Psychology and Technology, Teacher Education, and the Center for Improvement of Teacher Education and Schooling. The EDLF Department offers both an MEd and an EdD in Educational Leadership. Offered through the MEd degree, the School Leadership Program has two tracks: one is for full-time students, the Leadership Preparation Program (LPP), and one is for part-time students, the Executive School Leadership Program (ExSL). These two tracks were brought together in 2010 under a single BYU School Leadership Program, and under a single committee structure. Both tracks have the same learning outcomes, curriculum (38-40 credits of coursework), administrative internship requirements (3 credits), and student assessments.

The School Leadership Program has three program goals, each aligned with a TEAC quality principle: to “prepare effective school leaders” (QP 1), to “pursue dynamic stability and responsiveness to stakeholders” (QP 2), and to “provide a graduate program of excellence” (QP 3).

The Program admits 20-30 students per year across both full- and part-time tracks. In 2011, 9 students were admitted to LPP, 14 to ExSL. About half of these 23 students are male and all are Caucasian.

The Program has four primary “student learning outcomes” (claims), all aligned with TEAC Quality Principles and ISLLC standards, plus three additional claims addressing TEAC’s cross-cutting themes. The Program claims that its “graduates are prepared with the necessary knowledge and skills to:

1. “Lead with Professional Knowledge: Understand and apply the disciplinary knowledge in the graduate program coursework to the practice of school leadership.

2. “Lead Learning Communities: Engage in, facilitate and lead collaborative learning communities to improve teaching and learning of all students.

3. “Lead Strategic Decision Making & Systemic Change: Engage in, facilitate and lead strategic inquiry, data-driven decision making and systemic change, for school and student improvement.

4. “Lead with Effective & Caring Leadership: Effectively lead with integrity and act on their knowledge in a caring and professional manner with all school stakeholders.”



And in addition, “Graduates understand and are prepared to 5. “Use today’s technology to enhance the educational process.

6. “Be sensitive to the needs of a diverse population, and endorse and encourage the success of all students.

7. “Think critically, conduct research, self assess for professional development, and work collaboratively.”

Some of the evidence to support these claims is current and some is under development. Current evidence, now available only for Claim 1, includes course grades and GPA, student evaluations, Praxis scores, recommendations and placement information. Planned evidence is much more extensive, and includes a Student Learning Assessment, Tracking & Evaluation system (SLATE); internship reports; student and alumni surveys; an Internship Leadership Project (ILP); Praxis exam subscores; a comprehensive examination; and administrative placement reports. The SLATE report will link targeted course- and internship-level assessments to specific student-learning outcomes and program claims. A specific student survey will be the TELSA Self-Assessment of school leadership skills, to be done both pre-and post-program. Pilot data have been collected on most of these new assessments and modifications made accordingly. The program has also developed a “plan for continuous improvement,” based on what it has learned from preparing this Inquiry Brief Proposal (including the internal audit and Appendices B and C); this includes a Strengths, Weaknesses, Opportunities and Threats (SWOT) analysis and a five-year improvement plan centered on each of TEAC’s Quality Principles. Acceptance of the Summary of the Case The faculty accepted the Summary of the Case as accurate on September 27, 2012. Audit logistics Auditors examined documents primarily in Room 327 of the McKay Building on campus, and also held meetings in 307 McKay, 319 McKay, and the TEC lab. Auditors formally visited a class in 319 McKay and another at a local high school. Audit opinion Overall the Proposal earned a clean audit opinion, and each component of the TEAC system received a clean opinion. The auditors also concluded that the evidence supports the view that Brigham Young University is committed to the School Leadership Program.



II. Audit Map Audit tasks are organized by TEAC elements & components and are noted as Verified, Verified with Error, Not Verified, or Disclaimer. Audit Task numbers are hyperlinked to the audit tasks in the accompanying report.

TEAC Component Verified Verified

with Error

Not Verified Disclaimer

2.1 Rationale for assessments

A1, A2, A3, A4, B2, B3, B4, B6, B7

B1, B5

2.2 Decisions based on evidence B9 B8

2.3 Quality control system

B11, B12, B14, B15, B16 B10 B13



III. Method of the Audit The TEAC staff and the auditors selected a number of targets from the Proposal and created specific tasks designed to verify the targets. (A target is any aspect of the Proposal, such as text, data, or a figure, which is related to any of TEAC’s principles and standards.) In addition, the auditors created follow-up audit tasks based on their on-site experiences. With regard to any one component of the TEAC system, the auditors employed a range of tasks. Most tasks were straightforward probes designed to verify or confirm the target (e.g. recalculating figures, interviewing informants, examining catalogs, policy manuals). Some tasks sought to reconcile multiple representations of the same target in the Proposal for internal consistency (e.g., the figures in two tables on the same point, restatements of the target in other places of the Proposal). A few tasks sought to corroborate the target by examination of evidence not cited in the Proposal, but relevant to assertions in the Proposal. The auditors may corroborate the evidence in the Proposal by new or extended statistical analyses of the evidence cited in the Proposal and related evidence outside the Proposal (e.g., on-site and on-line surveys of key informants). The auditors also, whenever it is possible and feasible, examined the primary source for any target (e.g., the actual rating or survey forms, formal documents, student portfolios, artifacts, roll & grade books, classroom facilities, budgets, correspondence).

IV. Audit Findings The audit findings consisted of a series of audit tasks, which included a target from the Proposal and a probe about that target. The audit tasks were associated with specific components of the TEAC system, which are denoted in parentheses following the task number. Note: the program submitted an Addendum to the Proposal in response to auditors’ recommendations, and several Audit Tasks refer to this document.

A. Tasks Related to Quality Principle I: Evidence of Candidate Learning In an Inquiry Brief Proposal, the program is not required to directly address Quality Principle I.



B. Tasks Related to Quality Principle II: Evidence of a Quality Control System This section of the audit report addresses targets that are associated with Quality Principle II, which has the following requirements: 2.1 Rationale for the assessments. There must be a rationale for the program’s assessment

methods that explains why the faculty thinks the assessments are valid and why the criteria and standards the faculty have selected as indicating success are appropriate.

2.2 Program decisions and planning based on evidence. Where appropriate, the program must base decisions to modify its assessment systems, pedagogical approaches, and curriculum and program requirements on evidence of student learning.

2.3 Influential quality control system. The program must provide evidence, based on an internal audit conducted by the program faculty, that the quality control system functions as it was designed, that it promotes the program’s continual improvement, and that it yields outcomes specified in TEAC subcomponents 2.3.1 Curriculum, 2.3.2 Faculty, 2.3.3 Candidates, and 2.3.4 Resources.

The following audit tasks are from the TEAC online survey of the program’s candidates, faculty and cooperating mentors. Prior to the audit TEAC requests email addresses for program faculty, program candidates, and mentors who work with the program. Table A below indicates how many in each category were invited to take the email survey, how many of the emails were successfully delivered, how many opened the email, and how many and what percentage responded to the email survey:

Table A On-Line Survey Responses

Invited Email Delivered

Email Opened

Responses Completed

Percentage Responding

Program Faculty 15 15 11 9 60.00%

Program Candidates 41 41 28 17 41.50%

Cooperating teachers/mentors 26 26 16 8 30.80%

Candidates responding to the survey were asked to report their overall GPA. The mean GPA of the responding candidates was 3.89, which is slightly higher than the mean GPA of 3.81 of program candidates reported in the Brief, suggesting that candidates responding to the survey may have been academically stronger than candidates in the program overall. Audit Task A1 (2.1) Target: Results of professional knowledge assessments



Probe: Corroborate the results of the program assessments of professional knowledge by determining that TEAC on-line survey results reflect those of the program assessments. Finding: The TEAC on-line survey results are given in Table A1 below:

Table A1 On-Line Candidate, Faculty and Mentor

Mean Ratings on the Adequacy of the Candidates’ Accomplishments in Professional Knowledge

Topic of Survey Question Minimum Rating

Maximum Rating Mean Rating STD

Candidate ratings of own knowledge 2 5 3.94 0.83

Candidate ratings of adequacy of courses 3 5 4.24 0.66

Candidate ratings of adequacy of faculty 3 5 4.24 0.90

Faculty ratings of candidate knowledge 4 5 4.67 0.50

Mentors’ ratings of candidate knowledge 3 5 4.00 0.93

1=Inadequate, 2=Barely Adequate, 3=Adequate, 4=More than Adequate, 5=Excellent Ratings of professional knowledge by candidates, faculty, and mentors average at least “more than adequate,” with faculty ratings of candidate knowledge trending toward “excellent.” Verified Audit Task A2 (2.1) Target: Results of strategic decision-making assessments Probe: Corroborate the results of the program assessments strategic decision-making by determining that TEAC on-line survey results reflect those of the program assessments. Finding: The TEAC on-line survey results are given in Table A2 below:


Mean Ratings on the Adequacy of the Candidates’ Accomplishments in Strategic Decision-Making











Mentors ratings of candidate knowledge 3 5 3.88 0.83

1=Inadequate, 2=Barely Adequate, 3=Adequate, 4=More than Adequate, 5=Excellent Compared to ratings of professional knowledge, ratings of strategic decision-making average somewhat lower, although still in the “more than adequate” range. No one rated lower than “adequate.” Verified Audit Task A3 (2.1) Target: Results of leadership skill assessments Probe: Corroborate the results of the program assessments of leadership skill by determining that TEAC on-line survey results reflect those of the program assessments. Finding: The TEAC on-line survey results are given in Table A3 below:


Mean Ratings on the Adequacy of the Candidates’ Accomplishments in Leadership Skill






Candidate ratings of adequacy of mentors 3 5 4.50 0.76


Mentors ratings of candidate knowledge 3 5 4.12 0.78

1=Inadequate, 2=Barely Adequate, 3=Adequate, 4=More than Adequate, 5=Excellent Regarding leadership skill, candidates enjoyed high ratings from both faculty and moderately high ratings from mentors and self-ratings. Verified Audit Task A4 (2.1) Target: Results of cross-cutting themes assessments



Probe: Corroborate the results of the program assessments of cross-cutting themes by determining that TEAC on-line survey results reflect those of the program assessments. Finding: The TEAC on-line survey results are given in Table A4 below:


Mean Ratings on the Adequacy of the Candidates’ Accomplishments in The Cross-Cutting Themes

Topic of Survey Question Minimum

Rating Maximum

Rating Mean Rating STD

Learning how to Learn



Mentor ratings of candidate knowledge 3 5 4.63 0.74

Multicultural Perspectives and Accuracy




Technology




1=Inadequate, 2=Barely Adequate, 3=Adequate, 4=More than Adequate, 5=Excellent For each of the cross-cutting themes, candidates, faculty, and mentors all gave ratings of at least “adequate.” Mean ratings all approached or exceeded “more than adequate.” Verified Audit Task B1 (2.1) Target: “The mission of the BYU School Leadership Program is to prepare individuals to be competent, caring and qualified educational leaders who have a vision and a compelling sense of purpose for the continuous improvement of schools and learning for all students” (p.4). Probe: Interview faculty; ask, “Why is the mission of the program so generic, given the specific mission of the university?”



Finding: Auditors met with 12 full time program faculty, and received various responses to the question. The general sentiment was that use of “middle of the road language” allows the program to remain firm to social equity in spite of the dominant political environment, which is conservative. Verified with error: While the answer satisfactorily addressed the question, it was clear to the auditors that program values and goals were not well aligned with the mission as stated, and the mission had received relatively little attention by the faculty. See the Proposal Addendum, Page 2, for an account of program deliberations following the audit visit and the new program mission that resulted from these deliberations. Audit Task B2 (2.1) Target: “To identify more specific evidence for Program Claims #1-7, we are developing the Student Learning Assessment, Tracking and Evaluation (SLATE) Report–an aggregated assessment that links targeted course- and internship-level assessments to specific student-learning outcomes and program claims… The design of the SLATE Report is still under development. The current plan is to identify course- and internship-level assessments that address the learning outcomes based on assessment effectiveness, breadth and efficiency” (p. 22). Probe: Interview faculty; ask, “How is this new system working in relationship to your own courses? How will this process change what occurs in your system? How do you envision your work in the future?” Finding: Auditors met with 12 full time program faculty. All agreed that the system helped faculty perceive their courses to be part of a more integrated whole, typified by this comment: “I feel more included, rather than working in isolation.” Another noted, “There is a sense of shared responsibility. We want to be the best. There is a responsibility for following through with what we said we will do.” Regarding impacts on faculty work, one person said, “It’s helped with designing new courses and syllabi. Using the four critical outcomes was very helpful by informing areas needing program improvement.” Another said, “It provides a roadmap for the kids; it helps students self-calibrate.” Another: “SLATE is not just an exercise to get a grade and pass…it is an exercise to improve.” Finally, one of the senior faculty observed, “this whole process has brought relevance to accreditation.” Verified [Note: Audit Tasks B3 – B5 relate to Table 5.4, p. 56 of the IBP, as the Target. The Probe was, “Interview Proposal authors; ask, “What progress has been made on each of the new assessments?” Quotes are paraphrased from program responses.] Audit Task B3 (2.1) Target: Table 5.4 (p. 56 of the IBP) Probe: Progress on SLATE Finding: “When we started this process, everyone thought they taught everything. We checked current assessments related to outcomes and found many gaps. We developed criteria to determine which of the assessments fit in the overall schema of program outcomes. All syllabi now include both course outcomes and contributions to



program outcomes. We have no rubrics as yet; we need to look at Utah standards and then tweak what we have.” Verified (progress is consistent with what is reported in the Proposal). The Proposal Addendum also contains additional material reflecting recent development of the SLATE system, including simplification of the report from three courses to one, and identification of specific assessments for use in the SLATE report. Program response: The last sentence should read: “… including simplification of the report from three course assessments per learning sub-outcome to one course assessment per learning sub-outcome, and identification…” TEAC response: TEAC appreciates the clarification. Audit Task B4 (2.1) Target: Table 5.4 (p. 56 of the IBP) Probe: Progress on the Internship Activity Report Finding: “This is intended to be a reflective process using a portfolio with 16 elements of evidence, aligned with the student manual and syllabi. We are still in the process of aligning internship documents. We are trying to ensure that documents reflect what is occurring in internships and their intended outcomes.” Verified: At the time of the audit, October 2012, the program had no tangible evidence of progress on documents aligned with the Internship, which would have led to a judgment of Not verified. The Proposal Addendum, however, contains an Administrative Internship Report form (page 37 of the Addendum), which contains all of the prescribed elements. Audit Task B5 (2.1) Target: “We plan to work on assessment rubrics during Fall 2012” (p. 32). Probe: Interview faculty regarding progress on assessments referred to in Table 5.4 (p. 56 of the Proposal); obtain evidence of progress on rubrics and other assessment tools. Finding: Table 5.4 indicates that rubrics were to be designed during Fall 2012 for the Administrative Internship Activity Report and the Internship Leadership Project Reflection. At the time of the audit, October 2012, the program had no tangible evidence of progress on rubrics for these assessments, which would have led to a judgment of Not verified. The Proposal Addendum, as noted in Audit Task B4 above, contains an Administrative Internship Report form, which is a type of rubric in that all required elements are noted as either present or not present. The form does not, however, call for a judgment about the quality of student work. The Addendum also contains a Scoring Guide for the Internship Leadership Project Personal Reflection Paper (p. 47); this Guide does call for a judgment of the quality of student work. Verified with error: of the two new forms contained in the Addendum, one does not call for a judgment of quality. Audit Task B6 (2.1) Target: “To identify more specific evidence for Program Claims #1-7, we are developing the Student Learning Assessment, Tracking and Evaluation (SLATE)



Report–an aggregated assessment that links targeted course- and internship-level assessments to specific student-learning outcomes and program claims… The design of the SLATE Report is still under development. The current plan is to identify course- and internship-level assessments that address the learning outcomes based on assessment effectiveness, breadth and efficiency” (p. 22). Probe: Obtain worksheets being used to analyze courses and internship experiences; assess rationale for including assessments as evidence of accomplishment of the program’s stated learning outcomes (Table 2.1, p. 14). Findings: Auditors obtained copies of spreadsheets used to identify potential course assessments for SLATE. Spreadsheets were used to construct a detailed matrix, listing courses and their assignments along one dimension, and student learning outcomes and their subcomponents along the other. This is shown as Appendix F.4b. Auditors queried program faculty on the criteria for inclusion of course assignments under learning outcomes. Faculty detailed the process: they collectively examined all course syllabi to insure that each course syllabus identified learning outcomes and their relationship to program outcomes. They then identified assessments for each class and how each assessment measured one or more course objectives and, by extension, program objectives. Verified: The faculty were able to provide details of the rationale used to identify embedded assessments for SLATE. Audit Task B7 (2.1) Target: “The pilot data indicate that all student respondent groups rated the School Leadership program above 4 (on a 5 point scale) which surpasses the 80% success threshold” (p. 45). Probe: Interview Proposal authors; ask how this and other performance standards were determined. Finding: Auditors posed this question to Julie Hite, principal author. She responded, “We wanted to set a benchmark to set a defensible starting point. Since 80% represents the bottom of the range for a grade of “B” this seemed reasonable. We plan to monitor this as we collect more student performance data.” Verified Audit Task B8 (2.2) Target: “The SWOT analysis in Table 5.2 summarizes our findings regarding TEAC Quality Principle II” (p. 59). Probe: Interview Proposal authors for rationale for using SWOT. Finding: Auditors interviewed the five members of the program’s Leadership Team. Julie Hite created items for the SWOT table in early 2011, as a means of stimulating discussion and creating energy for 1 and 5-year plans. The table was circulated to program faculty for comment, and addressing the consensus weaknesses became the basis for the plan. One member of the team noted that because of some “movement,” “threats may now be opportunities.” Verified with error: The SWOT analysis served as a stimulus for faculty learning, not as a summary of faculty learning, as implied in the Proposal.



Audit Task B9 (2.2) Target: Table A.3a: “Program Quality Control System: Summative Judgment and Key Issues” (p. 48) Probe: Interview Proposal authors for evidence of response to “key issues” identified in the internal audit, in particular those components judged to be “fair” or “poor.” Finding: Auditors interviewed the five members of the program’s Leadership Team. Under Governance, labeled “fair,” a key issue was to develop a “QCS Handbook.” Auditors obtained a copy of the new Handbook, dated 3 October 2012. Under “student Outcomes,” labeled “poor,” issues included “lack of a database and control systems to inquire about or track student outcomes” and this system is under development. In general, respondents were conversant with the table and strategies needed to address the identified issues. Verified Audit Task B10 (2.3) Target: Follow-up on response to Audit Task B3 Probe: Examine course syllabi; verify program’s statement that “all syllabi now include both course outcomes and contributions to program outcomes.” Finding: Auditors examined course syllabi for all courses in the program. Nearly every syllabus lists course objectives and how each of which is aligned with one or more of these outcomes. Exceptions include 620, Law, and 677, Data Driven Decision Making II, which do not list specific course objectives. Verified with error: While the statement that “all” syllabi include both course outcomes and contributions to program outcomes, the substantial majority (11 of 13) do. Program response: Your exception included “620, Law,” however 620 is Finance and 622 is Law. Therefore, we’ve included all three links to the syllabi that were posted for the audit:

The 620 (Finance) syllabus posted for the audit on the TEAC documents webpage did list specific learning outcomes for the course, aligned with program learning outcomes: http://education.byu.edu/edlf/documents/EDLF%20620%20Syllabus%20-%20Fall%202012.pdf The 622 (Law) syllabus posted for the audit on the TEAC documents webpage did list specific learning outcomes for the course, aligned with program learning outcomes: http://education.byu.edu/edlf/documents/EDLF%20622%20Syllabus%20-%20Fall%202012.pdf The 677 (Data Driven Decision Making) syllabus posted for the audit on the TEAC documents webpage did list specific course outcomes in paragraph form (rather than list form) and aligned them with the program learning outcomes. http://education.byu.edu/edlf/documents/EDLF%20677%20Syllabus%20-%20Fall%202012.pdf



TEAC response: TEAC appreciates the correction of course identifying numbers. Auditors re-reviewed the syllabi listed for the three courses in question, all offered in the Fall of 2012. The 620 syllabus does list learning outcomes and aligns them with program outcomes. While learning outcomes stated as “understand concepts of school finance” lack precision, precision of learning outcomes was not part of the audit target. The 622 syllabus contains an extensive course description, assignments and grading policies, but no specific learning outcomes other than statements like, “we will become familiar with the animals in their legal zoo,” and no links to program outcomes. The 677 syllabus contains both student learning outcomes and links to program outcomes. TEAC’s judgment, modified slightly, then becomes: Verified with error: While not “all” syllabi include both course outcomes and contributions to program outcomes, all but one of them (12 of 13) do. Audit Task B11 (2.3) Target: Table 2.4: BYU School Leadership Student-Learning Outcomes Alignment by Curriculum. Probe: Examine syllabi for each of the courses designated as addressing “primary” leaning outcomes. Findings: Auditors examined all of the syllabi listed for courses in this table: 601, 602, 610, 614, 620, 622, 627, 629, 631, 650, 668, 676, and 677. Each course covers the topics highlighted in the table. Verified: the table is essentially as portrayed. Audit Task B12 (2.3) Target: “The QCS [Quality Control System] Handbook defines the quality control system and provides descriptions and information regarding each of the six quality control system program components” (Appendix A, p. 5). Probe: Obtain a copy of the Handbook and verify contents. Findings: Auditors examined the Handbook. It contains a matrix, listing program goals by program components (program governance, program entry, program design/delivery, student outcomes, program resources, program outcomes). Components are examined at several “levels of inquiry”: goals, processes, outcomes, processes for outcome use, outcome use, conclusions, and recommendations. The Handbook contains one section for each program component, including relationships with other components. Verified Audit Task B13 (2.3) Target: “The BYU School Leadership Program provides students multiple opportunities to learn in the field with administrative internships that are structured to provide substantial, sustained, standards-based work in real settings, planned and supervised cooperatively by trained university personnel and by school/district leaders who are trained mentors” (p. 7). Probe: Interview mentors and students in both program tracks, the Leadership Preparation Program (LPP, full time), and the Executive School Leadership Program



(ExSL, part time). Ascertain parity of these tracks, from both student and mentor perspectives. Finding: Auditors interviewed seven students (4 full time, 3 part time) and six mentors. Mentors indicated that because part-time students are otherwise engaged during regular school hours, their experience may be more “narrow” than that of the full time students. Part-time students indicated that, while they were drawn to the ExSL program by the opportunity to work full time and still go to school, they received less mentoring and support than the full-time students. One student said, “ExSL students have to get their own internships while LPPs are handed an internship.” Another said, “There are no clear expectations for a lot of projects and assignments.” None of the full time students expressed similar concerns. Not verified: Both mentors and students suggested that full time (LPP) students enjoy a richer and more supportive educational experience than part-time (ExSL) students. See section titled, “Alignment and Integration Across Two Program Tracks,” (Proposal Addendum, pp. 8 – 11), for a statement of how the Program has responded to this judgment. Program response: “One student said, ‘ExSL students have to get their own internships…’” While a student may have said this (that part may be accurate –we don’t know), this is an inaccurate representation of the ExSL internship process. No indication is given that the auditors sought to probe or further verify this statement with the Internship Supervisors or faculty. We do provide all students the option of pursuing more than the required 450 internship hours. The primary benefit of the ExSL program, as indicated in the Addendum, is its flexibility. An example of that flexibility is that ExSL interns have the option to request specific internship opportunities that are then arranged by our Internship Supervisor and the districts. The students are not required to pursue their own internship placements. TEAC response: This is a fair rejoinder: one student’s assertion of a fact does not make it so. The intent of the target was not to probe whether students have to obtain their own internships but to assess perceived parity of tracks from both student and mentor perspectives. The student’s comment reflected perceived lack of parity. A substantial part of the Addendum (pp. 4-11) focuses on how the program has resolved to improve parity of the tracks, an acknowledgment by the program that at least the perception of a problem exists. The judgment remains Not verified. Audit Task B14 (2.3) Target: “The program facilitates field-based administrative internships in a variety of school locations with trained mentor principals who work with students over time to facilitate their learning” (p. 8). Probe: Examine training materials used to prepare mentors for their roles. Finding: Auditors examined instructional materials used during the program’s annual orientation and training of mentors. These included notes from a planning meeting, June 6, 2012, the agenda for the “annual mentor and internship training” session on July 18, 2012, and copies of Powerpoint presentations. The agenda included presentations on five mentor standards, each led by an experienced mentor (including Garrick Peterson of the audit team), a description and discussion of the School Leadership Project, and small group discussions organized by program track.



Verified Audit Task B15 (2.3) Target: Mentors’ assessment of the quality of the BYU school leadership program Probe: Interview mentors; ask them to define their role as mentors; why they affiliate with BYU, and what they have gained from the experience. Finding: Auditors met with six program mentors. In addition to helping students “put theory into practice” and helping them “shape their beliefs, values, and practices,” mentors focused on their roles as helping students develop professional networks, and ultimately an administrative position. For example, one person said, “I introduce [students] to a broader circle of administrators”; another said, “I expand their visibility in the administrative community.” When asked why they affiliate with BYU, answers reflected both the quality of the students (“students are remarkably more prepared than those at other institutions”) and the quality of their interaction with students (“students are able to engage in deep conversations”). Mentors typically expressed benefits that are both personal (“I gain different perspectives”) and institutional (“student projects typically benefit the school”). Verified Audit Task B16 (2.3) Target: The validity of mentors’ ratings. Probe: Corroborate the program assertion of the validity of mentor ratings of candidates by determining that TEAC survey results reflect the raters’ preparation for their role. Finding: The results are given in Table B16 below:

Table B16 Mentor Ratings of Their Connections With the Teacher Education Program

Minimum

Rating Maximum

Rating Mean

Rating STD

Relationship with program faculty 3 5 4.38 0.92

Training for evaluation role 3 5 4.38 0.74

Understanding of program 3 5 4.50 0.76 1=Inadequate, 2=Barely Adequate, 3=Adequate, 4=More than Adequate, 5=Excellent

Mentors gave ratings of at least “adequate” regarding their relationships with program faculty, training for their evaluation role, and understanding of the program. Means in each case were well above “more than adequate.” Verified



Summary of Tasks Related to Quality Principle II: Evidence of a Quality Control System The auditors were able to verify that the program’s quality control system is more or less as it is described in Appendix A and that the internal audit occurred also as described. The rationale for assessment, an embedded assessment model (SLATE), is well thought-out. The internal audit is unusually comprehensive. At the time of the audit visit (October 2012) local assessments in the form of rubrics had not been fully developed, and no plan existed for evaluating their validity and reliability. Further, the dominant perception of both students and faculty was that the part-time (ExSL) track lacked parity with the full-time (LPP) track. The program addressed these concerns in an Addendum to the proposal, submitted on March 1, 2013. Program response: “Further, the dominant perception of both students and faculty that the part-time (ExSL) track lacked parity with the full-time (LPP) track.”

• The finding that this was a current “dominant perception” of the faculty surprises us (the faculty). Did you find evidence of this among the current faculty given all the work that we’ve done in the past 3 years to bring both tracks toward the parity?

• Given your interaction with a small sample of students, we also suggest that this may not be a “dominant” perception among students. It may be more accurate to indicate that this perception was found in interviewing the students.

• We acknowledge that the full-time track student have the option to pursue additional internship hours beyond the 450 required hours. However, as mentioned in the addendum, there is parity across the tracks in all required program components.

TEAC response: See response to Audit Task B13. The statement from the summary, “the dominant perception of both students and faculty was that the part-time (ExSL) track lacked parity with the full-time (LPP) track,” should be changed to read, “the dominant perception of both students and mentors was that the part-time (ExSL) track lacked parity with the full-time (LPP) track.” TEAC appreciates the correction.



In Table C.1 below, the auditors have indicated whether they have found evidence that satisfies each requirement for monitoring and control of program quality. Hyperlinked text refers to an audit task that explores the documentation further.

Table C.1 Quality Control of Capacity: Monitoring and Control (Component 2.3)

Documents were Found, Found in Part, Not Found, Not Checked or Not Available for Inspection with regard to parity between the program and institution in each area of TEAC’s Requirements

Finding Target (choose at least one for each subcomponent—indicate

chosen target in bold) Auditors’ Probe

2.3.1. Curriculum (Target #1)

VERIFIED Formal notification from the state that it has approved the program.

Verified by State Rep.

2.3.2 Faculty (Target #2)

FOUND

Minutes of a meeting show that the Brief Proposal was considered and approved by the faculty. “Approved by Faculty 30 August 2012” (IBP, Cover page)

Documentation was provided.

2.3.3 Candidates (Target #3)

FOUND Admissions policy of the program is published.

http://education.byu.edu/edlf/admissions_matriculation.html

2.3.4 Resources (Target #4)

Satisfactory TEAC survey results from faculty & students. Covered as audit tasks.

C. Tasks Related to Quality Principle III: Documentation of Program Capacity



In Table C.2 below, the auditors have indicated whether they have found evidence that satisfies each requirement for monitoring and control of program quality. Hyperlinked text refers to an audit task that explores the documentation further.

Table C.2

Parity Between the Program and the Institution (Component 3.1) Documents were Found, Found in Part, Not Found, Not Checked or Not Available for Inspection

with regard to parity between the program and institution in each area of TEAC’s Requirements

Finding Target (choose at least one for each

subcomponent—indicate chosen target in bold

Auditors’ Probe

3.1.1 Curriculum (Target #5)

FOUND

The number of credits required for degree at the institution and program are comparable. Appendix B, p..54

Department of Educational Leadership & Foundations requires 43 credits for LPP track and 41 credits for ExSL track. Brigham Young University requires a minimum of 30 credit hours for master’s degree.


FOUND

The percentage of faculty with terminal degrees in program and in the institution shows parity. Appendix B, p.55

The proportion of terminal degrees in Department of Educational Leadership & Foundations is 100 percent. The proportion of terminal degrees in Brigham Young University is 92.3 percent See Table - BYU Faculty Credential Alignment with Courses

3.1.3 Facilities (Target #7)

FOUND

The space and facilities assigned to the program and to similar programs shows parity. Appendix B, p. 56

All faculty members have their own office with average size of 120 sq. ft. in Department of Educational Leadership & Foundations. All faculty members have their own office with average size of 118 sq. ft. in Brigham Young University. Confirmed through faculty and student discussions, classroom visits, and tour of the university.

3.1.4 Fiscal and administrative (Target #8)



FOUND

The budget allocations per student in the program and in the institution show parity. Appendix B. p. 58

The EDLF personnel budget has remained steady over the past 3 years. The University allocates budgets to colleges. The University’s Board of Trustees determines the allocations to the colleges based on resource planning information provided by each college. Colleges determine allocation of department budgets.

3.1.5 Candidate support (Target #9)

FOUND

The program students have the same access to services as other students in programs at the institution. Appendix B, p. 59

Verified through discussions with the faculty and students. There is no difference in advisement, academic tutoring, financial aid and career placement as other students in the institution.

3.1.6 Candidate complaints (Target #10)

FOUND

Candidate complaints proportionally no greater or significant than the complaints by candidates in the institution’s other programs. Appendix B: p. 61

There is a published university policy for students wishing to submit complaints relating to faculty.

In Table C.3 below, the auditors have indicated whether they have found evidence that satisfies each requirement for sufficiency of program quality. Hyperlinked text refers to an audit task that explores the documentation further.

Table C.3 Quality Control of Capacity: Sufficiency (Component 3.2)

Documents were Found, Found in Part, Not Found, Not Checked or Not Available for Inspection with regard to parity between the program and institution in each area of TEAC’s Requirements

Finding Target (choose at least one for each

subcomponent—indicate chosen target in bold

Auditors’ Probe

3.2.1 Curriculum (Target #11)

FOUND

Credit hours required in the subject matter. Appendix B, p. 54

Department of Educational Leadership & Foundations require 43 credits for LPP track and 41 credits for ExSL track. Brigham Young University requires a Minimum of 30 credit hours for a master’s degree. http://education.byu.edu/edlf/new_project/program_tracks.html




FOUND

Full-time faculty selected at random have a terminal degree (major or minor) in the areas of course subjects they teach.

All faculty vitas were reviewed – See Table - BYU Faculty Credential Alignment with Courses


Satisfactory TEAC survey results from program faculty. Covered as audit tasks.


FOUND

Auditors’ observations of at least two class sessions find that the rooms and equipment constitute adequate instructional settings.

Auditors observed classes at both on the main campus and satellite campuses. Satellite campus facilities are in area high schools, two of which were visited by auditors. Classrooms are of ample size, and equipped with appropriate audio-visual technology.

3.2.4 Fiscal and Administrative (Target #15)

FOUND

Education faculty teaching load aligns with the institution average. Appendix B, p. 57

EDLF faculty generally teach between 12-15 credit hours per 10 month contract. BYU faculty generally teach 15-16 credit hours per 10 month contract Verified through discussion with the faculty, dean, and university administration.

3.2.5 Candidate support (Target #16)

Satisfactory TEAC survey results from students and faculty. Covered in audit tasks.

3.2.6 Policies and practices (Target #17)

Found An academic calendar is published. http://saas.byu.edu/calendar/2012.php


FOUND

Random selections of two pages in the catalog that deal with the program have no inaccurate statements about the program.

http://education.byu.edu/edlf/new_project/school_leadership_program.html


FOUND

Grading policy of the program is published and is accurate. Appendix B, p.63

Grading policies are communicated in course syllabi. Grading policy and information is also available on the university website: http://saas.byu.edu/registrar/records/grades.php




FOUND Transfer of credit policy and transfer of student enrollment policy are published.

http://education.byu.edu/edlf/admissions_matriculation.html


FOUND

Program has procedures for student complaints. Appendix B: p. 63

http://education.byu.edu/ted/student_info/complaint.html




N/A

If the audited program or any option within the program is delivered in a distance education format, the auditors verify that the program has the capacity to ensure timely delivery of distance education and support services and to accommodate current student numbers and expected near-term growth in enrollment.

Commitment to the program and the resources needed were verified in discussion with the faculty, dean, and VP. http://is.byu.edu/site/courses/coursepolicies.cfm#


N/A

If the audited program or any option within the program is delivered in a distance education format, the auditors verify that the program has a process to verify the identity of students taking distance education courses.

In Table C.4 below, the auditors have documented the results of the Call for Comment, which TEAC requires be distributed “to its communities of interest and to members of the public” according to Policy XXXVII (Operations Policy Manual, 2012 edition, page 43). Hyperlinked text refers to an audit task that explores the documentation further.

Table C.4 Call for Comment

Call for comment to third parties distributed as required by TEAC policy (Target #23)

# Positive Comments

# Negative Comments

# Mixed Comment

s

No comments

Summary of Tasks Related to Quality Principle III: Documentation of Program Capacity

Of the 23 potential targets associated with Quality Principle III, the auditors examined 18 and found that of these, all could be confirmed with the documentation provided by the program. These targets indicate that on the whole the program has parity with the institution and has documented its capacity for quality. Program capacity is confirmed because the preponderance (at least 75%) of the supporting evidence was consistent



with claims of capacity and commitment and/or further audit tasks verified associated capacity targets.

D. Tasks Related to Quality Principle III: Auditors’ Judgment of Institutional Commitment

Auditors met with Academic Vice President Brent W. Webb and Dean K. Richard Young. Webb noted that the department of Educational Leadership and Foundations has a “significant history”: education is “critical to everything at BYU,” and that unlike many other universities, “we make good use of the talent and expertise in the College at the University level [referring to the Center for Improvement of Teacher Education and Schooling], in particular the technology expertise in the department.” He cited Julie Hite, chief author of this IBP, as having such respect on campus that she had been asked to direct program reviews university-wide. He noted also that Educational Leadership and Foundations is one of only “two or three” departments in the entire university that is graduate-only. When asked, “what evidence do you have of institutional commitment to this specific program?”, Webb pointed to a 2009-2010 hiring freeze, when deans were told to submit a strategic plan for re-allocating faculty positions. Not only did Dean Young not cut any faculty positions in the program, he moved to fill vacant positions. Dean Young confirmed this. Finally, Vice President Webb identified that three top strategic priorities – international leadership, education policy, and school leadership – will all involve the department as a critical player. Program response: While Julie Hite was the chief author of the IBP, Dr. Steven J. Hite (also in the department) was the faculty member that was asked to work directly with the Assistant to the President to conduct program reviews university-wide, among other University-level responsibilities, and did so for 4 years. TEAC response: TEAC appreciates the correction. TEAC also surveyed students and faculty regarding aspects of the institutional commitment to the program. Results are in Table D.1 below:

Table D.1 On-Line Candidate and Faculty Mean Ratings on

Indicators of Institutional Commitment

Survey item Number Raters

Minimum Rating

Maximum Rating

Mean Rating

Standard Deviation

Candidate Ratings

Appropriateness of Classrooms, Equipment, Supplies

17 3 5 3.88 0.78

Availability of Classrooms, Equipment, Supplies

17 2 5 3.76 0.83



Helpfulness of Candidate Support Services

15 2 5 3.80 1.01

Availability of Candidate Support Services 15 3 5 3.87 0.92

Faculty Ratings

Institutional Commitment to Program

9 2 5 4.33 1.00

Resources for Teaching 9 1 5 4.11 1.36

Facilities for Teaching 9 2 5 4.22 0.97

Helpfulness of Candidate Support Services

9 1 5 4.11 1.27

Availability of Candidate Support Services 9 1 5 4.00 1.22

1=Inadequate, 2=Barely Adequate, 3=Adequate, 4=More than Adequate, 5=Excellent These findings are notable for the differences in mean ratings between faculty and students, with the former showing higher ratings overall, in the “more than adequate” range, while student ratings were in the “high adequate” range. The faculty data are also notable for the dispersion of ratings, which cover the entire range from 1 to 5.

Summary of Tasks Related to Quality Principle III: Auditors’ Judgment of Institutional Commitment

All empirical evidence points to strong institutional support for the program from top university leadership. The auditors did observe some disharmony between the program leadership and the leadership of the McKay school, tensions that preparation of this Proposal has helped alleviate.

V. Audit Opinion The scoring and meaning of the audit task findings. Each audit task is scored in one of four ways: • Verified, indicating that the auditors found that the target was accurately

described or represented in the Brief • Verified with Error, indicating that the auditors found some inaccuracy in the

target, but the inaccuracy did not alter the basic meaning of the target • Not Verified, indicating that the auditors found inaccuracy in the target that did

alter its the basic meaning • Disclaimer, indicating that the auditors were unable to undertake the task.



Table V: Audit Findings and Audit Opinions for the Brief

TEAC Element 1. Number of targets

2. Number of verified targets*

3. Number of targets with errors**

2/1 %

3/1 %

Audit Opinions

1.0 Evidence of

Student Learning

N/A N/A N/A N/A N/A N/A

2.0 Institutional

Learning and Quality Control

20 19 5 95 25 Clean

Overall totals 20 19 5 95 25 Clean

* Targets scored as Verified or Verified with Error ** Targets scored as Verified with Error or Not Verified Audit Opinion: The Inquiry Brief Proposal overall received a clean audit opinion. Since 19 of the 20 targets were verified, the Proposal (when augmented by the Addendum) was found to be acceptably accurate and trustworthy. The auditors are initially guided in their award of clean, qualified, adverse, or disclaimer audit opinions by the following considerations: an element receives a clean opinion if at least 90% of its associated targets are confirmed. An element is given a qualified opinion when at least 75%, but less than 90%, of its targets are confirmed; or if more than 25% of the targets reveal misstatements of any kind (that is, if the associated audit tasks are scored as either Verified with Error or Not Verified). If less than 75% of the targets can be verified, the element or component receives an adverse opinion if the examined evidence did not support the target or a disclaimer opinion if the audit tasks could not be performed or completed. These guidelines are not strict rules, because a simple counting of outcomes of probes may be misleading with regard to the trustworthiness of the Brief. Some audit tasks may be more revealing than others. For example, some may have targeted only minor points, and some may be merely following up on other audit tasks on a single point. Others may probe significant and central targets in the case for accreditation. The guidelines may prove unreliable in cases where the number of audit tasks is small. The auditors therefore do not treat the guidelines or heuristics as rules that can be mechanically applied. If the findings suggest anomalies that make the heuristic unworkable, the auditors rely on their good judgments, explaining in their audit report the difficulties they experienced and the reasons for their opinions. The auditors are also alert to evidence that is at variance with how the program is represented in the Brief, and report events and experiences during the audit that were not fully consistent with the manner in which the program is portrayed in the Brief.



Finally, it must be emphasized again that the audit opinion is not an opinion about the quality of the program or the degree to which the evidence in the Brief satisfies TEAC’s quality principles and capacity standards. It is solely an opinion about whether the Brief is accurate as written.

VI. Recommendations

Because this is an Inquiry Brief Proposal and not a full Brief, auditors are empowered to provide formative feedback to the Program following completion of their on-campus work. In an exit meeting with the Program, auditors offered the following observations, expressed as “strengths,” “concerns,” and “recommendations.” Auditors determined that because the program had not been able to provide draft evaluation rubrics, or plans for determining their validity and reliability (see Concern 3 below), the audit report would be delayed until the program filed an Addendum covering these topics. Strengths

1. The program is to be commended for a thorough and comprehensive process, including its internal audit.

2. The Proposal has a solid plan and rationale for assessment. The SLATE system shows good promise as the program’s core strategy for assessment of student learning.

3. Learning outcomes are well-crafted and aligned with ISLLC standards. 4. Program faculty display candid self-appraisal and open disclosure of issues. 5. Faculty have experienced professional growth from the process of developing

this Proposal, and show energy moving forward, optimism about improvement, and openness to feedback and guidance.

Concern 1: Program mission is disconnected from program and university values. Recommendation: Redefine the mission from the inside-out as you understand your learning community better and what it stands for. Concern 2: The two program tracks lack integration, and the ExSL is viewed by both faculty and students as less prestigious. Students feel less supported, less engaged, less part of the community. Recommendation: Engage in creative thinking about how to compensate for the lack of parallel, in-depth experience. What can this track offer substantively that is unique? Concern 3: The incomplete assessment plan will not pass muster with TEAC in its current form. TEAC requires programs to have assessment measures and plans for determining reliability and validity. Recommendations:

• Set priorities for instrument development (we suggest focusing on SLATE and internship evaluation)



• Develop and disseminate clearer expectations about the final project. All parties concerned (i.e., students, mentors, internship coordinators, and faculty) do not appear to be clear about what constitutes a project and how it should unfold given the two tracks.

• Be more purposeful in instrument development: consider what you really need to know and what will be done with data collected. Strike a balance between being comprehensive and parsimonious.

• Put together a plan for faculty calibration and inter-rater reliability. • Set a target date for response to this recommendation.



VII. Audit Schedule

TEAC Audit Visit Department of Educational Leadership and Foundations

Brigham Young University October 2-4, 2012

Monday (October 1)

3:00-6:00

Salt Lake City International Airport http://www.slcairport.com/ http://www.slcairport.com/rental-cars.asp Arrival and check-in http://www.marriott.com/hotels/travel/slcvo-provo-marriott-hotel-and-conference-center/ (reservations made for the audit team) Map of BYU campus http://map.byu.edu/

Marriot Hotel and Conference Center 101 West 100 North Provo, UT 84601 (801) 377-4700

6:30 Audit team dinner on own (includes local practitioner Garrick Peterson)

Tuesday (October 2)

9:00-10:00

Meeting with Brief Authors Julie Hite, Vance Randall, Shannon Dulaney

307 MCKB (Has been reserved for Oct 2-4; place for the audit team to hold interviews and visits.

10:00-12:00

Work Session

327 MCKB Wireless access, printer, copier see secretary in 306 MCKB Tech Support – 2-7796

12:00-1:00 Lunch (campus dining facility) with Aaron Popham

Skyroom or CougarEat - WSC



1:00-4:00 Work Session 327 MCKB

4:00-5:00 Meeting with Part-Time Faculty Denis Poulsen and Chris Sorensen 307 MCKB

Wednesday (October 3) 8:30-9:00

Work Session 327 MCKB

9:00-10:00

School Leadership Program Committee: Pam Hallam, Julie Hite, Shannon Dulaney, Mike Pratt 307 MCKB

10:00-11:00 Meeting with Full-time Program Faculty 307 MCKB

11:00-12:00

Visit LPP Class 319 MCKB

12:00-1:00

Lunch (campus dining facility) with Aaron Popham

Skyroom - WSC

1:00-1:15

Tour of the TEC lab http://education.byu.edu/teclab/ and introduction to MYLink Aaron Popham

175 MCKB

1:15-2:00 Work Session 327 MCKB

2:00-2:50

Meeting with Students 319 MCKB

3:00-4:00

Academic Vice President Brent W. Webb, Dean K. Richard Young

A-387 ASB

4:00-5:00

Meeting with Mentor Principals and University Supervisors *Mentor Principals ( in process) University Supervisors: Denis Poulsen and Chris Sorensen

307 MCKB

Dinner on own Thursday (October 4)

9:00-9:30 Next Steps Meeting Julie Hite, Vance Randall, Shannon Dulaney

307 MCKB

9:30-12:00 Additional Work Session (if needed) 327 MCKB