nacsa authorizer evaluation report

8/11/2019 NACSA Authorizer Evaluation Report

1/38

National Association of Charter School Authorizers

June 25, 2013

NACSA Authorizer EvaluationReport

Authorizer

District of Columbia Public Charter SchoolBoard

Board Chair

John H.Skip McKoy

Executive Director

Scott D. Pearson


2/38NACSA Authorizer Evaluation Report: District of Columbia Public Charter School Board

Contents Rating Categories 3

Rating System 3

About the Authorizer 4

Executive Recommendations 5

1. Application Decision-Making 7

2. Performance Management Systems 17

3. Performance-Based Accountability 26

4. Autonomy 31

Sources 37

Biographies 38


3/38

NACSA Authorizer Evaluation Report: District of Columbia Public Charter School Board

Rating CategoriesAuthorization quality is rated in two categories:

Established

Refers to the authorizers practices as set out

on paper whether by policy, protocol, or other

means. It also addresses the way that the

authorizer communicates information about its

practices to relevant stakeholders within the

authorizing agency and to schools. This category

rates the authorizer based on what it plans to

do.

Applied

Refers to the authorizers practices as applied.

This category rates the authorizer based on what

it actually does, in practice.

Within each part of the evaluation, the rating

categories are defined more specifically withrespect to the authorizers responsibilities in that

area.

Rating SystemFor each category (established or applied), the

authorizer receives a rating as follows:

Well-Developed

Commendable in that it meets or exceeds

NACSAs Principles & Standards.

Approaching Well-Developed

Fundamentally sound in that it contains most

aspects of a well-developed practice but requires

one or more material modifications to meet

NACSAs Principles & Standards.

Partially Developed

Incomplete in that it contains some aspects of a

well-developed practice but is missing key

components, is limited in its execution, or

otherwise falls short of satisfying NACSAsPrinciples & Standards.

Minimally Developed

Inadequate in that the authorizer has minimally

undertaken the practice or is carrying it out in a

way that falls far short of satisfying NACSAs

Principles & Standards.

Undeveloped

Wholly inadequate in that the authorizer has not

undertaken the practice at all or is carrying it out

in a way that is not recognizably connected toNACSAs Principles & Standards.


4/38


About the Authorizer

The District of Columbia Public Charter School Board (PCSB) was established in 1996 and

approved its first charter school to open in 1997. The DC Council passed mayoral control

legislation in 2007 that dissolved the DC Board of Education (DCBOE), making PCSB the

sole authorizer in DC. The legislation transferred 18 charter schools then supervised by the

DCBOE to PCSB. Currently PCSB authorizes 57 charter schools with an additional seven

schools in pre-operational status.

PCSBs mission is to provide quality public school options for D.C. students, families, and

communities through: a comprehensive application review process; effective oversight;

meaningful support; and active engagement of its stakeholders.

PCSB is overseen by a seven-member board. Board membership has been relatively

consistent over recent years. All of the current members have served on the board for the

last three years. One position has been held vacant for two years.

The staff currently numbers 31 and is organized by areas of authorizing expertise within the

following departments: executive leadership, school performance, human capital and

strategic initiatives, communications, and finance and operations. Many of the staff at PCSB

are new, as the organization saw significant staff turnover in 2011 and 2012 after long-time

leader, Josephine Baker, stepped down as executive director. The new executive director,

Scott Pearson, shifted the structure of the organization from a school-based model to its

current specialized model.

The 57 operating charter schools under the authority of PCSB are a mix of single-site and

multi-site schools. In recent years PCSB has seen an increase in the number of national

charter management organizations (CMOs) showing interest in opening schools in the

Washington, D.C. area. Almost 20 percent of the operating charter schools have an

alternative focus or serve special student populations (e.g. early childhood education or

adult education).

PCSB regularly evaluated the schools in its portfolio through the Performance Management

Framework (PMF) and categorizes them as Tier 1, Tier 2, or Tier 3. Tier 3 schools may be

eligible for a charter revocation hearing. Since the inception of the PMF, a number of Tier 3

charter schools have voluntarily closed or have not been renewed and the overall

achievement of students in schools authorized by PCSB has increased.


5/38


Executive Recommendations

Maintain Practice

The evaluation has identified the following policies and/or practices that the authorizer

should maintain:

Continue utilizing the robust set of best-in-class performance management systems thatestablish high expectations for academic, financial, and organizational performance.

These systems- the Performance Management Framework (PMF) and CHARM scores- are

thoughtful, thorough, and place a premium on school autonomy.

Continue the organizational structure of teams with specialized knowledge of different

aspects of authorizing.

Maintain a strong premium on charter school autonomy, as evidenced in policies and

which was affirmed by school operators.

Continue to operate, especially at the Board of Directors level, free from conflicts of

interest and outside interference.

Prioritize Practice

The evaluation has identified the following changes that should be priorities for addressing

weaknesses or gaps in authorizing practices:

Revise evaluation rubric to have clearly defined standards for approval. Conduct

evaluator training to ensure that those standards are consistently referenced in

evaluation and are the foundation for comments and recommendations for approval.

Clearly document its intervention policies including, but not limited to, defining the

types and triggers of evaluation and consequences for failure to improve.

Clarify how the PMF factors into intervention, revocation, and renewal decisions.

Consider developing policies that would allow it to grant further autonomy to high

performing schools.

Ratings SummaryEstablished Applied

Application Decision-Making Approaching Well-Developed Approaching Well-Developed

Performance Management Systems Well-Developed Approaching Well-Developed

Performance-Based Accountability Well-Developed Approaching Well-Developed

Autonomy Well-Developed Well-Developed


6/38


Develop Practice

The evaluation has identified the following ways in which the authorizer should further

develop promising practices:

Compile its various performance measurement tools- PMF, CHARM score, and Equity

Reports- into one document to create a robust picture of school performance from

various angles.

Continue developing alternative PMFs for nonstandard schools (Early ChildhoodEducation, Adult Education, Alternative Schools).

Continue to proactively meet with schools that will be up for renewal within the next

couple of years to clarify accountability goals and get ahead of the process.


7/38


Application Decision-Making

Does the authorizer approve applications based on applicants

demonstrated preparation and capacity to open and operate a qualitycharter school?

Established:Approaching Well-Developed

Applied:Approaching Well-Developed

Summary Assessment

Overall, the District of Columbia Public Charter School Board (PCSB) demonstrates sound

authorizing practices for application decision-making. PCSB through its application materials

request information from applicants that is aligned with NACSAsPrinciples and Standards

for Quality Charter School Authorizing. Specifically, applicants must provide information on

the educational program, budget and finances, board development and composition, and

operational capacity. All applications receive a thorough review by internal staff who

demonstrated significant knowledge and capacity to conduct thorough and rigorous

evaluations of the applications. Due to a recent strategic reorganization of the office, PCSB

no longer uses external experts for the application evaluation. While the current staff is

large and knowledgeable, the process does not have a consistent independent, externalperspective on the evaluation. PCSB does, however, consult with experts, including

members of their board, to review applications with a specific focus.

PCSB recently introduced a separate process for experienced operators. This application

process is reserved for school operators who currently run successful schools or networks.

It is part of a strategic shift to encourage the participation of national Charter Management

Organizations (CMOs). The experienced operator process focuses on the track record of

success of the schools or CMOs as well as plans for the proposed charter school including

information on the proposed educational program and development of an independent

board. The evaluation of these applications includes the growth plans of the CMO which

allows PCSB to evaluate the fiscal health and capacity of the organization, overall.

Evaluators have sufficient time to review the application and materials and schools have

ample time between authorization and opening day to plan. It was noted both during

observations and review of materials that the evaluations are not sufficiently rooted in key

criteria or standards. Although the comments on the applications were substantive, they did

not specifically reference how the application did or did not meet the standards for approval.


8/38


Recommended Actions

Revise evaluation rubric to have clearly defined standards for approval. Conduct

evaluator training to ensure that those standards are consistently referenced in

evaluations and are the foundation for comments and recommendations for approval.

Consider use of external consultants to provide independent, external analysis of

proposals. An investment has clearly been made in developing PCSB internal staff and

they are capable of conducting most application reviews; however, some reviews may

require content-specific knowledge.


9/38


Detailed Analysis

1.1

Application Materialsand Process

The authorizer provides clear

guidance and requirements

regarding application materials

and submission requirements and

runs a clear and well-structured

application process with realistic

timelines.

Established:

Well-Developed

Applied:


Analysis

PCSB is very thorough in both setting standards and expectations for

application review as well as communicating them to applicants. The

timeline, which is published in the charter application, sets forward a

detailed calendar for charter applicants that affords sufficient time for

the completion of all required tasks. The process commences in

September with the release of the application guidelines and

concludes with final recommendations approximately eight months

later in May. PCSB makes applicants aware of key dates (e.g.

information sessions and interviews) at the beginning of the process

and adheres to those dates.

To assist applicants (and reviewers) in ensuring that all application

materials have been submitted, PCSB has created a Charter

Application Filing Form. The form provides a checklist for all required

sections and attachments and allows applicants to submit missing

documents in a short but reasonable timeframe.

After reviewing samples of completed evaluations and observing

Defense Day it is evident that the evaluation standards and rubric

need further development. Charter application evaluations are

conducted almost exclusively by PCSB staff. There is an all hands on

deck approach to reviewing the applications that ensures that staff

are given sufficient time and resources to conduct a thorough

evaluation. The authorizer has a rubric that uses a multi-tiered system

to evaluate the written application. Although reviewers offered

substantive comments, they were not clearly aligned to the rubric or

evaluation criteria.

Better defining the standards for approval, and incorporating those

into the norming session, will produce evaluations and

recommendations that are more consistent across reviewers (in terms

of standards and format) and more focused. All evaluators participate

in an evaluation criteria norming session where the standards for

evaluation are shared along with some other basic information (ex.

different types of assessments). The evaluation criteria are also

shared with applicants and feedback is given to applicants after the

review has been completed.

The evaluation process consists of two in-person interviews for all

applicants: one conducted by staff and the second conducted by the

Board in a public hearing. PCSB also conducts a site visit for

experienced operators. Each of these processes could be further

developed. The documents we reviewed for the interview prep listed a

long set of questions but it was unclear what the purpose was

(clarification, assessing capacity to implement). Similarly, it was not

clear what rubrics or standards are used to assess the site visits to

schools run by experienced applicants.


10/38


1.2Educational Program

The authorizer has thorough

requirements and rigorous

evaluation criteria for the

proposed educational program

including the vision and mission

statements, educational

philosophy, curriculum and

instruction, teaching skills and

experience, calendar and daily

schedule, target population,

enrollment, and plans for

educating students with special

needs.

Established:

Well-Developed

Applied:


Analysis

The level of quality is consistently high throughout PCSB charter

application, including the educational program. The section related to

the educational program is robust and requests information from

applicants regarding the proposed schools mission and vision,

educational philosophy, educational needs of target student

populations, academic and operational goals, curriculum, methods of

instruction, standards for achievement and promotion, and how they

will meet the needs of all students.

PCSB, in its review, places significant emphasis on goals and academic

targets. The D.C. charter school law places much weight on a schools

success or failure in meeting the goals and targets outlined in its

charter. In fact, this appears to be the primary criteria for making

high-stakes decisions around renewal and revocation. As such, the

application focuses on the goals a great deal and considerable

guidance is given to applicants about the types of goals they should

be setting.

In practice, tracking and monitoring charter school goals over the

term of the charter, which in D.C. is 15 years, has proved challenging.

Interviews with staff revealed that some goals became irrelevant andwere no longer tracked by the school or the authorizer. Recognizing

this challenge, PCSB is encouraging applicants to use their

Performance Management Framework (PMF) for its academic goal

setting. Though the authorizer cannot mandate this, they have done

significant engagement with the current charter community and have

emphasized it in the application to encourage participation. Adopting

the PMF for academic goals is a good step toward ensuring

consistency and rigor in the schoolsgoals as well as simplifying the

monitoring process.

A review of sample completed evaluation rubrics demonstrated that

reviewers sought a high level of explanation of the connection

between the educational program and goals of the school and its

stated mission. This was evidenced both by specific questionsregarding the use of multiple methodologies for instruction and

curricular materials as well as in the budget section questioning if

the budget would support the trainers required by one of the

programs. PCSBs evaluation of applicants target student population

was also very rigorous, at times identifying demographic assumptions

of the proposed target population that were incorrect and potential

impact that it might have on the proposed program. Overall, reviewer

comments were substantive and demonstrated thorough content

knowledge. They did not, however, reference the stated standards for

evaluation.


11/38


1.3Organizational Plan




proposed organizational plan

including the effective governance

and management structures and

systems (including staffing);

founding team members

demonstrating diverse and

necessary capabilities; and

understanding of legal

requirements related to opening

and operating a charter school.

Established:

Well-Developed

Applied:


Analysis

The level of quality is consistently high throughout PCSB charter

application, including the organizational plan. The application requires

a very thorough organizational plan that is singularly directed toward

guaranteeing successful opening and ongoing operation of the school.

PCSB requests a sufficient level of detail to ensure that information is

provided for all levels of the organization from the board, to school

leadership, to students.

PCSB requires extensive information related to the governance and

ongoing management of the school. The D.C. charter school law

requires an independent board of trustees to govern each charter

school. Furthermore, each board must be comprised of at least a

majority of D.C. residents and at least two parents of children

attending the school. The application asks for a board development

plan or specific board members if already identified. Questions are

designed to assess whether the board will be sufficiently diverse and

capable to effectively govern the school. The application also requires

submission of board policies and by-laws and PCSB will not grant final

charter authorization until they have been submitted and accepted.

The management or leadership structure of the school is also givensignificant weight. Applicants are asked to identify the key leadership

roles for the school. If candidates have been identified already,

information is requested on the background and experience of the

proposed leader. If key staff have not been identified, PCSB requires a

detailed plan for the recruitment, selection, and hiring of the

candidates. PCSB requests additional information related to operations

including a plan for the recruitment and admission of students to

ensure that the school will be fully enrolled on day one. Applicants

must also identify a facility or propose a plan for the identification of a

facility. A final charter will not be granted until a facility has been

identified and approved.

Similar to previous sections, reviewers provided substantive

comments (specifically about the lottery process including dates,sibling preference, etc.) but the comments were not rooted in the

evaluation criteria.


12/38


1.4Business/Financial Plan




proposed business plan including

financial viability of the plan

demonstrated through budget

projections that are aligned with

the proposed educational

program.

Established:

Well-Developed

Applied:

Partially Developed

Analysis

PCSBs business plan requirement is consistent with the overall high

level of quality throughout the charter school application. For all

schools, PCSB requests specific information on anticipated funding

levels and sources (both government and private), anticipated costs,

and contingencies. The application requests information that is

focused on the successful opening of a school (ex. contingencies if

funds are delayed or lower than expected) as well as continued

success in operation (five year budget including goals and objectives).

Additionally, all schools are required to adopt Generally Accepted

Accounting Principles for the maintenance of their financial records.

The process is appropriately differentiated for experienced operators.

The application requests three years of audited financial statements

and most recent internal financial statements for the organization as a

whole. The authorizer should consider adding the requirement for a

forward looking business plan for all CMOs to ensure the

organizations fiscal health will support its projected growth.

The application evaluations that were reviewed did not demonstrate

the same level of scrutiny as many of the other sections. For instance,

in one rubric, the finance section was entirely blank. Although therewere passing references to whether the budget would support the

proposed educational plan of the school, that analysis was not evident

in the review of the finance sections.


13/38


1.5Capacity


requirements and rigorous criteria

for evaluating the applicants

capacity to implement the school

plan effectively, including but not

limited to a substantive in-person

capacity interview with the

applicant group.

Established:

Partially Developed

Applied:


Analysis

The authorizer has a rigorous application process overall, however,

some elements of assessing a founding teams capacity are under-

developed. The application is strong in requesting sufficient

information regarding the founding teams. They are required to

provide biographies or resumes as well as state the teams purpose

for wanting to start a school in the community. Additionally, all

applicants participate in an in-person interview as part of the

application process.

Although all applicants receive an in-person interview, there are no

stated guidelines or criteria for the interviews or expectations around

participation from the founding team members. The rubric offers

reviewers a section to note potential questions for the interview but

the evaluation team did not observe any guidance around the content

or purpose of these questions. The interview preparation document

included a lengthy list of questions but it was not evident how the

questions were organized for the actual interview.

PCSB does conduct some due diligence of applicants, though it

appeared to be ad hoc. During observations ofDefense Dayit was

evident that reviewers had researched some founding teams beyondthe applications they received, but it was not clear that there were

guidelines for doing so. Similarly, site visits are conducted when

possible at campuses of experienced operators, though no rubric was

observed for those visits.


14/38


1.6Priorities andApplication Adaptations

The authorizer adapts the basic

application as necessary based on

identified needs including

specialized applicant types that

are commonly received and/or

desired program types.

Established:


Applied:


Analysis

The application process does not yet fully reflect the diversity of the

school types offered in D.C. While the staff and evaluators appear to

have the content knowledge and expertise to assess a variety of

proposed school types, the application itself is not modified to address

them. Similarly, the same rubric and evaluation criteria are applied to

all schools, regardless of type. This could pose challenges for the

authorizer in appropriately determining the quality of the proposed

program. D.C.s charter community is incredibly diverse and serves

students of almost all ages from pre-K through adult education. It also

has alternative schools that defy easy classification and serve a wide

range of students from special education to drop-outs. PCSB has also

received applications for blended and virtual models and the

authorizer is actively seeking to add a number of national CMOs to its

mix of home-grown schools.

Recently, PCSB has attempted to modify its application to address the

different needs of certain school types by creating an experienced

operator application. Though some revisions to this application have

been noted elsewhere in this report, the move signifies PCSBs

recognition of the differences in school and operator types.

Staff and stakeholders indicated, in interviews with the evaluation

team, a desire for a process to reflect the needs of the city and

various communities in the application. Currently, the application is

singularly geared toward assessing the objective quality of the

proposed program. Although applicants must demonstrate need for

the school, the evaluation does not thoroughly considerdemand for

the proposed school in one area versus another. PCSB is in the early

stages of working to identify and incorporate these priorities in their

application and evaluation process.


15/38


1.7Decision Alignment

The authorizer makes application

decisions that are informed by

and align with documented

evidence and analysis of the

extent to which the plan satisfies

approval criteria and the extent to

which applicants demonstrate

strong preparation and capacity to

establish and operate a quality

charter school.

Established:

Well-Developed

Applied:

Well-Developed

Analysis

PCSB has a demonstrated track record of basing approval and denial

decisions of new applicants on the quality of the proposal. Through

interviews at all levels (staff, leadership, and the Board) it was

consistently reported that decisions were not influenced by outside

factors and were based solidly on the overall quality of the proposed

school and the recommendations of staff. With the new organizational

structures, staff prepares recommendations with supporting

information and data to the Board for their consideration. Interviews

with staff and board members revealed a high level engagement with

the Board in reviewing the recommendations and supporting

documents as well as an appropriate amount of deference to staff

recommendations.

Over the three year history of application decisions that was provided

and reviewed, the overwhelming majority of Board votes followed staff

recommendations. There were, however, some instances when the

Board deviated from staff recommendations. Although these instances

were rare, staff were probed during interviews and provided

satisfactory explanations.


16/38


1.8Transparency

The authorizer has transparent

processes for both application

evaluation and application

decision-making.

Established:

Well-Developed

Applied:

Well-Developed

Analysis

Overall, PCSB is very thorough in communicating both its process for

new school applications as well as the expectations for approval.

Supporting materials that ensure transparency in the application

process include a timeline, guidance for how to apply, checklists for

applicants, standards for evaluation, its Performance Management

Framework, and training given to evaluators. Much of this information

is included in the application itself as well as on the website.

Additionally, PCSB provides much of this information at the beginning

of the process allowing applicants a complete view of the process and

sufficient time to plan and prepare all portions of the application.

The timeline specifies a date for a public information session at the

start of the process for prospective applicants. Additionally, staff

reported that they are available to assist applicants in the tactical

elements of submitting a charter school application.

Given the Boards solid track record of following staff

recommendations and the clear communication of criteria, applicants

have a satisfactory sense of what it takes to gain approval. This

sentiment was echoed by many in the charter leader round-table that

was held. In the event that an applicant is denied, PCSB staff hostfeed-back sessions with the applicant to detail the areas where the

application failed to meet the standards. Several applicants have

reapplied after an initial denial and received approval suggesting

that the feedback offered is substantive and relevant.


17/38


Performance Management Systems

Does the authorizer have effective systems for establishing and monitoringschool performance expectations and for holding schools accountable asnecessary to protect student and public interests?

Established:Well-Developed


Summary Assessment

PCSB has created a robust set of best-in-class performance management systems that

establish high expectations for academic and operational performance. These systems- the

Performance Management Framework (PMF), Equity Reports, and CHARM scores- are

thoughtful, thorough, and place a premium on school autonomy. The authorizer has

demonstrated an unwavering focus on quality and protecting the interest of students and

families. The PMF has given PCSB an effective tool for communicating school performance

against a set of common measures. The PMF has begun to reshape the charter landscape in

Washington, D.C. Since its inception the number of high performing schools has increased

and parents are overwhelmingly choosing to enroll in higher performing schools. This shift

to include a focus on standardized accountability measures has caused tension with some

charter operators and advocates who believe charters should only be held accountable for

mission-specific goals. PCSB has deftly walked the line between increasing accountability

and overstepping its statutory authority and should continue to work with school operators

and advocates to address their concerns.

PCSBs financial monitoring tools, particularly the CHARM score, are truly exceptional. These

tools strive to create a robust and comprehensive picture of each schools financial health

and communicate findings in a clear and straightforward manner. The process of developing

the CHARM score metrics has helped schools better understand expectations for financial

management and has resulted in stronger financial performance in the portfolio. PCSB is

also piloting Equity Reports this year, which report a number of metrics such as attendance,

attrition, and academic results by subgroup to help illuminate how well schools are serving

different groups of students. Information from these reports will inform decisions about the

level of monitoring each school receives. This is another example of PCSB using the power

of setting high standards and shining a light on results to influence school actions to serve

the interests of students.


18/38


The current leadership of PCSB has strengthened the authorizers approach to performance-

based accountability. For example, in recent years PCSB has scaled back and simplified its

intervention policy, as part of a new approach that is both effective and sensitive to school

autonomy. While the authorizer has established certain triggers that automatically move

schools to a revocation hearing, it has taken a more flexible approach to working with

schools that are out of compliance or in need of improvement. While the authorizers current

approach to school intervention is reasonable, it is not clearly enough defined in policy.

PCSB has only recently begun conducting a charter renewal process, beginning in 2011. Thelength of the charter terms and the evolution of school and authorizer practices have caused

some difficulty in the renewal process. Many schools modified or abandoned goals over time

and the authorizer did not always clarify or consistently track performance against charter

agreement goals. The current administration of PCSB has made a painstaking effort to

clearly identify the appropriate accountability goals to include in the renewal process but

some stakeholders report that they are still unclear on what goals they will be evaluated

and the criteria for renewal.

Recommended Actions

Continue to prioritize the development and piloting of PMFs for nonstandard schools

(early childhood education, adult education, alternative schools).

PCSB should clearly document its intervention policies including, but not limited to,defining the types and triggers of evaluation and consequences for failure to

improve.

The authorizer should continue to proactively meet with schools that will be up for

renewal within the next couple of years to clarify accountability goals and get ahead

of the process.


19/38


Detailed Analysis

2.1 Contracting

The authorizer has a timely and

efficient process for developing

and executing the charter

agreement.

Established:

Well-Developed

Applied:

Well-Developed

Analysis

PCSB has well-developed systems and processes for establishing

performance contracts with the schools it authorizes. The authorizer

has developed a comprehensive standard charter agreement that is

customized with school-specific information and performance goals.

While the School Reform Act does not require the authorizer toestablish a charter agreement within a set timeframe, PCSB has not

had any documented issues with executing charter agreements in a

timely manner. Relevant pieces of the charter petition are

incorporated into the charter agreement and the authorizer has a clear

policy and process for schools to request charter amendments for

material changes to the charter. The charter amendment policy clearly

defines what constitutes a material change, including mission or

educational philosophy, articles of incorporation or by-laws, grade

levels served, and governance structure. School operators reported

overall satisfaction with the charter amendment policy and how it has

been implemented.

PCSB has established a robust oversight system that allows it to

monitor compliance with the terms of the contract. For example, theauthorizer utilizes industry specific software to track compliance with

regulatory reporting requirements. This has allowed the authorizer to

efficiently and effectively manage a heavy compliance load and work

with schools to resolve compliance issues when they arise. The

authorizer has also reorganized its staff structure to better track

charter compliance by creating the Equity and Fidelity Team that is

specifically tasked to monitor each schools compliance with the terms

of the charter.


20/38


2.2 School Opening

The authorizer ensures that

approved schools are prepared

adequately for opening.

Established:

Well-Developed

Applied:

Well-Developed

Analysis

PCSB has a clear and comprehensive checklist for monitoring new

schools readiness to openand their capacity to serve students with

disabilities. The checklist includes expectations for school enrollment,

staffing, health and safety, facilities, as well as instructional and

special education programs. Authorizer staff meet with new schools at

their site to complete the checklist and document specific items that

need to be in place to successfully open. If a school is unable tosatisfy the requirements they are given additional time and authorizer

staff will visit repeatedly until the checklist is complete.

In practice, many schools have struggled to receive their Certificate Of

Occupancy in a timely manner. Authorizer staff report that this is an

unpredictable process and that PCSB has helped schools obtain their

COOs on several occasions. If a school is unable to obtain their COO,

however, PCSB will not permit them to open.


21/38


2.3 Ongoing Monitoring

The authorizer has an effective

process for monitoring education,

financial, and organizational

performance of the schools it

authorizes.

Established:

Well-Developed

Applied:


Analysis

The authorizer has developed a system for ongoing monitoring of

school performance that is effective, efficient, and respectful of

autonomy. PCSB divides monitoring processes into three areas-

academic performance, financial performance, and compliance

monitoring. Together, these monitoring practices allow the authorizer

to stay reasonably informed of each schoolsperformance and to

proactively address areas of concern as they arise.

The Performance Management Framework (PMF) was created in 2009

to facilitate the evaluation of charter school performance against a

common set of academic performance metrics. The authorizer

involved stakeholders in the development of the PMF and even

scrapped an earlier version of the framework due to feedback from

school operators and charter advocates. The PMF establishes three

tiers of academic performance and includes measures on

achievement, growth, and key gateway (i.e. graduation rates) and

leading (attendance, retention) indicators. The authorizer uses data

from the PMF to create an annual School Performance Reports

document that is distributed widely in both print and electronic

formats. The School Performance Reportsare well designed and

communicate a complex set of information in a clear and

straightforward manner. The authorizer has several task forces in

place to develop appropriate measures for nonstandard schools (early

childhood education, adult education, and alternative schools), which

comprise 40 percent of the portfolio. Completion of these metrics

should remain a top priority.

The PMF has had a significant and immediate impact on the quality of

the D.C. charter sector. Over the last two years the number of Tier 1

schools has increased while the number of Tier 3 schools has declined

(several Tier 3 schools have even voluntarily closed). The authorizer

has also made information from the PMF widely available to parents,

who seem to be using it to inform their choices. Demand for Tier 1

schools, as evidenced by student enrollment and wait lists, is

extremely strong while demand for Tier 3 schools is weak.

PCSB has created an exemplary system to monitor charter financial

performance. The centerpiece of PCSBs financial monitoring is the

innovative CHARM score. These scores are based on a sophisticated

set of financial indicators pulled from audited financials and

benchmarked against other PCSB charter schools. CHARM scores that

are outside the norm trigger site visits and additional monitoring by

PCSB. The vast majority of schools with low CHARM scores in the first

year made significant improvements and were moved off the

concerned list by the second year of implementation. The authorizer

has not yet made CHARM scores public but is considering doing so this

year.

The new site visit protocol focuses exclusively on instructional quality

in schools, using a rubric that was designed in consultation with school

operators, moving away from a more wide-ranging and compliance-

based approach in previous iterations. These new rubrics provide

valuable information about both the quality of instruction at the school

and the alignment of the instructional program with the vision of the

school, which is factored into intervention and renewal decisions.


22/38


2.4 SchoolIntervention/Revocation

The authorizer has effective

policies and practices for school

intervention and revocation and

conducts merit-based

interventions, including revocation

where appropriate, in response to

clearly identified deficiencies in

the schools record of educational,

organizational and/or financial

performance.

Established:


Applied:


Analysis

PCSB has integrated the PMF into its monitoring strategy, using

information from School Performance Reportsto determine

interventions, and schools with three consecutive years in Tier 3

status are automatically brought before the Board for a charter

revocation hearing.

Over the last several years PCSBsapproach to school intervention hasshifted from a hands-on, tiered intervention system to a less

centralized process that emphasizes school autonomy and public

accountability. While the authorizer has established certain triggers

that automatically move schools to a revocation hearing, it has taken

a more flexible approach to working with schools that are out of

compliance or in need of improvement. For example, schools that are

out of regulatory compliance will receive notices about the problem

from the authorizer and will be brought before the Board if the

situation is not remedied in a timely manner. Schools with low

academic performance can be placed on a probationary status

requiring additional monitoring and program improvement planning.

PCSB previously had one comprehensive intervention policy whereas

now interventions are a part of several different policies (the data

submission policy, the special education audit trigger policy, etc.).

While the authorizers current approach to school intervention is

reasonable, it should consider documenting it in a single

comprehensive policy to give schools clarity about what would lead to

an intervention.

PCSB has the authority to approve which auditors work with charter

schools and has used the CHARM process to proactively educate them

to achieve a more uniform quality and standard of audit reports. After

the first year of working with auditors the number of reportable

findings in audits fell by one third, suggesting that PCSBs clear

expectations of financial management are having a positive impact on

charter schools. This is an excellent example of how charter

authorizers can use high expectations to impact the quality of the

charter sector.


23/38


2.5 Renewal

The authorizer runs a well-

structured renewal process

including clear requirements; a

meaningful opportunity for the

school to present information and

respond to the authorizers

findings; clear communication;and prompt notification of

decisions.

Established:


Applied:


Analysis

PCSB has only recently begun conducting a charter renewal process,

beginning in 2011, because of its 15-year term charter agreements.

The authorizer has created a comprehensive document that details the

renewal process, including specific timelines and criteria in the 2012-

2013 Charter Renewal Guidelines document. This document provides

step-by-step instructions for preparation of the charter renewal

application and encourages schools to adopt the PMF as their goals fortheir next contract. Some school leaders report a reticence to adopt

the PMF as their accountability goals in the contract as it is a living

document and subject to change beyond the schools control. PCSBs

renewal process gives schools the opportunity to request a public

hearing to respond to the cumulative record of performance and make

the case for their renewal.

In practice, the length of the charter terms and the evolution of school

and authorizer practices have caused some difficulty in the renewal

process. Many schools modified or abandoned goals over time and the

authorizer did not always clarify or consistently track performance

against charter agreement goals. The current administration of PCSB

has made a painstaking effort to clearly identify the appropriate

accountability goals to include in the renewal process but some

stakeholders report that they are still unclear on what goals they will

be evaluated and the criteria for renewal. The authorizer should

continue to proactively meet with schools that will be up for renewal

within the next couple of years to clarify accountability goals and get

ahead of the process.


24/38


2.6 Closure

Following non-renewal, revocation

or voluntary return of the charter,

the authorizer has an effective

plan for and ensures orderly

closure of schools.

Established:

Well-Developed

Applied:

Well-Developed

Analysis

The authorizer has a comprehensive school closure policy that clearly

identifies roles and responsibilities for all parties in the orderly closure

of a school. The plan provides for securing of records, securing of

public property, orderly wind up of the schools finances, and

communicating with affected families.

The authorizer works proactively to coordinate closure efforts withboth D.C. Public Schools and the Office of the State Superintendent

and meets on-site with the affected school early in the process to

ensure strong financial controls and accountability. The authorizer has

dedicated significant resources to working with families affected by

school closures to ensure every student is placed in a new school. To

this end, the authorizer hires part-time specialized staff to handle

closures, both at the administrative level and to work directly with

families through the closure process.


25/38


26/38


Performance-Based AccountabilityDoes the authorizer have rigorous, appropriate standards by which it holdsschools accountable for results? Are decisions made with the intent tomaintain high standards and protect the students and the publicsinterests?



Summary Assessment

PCSB establishes very high standards of performance for its schools and uses these

standards to hold schools accountable for results. PCSB has a track record for enforcing its

standards and making difficult decisions regarding charter status and has revoked charters

for academic, financial, and organizational reasons. The authorizer holds schools

accountable for mission-specific goals established in the contract and regulations defined in

state and federal law. PCSB has also developed the PMF to evaluate schools on a set of

common academic and financial performance indicators. While this information is used toguide monitoring and accountability decisions, it is not part of the criteria for charter

renewal or revocation. Many school operators reported confusion over the role of the PMF in

the renewal process. PCSBs renewal decisions also indicate a commitment to upholding

high standards, with all schools recommended for renewal ranking in the top two tiers of the

PMF (with the exception of one alternative school that will be eligible for alternative

measures that are now being piloted).

As an organization that makes high-stakes, data-based decisions, PCSB must be able to rely

on the quality of its data. In order to fulfill its monitoring and oversight requirements PCSB

requires schools to submit large amounts of data on a regular basis. Many school operators

report that the current system is time consuming and often results in data errors. PCSB has

made an effort to address these problems but school operators indicated that they continue

to persist.

Recommended Actions

Clarify how the PMF factors into intervention, revocation, and renewal decisions.

Work with school operators to develop more effective data collection processes.


27/38


Detailed Analysis

3.1 EducationalPerformance

The authorizer holds schools

accountable for academic

performance using objective and

verifiable measures that address,

at a minimum, student

achievement, student growth, and

post-secondary success as the

primary measures of school

quality.

Established:

Well-Developed

Applied:


Analysis

PCSB establishes high standards of educational performance for all of

the schools it authorizes. These expectations are clearly defined for

each school in the charter agreement and in the PMF. As established

in law, the goals defined in the charter agreement guide renewal andrevocations decisions made by PCSB. The individualized goals of the

charter agreements, however, limit the authorizers ability to assess

the overall and relative level of performance of schools in its portfolio.

The PMF was created to overcome this challenge and articulates a high

bar for performance. This information is used to inform the public

about the performance levels at each school and help guide PCSB

decisions on school monitoring and is not used as the basis for

renewal or revocation decisions, which are driven solely by each

schools ability to prove that it has met the goalsdefined in the

charter agreement.

In practice, the PMF has caused some confusion among charter

operators about accountability because PCSB has integrated

information from the PMF into its monitoring process. PCSB hasinstituted a policy that any school remaining on Tier 3 for three

consecutive years will automatically face a charter revocation hearing.

Though the actual revocation is based upon the schools ability to

prove that it has met the goals established in the charter agreement,

this is not clear to the majority of the stakeholders interviewed for the

evaluation. To alleviate this confusion the authorizer should clarify

exactly how the PMF is used in accountability decisions.


28/38


3.2 Financial Performance


accountable for financial

performance using appropriate

near term and sustainability

measures as the primary

indicators of a schools financial

viability.

Established:

Well-Developed

Applied:

Well-Developed

Analysis

PCSB uses the CHARM score to effectively monitor the financial

performance of each school. The CHARM score includes a number of

appropriate measures of financial performance, using data from

audited financial statements. These measures include key ratios such

as current assets position, capitalization ratio, and months of cash on

hand. Like the PMF, the CHARM score is not used as renewal or

revocation criteria, rather it guides staff decisions on what schools

need additional monitoring and support. Schools also regularly submit

financial statements as part of their board packets, with schools in

good standing required to submit on a less frequent basis. Charter

schools are required to submit four years of audited financial

statements as part of the renewal application.

All of the information on financial performance factors into staff

recommendations for renewals or revocations. Staff and board

decisions are guided by the fiscal management criteria listed in the

charter agreement as reasons for charter revocation or nonrenewal; a

pattern of non-adherence to generally accepted accounting principles

and no longer being economically viable. PCSB has closed a number of

charter schools for failing to meet these terms of the charter

agreement.


29/38


3.3 OrganizationalPerformance


accountable for compliance with

organizational performance

requirements established in law or

the charter contract, including

educational programrequirements, governance and

reporting, financial management

and oversight, and operational

requirements related to students,

employees, and the school

environment.

Established:

Partially Developed

Applied:

Partially Developed

Analysis

PCSB has comprehensive systems for monitoring organizational

performance and compliance but the effectiveness of one of these

systems is questionable. Schools are required to upload a variety of

data as part of the organizational performance monitoring process.

While the EpiCenter system for uploading board documents and other

forms received positive reviews from the stakeholders we interviewed,

a number of school operators reported significant problems uploading

their enrollment, attendance, and discipline data using the ProActive

software, in some cases causing erroneous information to be reported,

and requiring time-consuming problem solving and resubmissions.

Authorizer staff acknowledged that there were problems with the

ProActive system in the past, but believed the problems have largely

been resolved. This is incongruent with reports from school operators

that they are still struggling with the system. Given that this data is

used in accountability decisions and is incorporated into the PMF it is

imperative that data integrity be strong beyond reproach.


30/38


3.4 Decision Alignment

The authorizer makes

accountability decisions that are

informed by and align with

documented evidence and

analysis of the extent to which the

school satisfies performance

expectations. The analysis

presented to decision-makers is of

high quality and the merits of the

decisions themselves show

decision-making is based on

thoughtful analysis ensuring that

only the charter schools that meet

or exceed expectations are in

operation. (Note: this section

focuses on decisions by the

authorizer other than the

application, which is addressed in

1.7.)


Applied:


Analysis

The authorizer has a demonstrated track record of making evidence

based decisions regarding charter status. All charters receive

automatic reviews after 5 and 10 years of operation and the

authorizer uses data from the PMF and compliance monitoring to

determine what schools receive additional site visits. PCSB has

created a comprehensive guide for charter renewal that walks school

operators through the process step by step. The guide clearly states

that renewal decisions are based on criteria defined in the charter law-

adherence to the terms charter contract, including evidence that the

school has met the academic performance goals included therein.

PCSB staff review the renewal application and make recommendations

to the Board that include well-reasoned and data rich arguments for or

against renewal. PCSB has considered 11 charter renewal applications

and approved 10. The school that was not renewed was rated a Tier 3

school and had clearly not met the academic outcomes defined in the

charter. Of the 10 schools that have been awarded renewal, nine of

them are rated in Tier 1 or Tier 2- indicating a strong alignment

between renewal decisions and quality indicators. The one Tier 3

school that was renewed is an alternative school that will be included

under an alternative set of PMF measures in the near future. The factthat PCSBs portfolio includes a number of nonstandard schools

highlights that the adoption of alternative PMF measures must remain

a top priority of the Board.


31/38


Autonomy

Do schools have the autonomy to which they are entitled?


Applied:Well-Developed

Summary Assessment

PCSB fundamentally understands its role as authorizer is tasked with affording its schools

the autonomy to which they are entitled while holding them accountable based on the law

and the terms of their contract. In interviews with the executive director and several

members of the Board, we heard that preserving charter school autonomy was among their

chief tasks. Overall, this balance is maintained across PCSBs authorizing practices. The D.C.

charter school law is clear in the autonomies afforded charter schools and the way in which

they may be held accountable. PCSB has tried to ensure that they are able to evolve in the

way that they are holding schools accountable while still acting in accordance with the law.

The chief test to this has been the introduction of the Performance Management Framework.

Though the authorizer went through several iterations and sought charter input and

feedback, they still face resistance to it and charges that it infringes on the schools

autonomy and right to be held accountable to the terms of its contract. PCSB is addressing

this in a way that appears fair and in accordance with the law by using the PMF as a guidewhile ensuring that decisions on current schools are based on the terms of their contract.

Looking forward, they are working to encourage new schools to adopt the PMF as their

academic achievement goals in their charter application and contract.

PCSB employs EpiCenter to manage compliance requirements for the schools. The system

allows them to collect required compliance items from schools in a way that is streamlined

and efficient. Some elements of their ongoing academic monitoring are very specific and can

tend toward a focus on inputs, including a Danielson-based evaluation of teacher practice,

however, we did not hear push back on this from the schools or other stakeholders. In fact,

many of the schools and stakeholder groups reported that PCSB largely respects the

autonomy of schools and that most appreciate some of the more involved feedback fromschool visits and reviews.

Recommended Actions

Continue to move toward output based accountability standards and decisions.

Continue engagement with stakeholders on PMF and ensure that its application is in

compliance with law and respectful of school autonomy.


32/38


Detailed Analysis

4.1 Autonomy

The authorizer defines and

respects the autonomies to which

the schools are entitled based on

statute, waiver, or authorizerpolicy. The authorizer does not

reduce school autonomy unless

there is a compelling reason to do

so.

Established:

Well-Developed

Applied:

Well-Developed

Analysis

The Charter Agreement signed by PCSB-authorized schools explicitly

states the autonomies afforded it by the D.C. charter school law.

Specifically, the agreement identifies the laws with which it must

comply as well as stating that all public charter schools are exemptfrom the regulations governing District of Columbia Public Schools.

The monitoring requirements set forth by PCSB are in line with the

autonomies afforded charter schools in D.C.

In addition to the autonomies outlined in the law and charter

agreement PCSB proactively seeks to minimize the compliance burden

on schools. Given the unique structure and status of the District of

Columbia there are numerous entities with jurisdiction over a

relatively small number of schools. PCSB has collaborated chiefly with

the Office of the State Superintendent of Education (OSSE) to improve

the efficiency of the monitoring and data collection. The two agencies

have worked to identify redundant or unnecessary compliance items

and to streamline the processes where possible. For instance,

previously, newly approved charter schools had to submit a lengthyapplication to OSSE for start-up funding. The two agencies have now

merged their processes, making it more efficient for applicants as well

as the agencies.


33/38


4.2 Educational Program


respects school autonomy over

the educational program.

Established:

Well-Developed

Applied:

Well-Developed

Analysis

PCSB appropriately respects a schools ultimate control over its

educational programs. The authorizer evaluates the programs largely

based on outputs and whether it is achieving the goals set forth in its

charter agreement. At this time, PCSB does not identify any priorities

for educational programs (ex. blended learning) at the time of

approval. The ongoing monitoring and evaluation of the schools also

respects the autonomy the schools have to execute their educational

program. In interviews with the executive director he identified some

instances in the past when Program Development Reviews (PDRs)

resulted in findings that were highly subjective and reflected the

biases and opinions of the observer, rather than an objective

evaluation of the quality of the program. He has sought to remedy this

by redesigning the evaluation process and reorganizing the staff.

Some schools, as well as the charter association, asserted that the

introduction of the PMF was encroaching on the autonomy over their

educational programs because it failed to account for mission-specific

goals and uniformly applied the same evaluations to all schools

regardless of the unique features of a particular educational program.

Although the PMF does not specifically include mission-specific

measures, schools are still evaluated by meeting the goals in theircontracts, which include mission-specific goals.

It appears that schools are allowed to make minor changes to their

programs without approval from PCSB; however, more substantial

changes require a review from the authorizer. In those instances

where the change warrants review, it is unclear what the standard for

approval is.


34/38


4.3 Financial Management


respects school autonomy over

financial operations.

Established:

Well-Developed

Applied:

Well-Developed

Analysis

PCSBs financial oversight is informed largely through the collection of

monthly or quarterly financial reports, audited financial statements,

annual budgets, and IRS Form 990s. Additionally, financial data is

evaluated using the authorizers CHARM framework which is geared

toward assessing the financial health of the organization. All of this

information is communicated thoroughly to the charter schools. The

processes used by PCSB for financial oversight are in line with their

authority and preserve the schools autonomy to make budget

decisions that they believe best support the mission and program of

the school. The financial policies also ensure that PCSB can

appropriately hold schools accountable.

Recently PCSB moved away from its GPA system for financial

oversight and introduced a new model called CHARM. PCSB is taking

steps to use the results of CHARM along with the PMF in their

comprehensive evaluations of schools. Finally, interviews with charter

school leaders yielded unanimously positive statements in support of

the financial oversight offered by the CFO and staff.


35/38


4.4 Conflicts of Interest

The authorizer operates free from

conflicts of interest.

Established:

Well-Developed

Applied:Well-Developed

Analysis

PCSB functions as an independent body at every level. Board

members are free of conflicts that could compromise their ability to

act independently on charter decisions. Although members are

appointed by the mayor and (newly) approved by Council, they

demonstrated a clear independence from any influence, perceived or

actual, by these groups.

There was no evidence that there were any structures in place within

the authorizer that created a financial incentive for keeping any

particular school open. There is an authorizer fee as prescribed by

law, though it is minimal.


36/38


4.5 Earned Autonomy

The authorizer periodically

reviews compliance requirements

and evaluates the potential to

increase school autonomy based

on flexibility in the law,

demonstrated school

performance, and other

considerations.

Established:

Partially Developed

Applied:


Analysis

Charter schools in D.C. have a high baseline of autonomy granted to

them in their charter agreements. This autonomy is understood and

respected by the authorizer, as evidenced through interviews with

staff, board members, and other stakeholder in addition to evidence in

their authorizing practice. We did not find, however, that there is a

clear or well-defined path to earned or increased autonomy. The law

does not appear to limit the differentiation of autonomy, though it

does not expressly permit it either.

There has been some movement toward earned autonomy with the

advent of the PMF. PCSB has started using a schools Tier Rating on

the PMF to determine the frequency with which it is evaluated, with

higher performing schools receiving less frequent evaluations.

Specifically, there is differentiation in monitoring for schools based on

school performance. For instance, high performing schools receive

QSRs at greater intervals, are afforded preference in enrollment

increase requests, and are required to submit quarterly vs. monthly

financials. Likewise, lower performing schools or schools with flags

identified through the data submission and special education policies

will trigger increased monitoring and site visits. All schools go through

a common process when requesting expansion or renewal.

The authorizer has only recently launched its differentiated monitoring

process, so it was not possible to fully evaluate the details of the plan,

what the standards are, and how evenly they are applied. PCSB

recognizes charter school autonomy has a key element of its work and

seems poised to move further in that direction, where appropriate.

PCSB also has strong will from schools and other stakeholders

encouraging them to move further in the direction of increased

autonomy.


37/38


Sources

Application Decision-Making

Charter School Application-2013

Sample Charter School Application

Interview Materials & Schedule

Application Evaluation Process-2012

Review & Evaluation TemplatesThree Year Record of Application Decisions

Monitoring Operations

Charter Agreement-Draft

Charter School Annual Reports (2011-2012)

Charter Amendment Handbook & Policy

Fiscal Policy Handbook

Renewal Guidelines Overview (2012-2013)

Authorizer Monitoring Progress Reports

Compliance Materials

Pre-Opening Site Visit ChecklistQualitative Site Review Protocol-2013

Template of SPED Desk Audit

PCS Audits (2010-2011)

Performance-Based Accountability

Charter School Performance Reports-2012

Performance Management Framework-2011

Case Histories

Charter Applications

Charter AgreementsCharter Amendment Requests (2004; 2006;

2012)

Charter Renewal Applications

Program Development Review Reports

(2001-2011)

Qualitative Site Report-2013

Audits & Annual Reports (2010-2012)

Performance Analysis (2008-2009)

5th Year Charter Reviews

Study Review Report

Background

PCSB School Leader Survey Results

PCSB Self-Evaluation Survey Results

Authorizer Data Summary Template

D.C. Charter School Law

Organizational Chart-2012

Organizational Change Narrative


38/38

Biographies

Justin Testerman leads the implementation efforts of the Tennessee Charter School

Incubator as the chief operating officer, driving strategy and communicating with staff,

fellows, and partners to create an environment conducive to charter school growth and

success in Tennessee. Prior to coming to Nashville, Testerman served as the director of

education programs for Volunteers of America in Minnesota. He has long been involved on

the national charter school scene as a board member of the National Association of Charter

School Authorizers and as a charter school specialist in the Minnesota Department of

Education. Testerman began his career in education as a middle school teacher in Newark,

N.J. through Teach For America.

Carly Bolger is a consultant focused on supporting districts, state departments of education,

and other agencies to design and implement high-quality, strategic authorizing practices.

She has seven years of authorizing experience at both the district and state level. Carly was

part of the School District of Philadelphia's efforts to launch its Renaissance Schools

initiative, a process that selected high-performing charter operators to manage the district's

lowest performing schools. She has also headed the authorizing office for the state of New

Jersey and the Office of New Schools at Chicago Public Schools. Carly received a B.A. in

economics from Miami University and a Master in Government Administration from the

University of Pennsylvania.

nacsa authorizer evaluation report

Documents