1

Department of Veterans Affairs Access Audit

System-Wide Review of Access

Results of Access Audit Conducted May 12, 2014, through June 3, 2014

2

OVERVIEW

These Access Audit findings address the Secretary of The

Department of Veterans Affairs directive that the Veterans

Health Administration conduct a system-wide audit of

scheduling and access management practices. This audit

assesses the integrity of these practices and recommends

next steps to improve service to Veterans.

3

Access Audit Results At the direction of the Secretary of the Department of Veterans Affairs (VA), the Veterans Health Administration (VHA) conducted an Access Audit to determine if allegations about inappropriate scheduling practices are isolated instances of improper practices or if broader, more systemic problems exist. The audit was designed to:

1. Gauge front-line staff understanding of proper scheduling processes; 2. Assess the frequency and pervasiveness of both desired and undesirable

practices employed to record Veteran preferences for appointment dates, manage waiting lists, and process requests for specialty consultation; and

3. Identify factors that interfere with schedulers ability to facilitate timely care for Veterans.

The Access Audit was by necessity a rapidly deployed, system-wide assessment of scheduling practices across VA, and was not intended as a formal investigation of individual staff or managers. Site survey teams were not able to interview all employees, and time did not allow assessment of intent or potential culpability. All of the information collected from audit site visits has been shared with VAs Office of Inspector General (OIG). Audit Scope The audit was conducted in two phases. Phase One covered VA medical centers (VAMC) and large Community-Based Outpatient Clinics (CBOC) serving at least 10,000 Veterans. Phase Two covered additional VA facilities, including Hawaii VA and Phoenix VA Health Care Systems. Combined, the two phases covered 731 total facilities, including 140 parent facilities and all VAMCs. During the course of the audit, over 3,772 staff were interviewed. Audit Findings The Phase One findings were a strong basis to commence immediate action, even while Phase Two data were being collected. Ultimately, VA chose to limit Phase Two data collection after initial assessments restated high consistency with the findings of Phase One. The Access Audit was subject to certain limitations (emphasized in later sections of this report) that were unavoidable given the scope and accelerated timeframe. Notwithstanding these limitations, findings include:

1. Efforts to meet needs of Veterans (and clinicians) led to an overly complicated scheduling process that resulted in high potential to create confusion among scheduling clerks and front-line supervisors.

2. Meeting a 14-day wait-time performance target for new appointments was simply not attainable given the ongoing challenge of finding sufficient provider slots to

4

accommodate a growing demand for services. Imposing this expectation on the field before ascertaining the resources required and its ensuing broad promulgation represent an organizational leadership failure.

3. The concept of desired date is a scheduling practice unique to VA, and difficult to reconcile against more accepted practices such as negotiating a specific appointment date based on provider availability, or using a return to clinic interval requested by providers.

4. Overall, 13 percent of scheduling staff interviewed indicated they received instruction (from supervisors or others) to enter in the desired date field a date different from the date the Veteran had requested. At least one instance of such practices was identified in 76 percent of VA facilities. In certain instances this may be appropriate (e.g., a provider-directed date can, under VA policy, override a date specified by a patient), but the survey did not distinguish this, nor did it determine whether this was done through lack of understanding or malintent unless it was clearly apparent.

5. Eight percent of scheduling staff indicated they used alternatives to the Electronic Wait List (EWL) or Veterans Health Information Systems and Technology Architecture (VistA) package. At least one of such instance was identified in 70 percent of facilities. As with desired date practices, we did not probe the extent to which some of these alternatives might have been justified under VA policy. The questionnaire employed did not isolate appropriate uses of external lists.

6. Findings indicate that in some cases, pressures were placed on schedulers to utilize inappropriate practices in order to make waiting times (based on desired date, and the waiting lists), appear more favorable. Such practices are sufficiently pervasive to require VA re-examine its entire performance management system and, in particular, whether current measures and targets for access are realistic or sufficient.

7. Staffing challenges were identified in small CBOCs, especially where there were small counts of providers or administrative support.

Obstacles to Timely Access Critical insights came from asking front-line staff members to rate the degree to which certain factors interfered with timely access to care. The highest scored single barrier or challenge was lack of provider slots, closely followed by the peculiarities of the fourteen day goal1. Limited clerical staffing was also deemed a significant obstacle. Obstacles that have been posited as significant inhibitors to scheduling timely appointments, such as inadequate training of schedulers, or the inflexibility of the legacy VistA scheduling software system, were cited much 1 For example, a Veteran might have been entirely satisfied with a negotiated appointment time the following

month, but that could have been viewed as failing the 14 day standard.

Average (Mean) Scores of Barriers and Challenges to Providing Timely Access to Care (Higher number means greater challenge to staff members)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training forAdministrative orScheduling Staff

Using the VistAScheduling

System

Lack of ProviderSlots to Offer

Veterans

The 14 DayStandard

Scheduler staffing

2.0 2.0

3.02.8 2.8

Mean Score

5

less often during this audit. We also highlight that there were many potential opportunities identified that could improve the consistency of desirable practices among schedulers, such as calling Veterans about upcoming appointments, addressing other obstacles, making performance improvement activities more routine, and ensuring that clinic operations data are regularly reviewed at team and management meetings. Further Actions VA will establish follow-up accountability actions based on the results of the audit. Senior leaders will be held accountable to implement policy, process, and performance management recommendations stemming from this audit and other reviews. Where audited sites identify concerns within the parent facility or its affiliated clinics, the VA will trigger administrative procedures to ascertain the appropriate follow-on actions for specific individuals. Based on the findings of the audit, VA will critically review its performance management, education, and communication systems to determine how performance goals were conveyed across the chain of command such that some front-line, middle, and senior managers felt compelled to manipulate VAs scheduling processes. This behavior runs counter to VAs core values; the overarching environment and culture which allowed this state of practice to take root must be confronted head-on if VA is to evolve to be more capable of adjusting systems, leadership, and resources to meet the needs of Veterans and families. It must also be confronted in order to regain the trust of the Veterans that VA serves.

6

I. ACCESS AUDIT RESULTS The following section provides details about the conduct of and results from the VA nationwide audit of scheduling and access management practices.

7

1. Research Question(s) The essential discovery questions built into the audit are listed below:

Do front-line staff receive appropriate training, supervision, and feedback to correctly perform their scheduling and access management practices?

Do front-line staff members exhibit the proper understanding of scheduling and access management policies and practices?

Do front-line staff members receive instruction to modify dates when a Veteran wants to be seen, and if so, how and why do they receive that instruction?

What are the main barriers and challenges staff members face in offering Veterans timely access to care? Do they feel personally capable of delivering high-quality service?

2. Data Collection Method(s) The audit management team assessed various techniques to support the collection of data. It was determined that, given the sensitivity of information to be collected, that face-to-face interviews would need to be collected by independent site audit teams using confidential, in-person administered questionnaires. 2.1 Site Audit Teams Phase One site audit teams were comprised of four senior field and headquarters staff members. These staff members were typically General Schedule (GS)-14, GS-15, Senior Executive Service (SES), and SES Equivalent. Phase Two site audit teams were comprised of two field and headquarters staff members. These staff members were typically at the GS-14 and above level. Staff members selected were senior leaders in the organization familiar with conducting audits and site visits, e.g., administrative investigations where sworn testimonies are collected; consultative site visits based on defined technical criteria. Further, these staff members would carry authority and stature sufficient to ensure access to key staff members in the field and independence in performing their functions. A final rationale behind the selection of these leaders was to create a shared awareness and learning of scheduling and access management practices across the system. To ensure independence, no member of any site audit team either worked in the facility being audited, the Veterans Integrated Service Network (VISN) overseeing that facility, or any other facility in same VISN as the facility being audited. Over 205 staff supported Phase One, including staff that assisted in data management and analysis. Over 264 staff supported Phase Two, including staff that assisted in data management and analysis.

8

2.2 Dates of Site Audits Audits commenced May 12, 2014, and concluded June 3, 2014.

2.3 Sites Visited

Site audit teams visited 731 total facilities, including 140 parent facilities and all VAMCs The list of sites visited is contained in Appendix B. Each site audit was initiated with a joint in-briefing to local union leadership and facility management. During that in-briefing, the list of requested interviewees was identified. 2.4 Questionnaires Site audit teams were responsible for collecting data from front-line staff through the use of detailed questionnaires. Additionally, site teams generated nightly reports summarizing their site audits and identifying any issues of concern. An example of the questionnaire is contained in Appendix E. 2.5 Sampling Method The audit management team created a listing of all 30,000 VA employees with access to the VistA scheduling package who had actually performed scheduling functions. This included mainly front-line scheduling staff (typically a position called a medical scheduling assistant) and clinic managers (typically a nurse clinic manager) who were all eligible to be interviewed by the site audit teams. From the list of eligible schedulers at the site they were visiting, the audit teams selected their respondents on the day of their site audit, not announcing their selections until the in-briefing. In order to complete all data collection as quickly as possible, no more than 1 business day was feasible at each site. A total of 10 respondents were selected per facility comprised of 9 front-line schedulers and one clinic manager. Overall, each site audit team was expected to interview:

The Chief of Health Administration Service, Medical Administration Service, or Chief Business Officer

Nine interviews with schedulers, including: Between three and four with primary care schedulers; Between two and three with mental health schedulers; Between two and three with specialty care schedulers; and One clinic manager.

9

2.6 Interviews Completed In total, 3,772 interviews were conducted with VAMC and CBOC staff members. Interviews were conducted in private and no names of interviewees were recorded with questionnaire data. Each interview was allotted a full-hour but lasted approximately 45 minutes. Staff members interviewed by the site audit teams were informed that if they wanted union representation, this was both permitted and encouraged. Further, if potential interviewees were uncomfortable with being interviewed, they were informed they would not be required to complete the interview. 2.7 Site Reports Submitted During the site audits conducted, 596 site audit summary reports were submitted by the site audit teams. These summary reports rolled-up to 140 parent VAMCs (facilities). 3. Audit Limitations VA undertook an ambitious effort to schedule, prepare, coordinate, train and deploy over 400 staff across the Nation over 5 days. This accelerated effort led, unavoidably, to a number of limitations, which serve to caution against over-interpretation of these findings, including:

Design of the survey which was intended to provide a very low threshold (i.e., high sensitivity) for eliciting potentially improper scheduling practices.

o VA intentionally designed the survey to be sensitive to non-conforming scheduling policies. As such, the results will group misunderstanding of proper scheduling methodology together with intentional instruction to report alternate waiting times. The sensitivity in the instrument enables VA to identify a broader set of sites with potentially problematic practices.

The Audit Survey tool itself did not undergo pre-testing to ensure all respondents would understand the intent of each item.

o Certain items on the questionnaire may have been misunderstood.

Individual questions were not worded to ascertain the reason that policy may have been violated.

o Therefore, findings from this audit cannot be extended to identify deliberate deception, fraud, or malfeasance.

The scope of the audit precluded independent verification of any narrative statements, though all data collected throughout the Access Audit have been shared with VAs OIG.

o Furthermore, the audit did identify sites necessitating more intensive management investigations. VHA will ensure that accountability for inappropriate practices is pursued through further investigations to substantiate initial findings. In pursuing accountability, VHA will follow

10

statutory and regulatory due process requirements accorded to all Federal government employees.

Site audit teams had limited time (90 minutes of pre-survey coaching plus additional document review) for training.

o While site teams were generally knowledgeable about audits, investigations, and consultative visits, not all were experts in all the complexities associated with scheduling and access management.

Sampling of staff was based on availability. o Staff selected for interviews may not have been available to complete the

requested interview. In these cases, the site audit team selected another candidate.

Treatment of respondents prior to interview o In certain instances staff selected for interviews had experienced recent

training (e.g., within days of the requested interview). This treatment may have altered results, affecting baseline assessments of understanding of scheduling policies and practices.

Limited validation of responses o Survey science includes methodology for internal validation to ensure

consistency of responses. This is limited in the audit and where included does not support a high correlation (see 5.1 of this audit results for details).

4. Site Visit Reports Overall Impressions

4.1 Site Visit Reports Overall Impressions

By May 17, 2014, 596 close-out reports had been submitted by site audit teams. Of these, 229 (38.4 percent) indicated Concerns (they) wished to report to the National Stand Down Team. Out of the 229 site reports indicating some degree of concern, 112 (or roughly 19 percent of all reports) were flagged because of concerns that indicated undesired scheduling practices or because detailed responses by interviewed staff indicated they had received instruction to modify scheduling dates (or similar concerns). This listing of sites requiring further review is based on a review of responses by front-line staff contained in site audit reports. VA is providing the list of sites requiring further review to OIG for further investigation; however, the listing of these sites should be understood as a preliminary step, and further review will be necessary to determine the extent of issues related to scheduling and access management practices. CBOC sites of concern in Phase Two of the audit tended to correlate with parent facilities. Negative practices identified in site reports included:

Staff being instructed by supervisors to alter desired date;

11

Staff keeping manual logs of appointment requests outside of electronic systems (VistA or the EWL);

Staff lacking familiarity with scheduling policies;

Other practices inconsistent with policy: o Non-count clinics ; o Cancelling consults; o Cancelling appointments; and

Employees indicating reluctance to participate in the survey due to fear they would be subject to disciplinary action due to deviation from national policy.

12

4.2 Staff Questionnaire Responses from Site Audits

What follows are summaries, both quantitative and qualitative, for each question in the survey. The questions are grouped by theme and are not necessarily presented in the order in which they were administered during the survey visit.

Because the number of respondents at an individual site is low, we are unable to make statistically valid distinctions in performance at site level, and present national summary data in order to answer the question are problems isolated or pervasive.

For Yes/No questions, we present the following information:

Total responses received;

Percent of respondents indicating Yes; Percent of facilities in which AT LEAST 1 RESPONDENT indicated a Yes; Percent of facilities in which AT LEAST ONE QUARTER of RESPONDENTS

indicated a Yes; Percent of facilities in which AT LEAST THREE QUARTERS of

RESPONDENTS indicated a Yes; This approach was adopted in order to convey the frequency and

pervasiveness, of various scheduling practices, both desirable and undesirable, across VAs health care system. For instance, it is possible that an undesirable practice such as using alternatives to VistA or the EWL is endorsed by a small proportion of schedulers (low prevalence) but is found at least once in a high percentage of VA facilities (pervasive across the system). The consistency of either desirable or undesirable practices within a given facility can be inferred by comparing values for AT LEAST 1, AT LEAST ONE QUARTER, and AT LEAST THREE QUARTERS of RESPONDENTS;

For questions where responses were scored on a 5-point Likert scale, we provide a Mean Score (overall) as an estimate of prevalence. Pervasiveness is indicated by noting the Percent of facilities where the MEAN among respondents is HIGH (score of 3 or better among all those responding from a single site) or LOW (mean less than 3 among all respondents at a site); and

Questions relating to undesirable practices are highlighted in yellow in the tables below.

5. Scheduling Practices

For purposes of establishing the percentage of responses, further analysis is aggregated at the parent facility level. The sites and locations visited roll-up to 140 parent facilities (e.g., one hospital has numerous outpatient clinics).

13

5.1 Staff Understanding about Scheduling Policy The figure provides a summary of responses about staff members understanding of scheduling practices.

* indicates mandatory questions Responses

% "Yes"

No.(%) Facilities Where >=

one Response

"Yes"

No.(%) Facilities

Where >25%

Responses "Yes"

No.(%) Facilities

Where >75% of

Responses "Yes"

Q6. * Based on the response above, does the clerk report the correct procedure for determining desired date?

3,208 78 140 (100%) 138 (99%) 92 (66%)

Q8. * Based on the above, does the clerk report the correct procedure for determining the desired date?

3,208 75 140 (100%) 139 (99%) 84 (60%)

Q10. * Based on the response above, does the scheduler report correct use of the Electronic Wait List (EWL)?

3,208 49 140 (100%) 115 (82%) 16 (11%)

Q11. * Do you track appointment requests in places other than the VISTA scheduling system or EWL?

3,208 8 108 (77%) 5 (4%) 0(0%)

Q39. * Do you track appointment requests in places other than the VISTA scheduling system or EWL?

3,208 7 98 (70 %) 4 (3 %) 0 (0 %)

Q41. * Are you aware of the (New Enrollee Appointment

3,208 31 140 (100 %) 78 (96 %) 1 (1 %)

14

* indicates mandatory questions Responses

% "Yes"

No.(%) Facilities Where >=

one Response

"Yes"

No.(%) Facilities

Where >25%

Responses "Yes"

No.(%) Facilities

Where >75% of

Responses "Yes"

Request) list?

Q45. * Are you aware of any consults that are used specifically to request that appointments be scheduled?2

3,208 57 139 (99%) 126 (90%) 34 (24%)

Scheduling staff were twice asked the question do you track appointment requests in places other than the VistA scheduling system or EWL, spaced by several minutes, in order to allow an internal check of survey reliability. Although the summary values for each question are very similar, the calculated agreement in this instance is only 0.53 using the Kappa score (a statistical measure of inter-rater agreement), which is quite low as compared to modern survey standards, but understandable in light of the rapid deployment of this national initiative (see Limitations above). While any individual item on the survey might be suboptimal, several related survey items and their accompanying verbal comments allows a more confident estimate of the prevalence of problematic scheduling practices. For instance, a question asked of clinic managers provided responses that closely paralleled the responses of scheduling staff:

Responses

Within 1 day

Within 3 days

Within 7 days

More than 7 days

Don't know

Q46. If you are aware of any consults that are used specifically to request that appointments be scheduled, how often are they processed? (PERCENT)

3,208 57 139 (99%) 126 (90%)

34 (24%) 3,208

2 CPRS contains the ability to generate a consult request for appointments. Staff should be aware of these

requests in order to accurately gauge Veterans access needs.

15

5.2 Facility Instruction to Record Dates Other than Dates Veteran Wants to be Seen Thirteen percent of schedulers indicated they received instruction to enter a desired date other than the date a Veteran asked to be seen. In 76 percent of VA parent facilities, at least one respondent indicated that she or he received instruction to modify the date when a Veteran wanted to be seen. However, it must be noted that this includes instances where one respondent in one CBOC associated with the parent facility indicated she/he received instruction to modify dates. The practice was not seen consistently among scheduling staff within a given facility - only 15 percent of facilities, for instance, had over 25 percent of respondents indicating yes, and in no facility did over three-quarters of respondents indicate such a practice. The reasons for entering a Desired Date different from the one voiced by the Veteran varied greatly.

* indicates mandatory questions Responses

% "Yes"

No.(%) Facilities Where >=

one Response

"Yes"

No.(%) Facilities

Where >25%

Responses "Yes"

No.(%) Facilities

Where >75% of

Responses "Yes"

Q12. Do you feel you receive instruction from the facility to enter a desired date other than the date a Veteran asks to be seen?

3,036 13 107 (76%) 20 (14%) 0 (0%)

5.3 Qualitative Probe Questions Initial qualitative analysis yielded several themes. The audits included 3,722 participant

interviews, of which 2,218 were non-clinicians. Initial qualitative analysis yielded

themes similar to those in Phase I.

Extent of scheduling practices that were not concordant with policy A minority of respondents (188 or 8% from 119 clinics, 29%) reported that they had previously selected desired date from among available dates, rather than determining desired date based upon the Veterans preferred date or the date requested by the provider. A majority of respondents (1,575 or 71% from 346 clinic sites, 85%) reported that they had previously presented the Veteran with available dates from which to select a desired date. Many respondents reported that this scheduling policy (i.e., selecting the desired date irrespective of available dates) was not well-understood previously, and that they began

16

recording desired date correctly after receiving recent additional training. Some respondents specifically cited a web-based training session in VAs online Talent Management System (TMS) as helping to provide the clarification that was needed to do this correctly.

17

Reports of potentially fraudulent practices Respondents at 90 clinic sites provided responses indicating they had altered desired dates that had been entered. In virtually all cases, they indicated they were instructed by supervisors, but many believed the policy of altering dates was coming from facility leadership. In at least 2 clinics, respondents believed someone else (not a scheduler) was routinely accessing records and changing desired dates in order to improve performance measures. In 24 sites, respondents reported that they felt threatened or coerced to enter specific desired dates. Respondents at 14 sites reported having been sanctioned or punished over scheduling practices. Respondents at 2 sites reported having been sanctioned (written up) for either not complying with supervisors orders to inappropriately enter or alter recorded desired dates, or for expressing concerns over what they were being asked to do

A number of respondents presented detailed descriptions of instructions from supervisors to change or alter data in order to affect reported wait times. The descriptions reflect a perception of the practice as both widespread and overt.

Although uncommon, several respondents expressed concerns with or related

reports of punitive actions related to a demand to manage reported wait times.

Work demands and difficulty keeping up were described as exacerbating

factors in problematic scheduling and wait list management.

Factors Contributing to Inappropriate Scheduling Related Activities When explaining the context of inappropriate scheduling activities respondents described a numbers driven system with unrealistic performance measures as having created a highly stressful work environment that limits the focus on serving the Veteran.

6. Findings : Barriers and Challenges to Providing Veterans Improved Access

to Care

6.1 Main Challenges This section provides summary results regarding primary challenges and obstacles staff face in offering Veterans timely access to care. When front-line staff members were requested to score from a range of 1 (never a barrier or challenge) to 5 (always a barrier or challenge) the degree to which various aspects posed barriers and/or challenges to providing Veterans with timely access to care, the highest scored barrier or challenge was the lack of provider slots to offer Veterans, closely followed by limited clerical staffing and the fourteen day standard. These previous barriers and challenges far outscored training and the legacy scheduling software system.

18

Challenges and Obstacles to Providing Timely

Access to Care (Score 1 = never, 2 = rarely, 3 =

sometimes, 4 = often, and 5 = always)

Responses Mean Score

No. Facilities with mean rating 3 or

greater

No. Facilities with mean rating less

than 3

Q14. Rate how often "Training for Administrative or Scheduling Staff like Me" presents challenges

2,719 2.0 1 137

Q15. Rate how often using the VISTA Scheduling System presents challenges

2,834 2.0 5 134

Q16. Rate how often Lack of Provider Slots to Offer Veterans presents challenges

2,867 3.0 72 67

Q17. Rate how often the 14 Day Standard presents challenges

2,856 2.8 45 94

Q20. Rate how often other obstacles present challenges

1,416 3.5 122 18

Q26. How often is scheduler staffing an issue?

2,876 2.8 44 96

The most common other obstacles identified by schedulers in responses are listed below:

Staffing problems were frequently cited as an other obstacle to scheduling

Veterans and were reported at many sites. Respondents commonly reported

difficulties and distress related to being understaffed, both in terms of scheduling

staff and providers.

Respondents attributed staffing problems to turnover, difficulty hiring and, in some

cases, salary.

The challenges of hiring were described as most problematic for providers.

Although far less frequently reported than HR issues, low morale was described as

both a result of, and cause of low staffing.

Scheduling software and, to a lesser degree, telephone equipment were frequently

described as antiquated and problematic.

Some respondents identified training issues, both the lack of customer service

training and the time training pulls staff away from customer service tasks, as a

problem for scheduling.

19

Scheduling policies were described as sometimes unclear and frequently

problematic. The most prevalent issues cited in this area were handling walk-ins

and phone calls.

A less commonly reported obstacle was changes in care delivery model, specifically

Patient Aligned Care Teams (PACT). However, other respondent perceived PACT as a

potential solution to access problems.

Another less common concern was that the lack of facilities as an obstacle for patient

access, both at the individual clinic and system level.

Additional data regarding customer service, training, supervisory oversight, and scheduling improvement activities are provided in Appendix D. It is noteworthy that VA is engaging in the opportunities (highlighted in yellow in Appendix D) to increase the consistency of desirable practices within VA facilities such as:

Calling Veterans who have missed prior appointments (missed opportunities) to remind them of upcoming appointments

Involving schedulers regularly in performance improvement activities

Reviewing clinic operations data regularly in team meetings

20

II. STRUCTURE AND PROCESS CHANGES FOR MANAGING ACCESS

TIMELINESS TO BE IMPLEMENTED IMMEDIATELY

The following section provides systematic and broad reaching actions that VA has already initiated.

21

1. Overview VA and VHA will perform thorough analyses, assess root causes and develop a long term plan to ensure that VHAs system of care employs valid assessments of demand, capacity and productivity. Notwithstanding this need for deliberative development of longer-range efforts, VHA can and must address evident and solvable problems immediately. Paramount in this immediate set of actions is to identify where Veterans are waiting for care and ensure that the have access to quality, timely care. VHA will make rapid and definitive changes to ensure integrity in managing Veterans access to care so the agency can maintain its focus on providing Veterans timely access. Additionally, VHA will provide hands-on attention to all staff engaged in providing Veterans health care and managing access. The actions in the sections to follow will be coordinated through a near term plan that commenced May 23, 2014.

Accelerate Care for Veterans Currently Waiting for Care Assess Care Delivery Capacity vs. Health Care Demand to Ensure Resource Levels

Remove 14-Day Performance Goal from Performance Contracts

Revise and/or Rescind Scheduling Directive

Suspend VHA Executive Performance Awards for FY14

Face-to-Face Engagement with Medical Support Assistants, Clinic Managers and Other Critical Front-line Staff

Communicate VA Values Applicability to Day-to-Day Performance Review and Modify Performance Plans for Wait Time Accountabilities

Modify Management Dashboards Designed for Organizational and Operational Levels

Enhance Patient Satisfaction Monitoring to Assess Satisfaction with Access and Experience

Implement Medical Center Access Audits, Ongoing Monitoring, Elevation Triggers, and Clear Line Accountability Including Specific Requirements for Regular Inspection and Reporting

Enhance VHA National Program with Focus on Access to Care

Implement VHA-Wide Site Inspection Process

Cross-Organization Surveying of Scheduling and Access Best Practices

Review Medical Support Assistant Classification to Ensure Correct Grading

Revise, Enhance and Deploy Scheduling Training

Assess Position Management Practices and Staffing Required to Fully Support VA Medical Centers

Establish Wait-Time Based Guidance for Non-VA Care Referral

Assess Implementation of System-Wide Contracts for Primary Care

22

2. Accelerating Care and Ongoing Capacity Assessments In the immediate term, VHA will accelerate care for Veterans experiencing delays in receiving their care. Additionally, VHA will develop and deploy quantitative mechanisms to assess demand, capacity and delays. VHA will supplement the considerable range of productivity measures that have already been deployed and will use these measures to assess the adequacy of resources for the provision of timely access to care for Veterans. Accelerate Care for Veterans Currently Waiting for Care Commencing May 23, 2014, VHA deployed the expanded Accelerating Access to Care Initiative. This initiative has identified roughly 100,000 Veterans who are currently experiencing long wait times for receipt of their VA health care. VHA has provided training to VHA, VISN and facility staff to implement this plan. On the first day of the Accelerating Care initiative, VHA provided training to over 900 VHA field staff. Specifically, for Primary Care, Mental Health, and Specialty Care, VHA is assessing each of its clinics using productivity data to determine if greater productivity can be gained (e.g. for clinics with lower productivity). Additionally, each VHA medical center has assessed if it can provide expanded clinic hours to increase clinic capacity. Lastly, each VHA medical center is assessing if care is available through non-VA care or through the national, Patient Centered Care in the Community (PC3) contract.

These immediate tactics to accelerate care rely heavily on financial and other resources (e.g. overtime, etc.). Each VHA medical has assessed mechanisms to increase productivity, capacity or care in the community, and then each medical center is contacting Veterans directly to accelerate their care. As additional resources are required to accelerate access, these requests are being provided to VHA corporate office to identify available resources.

VHA will track progress through online corrective plans from each facility that address productivity enhancements, capacity increases, non-VA care acquisitions, Veteran contacts, and, ultimately, reduction in the number of Veterans waiting 30 days or more for health care services.

23

Appendix A: Analytic Methods

A.1 Coding of Quantitative, Ordinal and Categorical Variables Numerical and categorical responses (i.e., Yes/No/Do not Know, Rating Scale 1-5, Time Scale [e.g. Daily/Monthly/Quarterly/etc.] to questionnaire items were provided by 2,290 individuals who were interviewed by the site visit teams. These items included Q6, Q8, Q10, Q11, Q12, Q14, Q15, Q16, Q17, Q18, Q20, Q21, Q22, Q24, Q25, Q26, Q29, Q30, Q31, Q32, Q35, Q37, Q39, Q41, Q42, Q45, Q46, Q51, Q53, Q55, Q56, Q57, and Q58. Items that provided numerical responses included Q44, Q48, and Q52. Responses to these items were summarized at the level of National VA, VISN and parent facility.

Several steps were taken to prepare the data and conduct analysis:

A copy of the excel data set was created with question number defined as column names to replace the question text column headers. Data from the excel file were then entered into SAS.

A facility crosswalk was created to address the multiple VISN assignments issue stated below and to determine the parent facility station number for analysis purposes.

A reference table with distinct combinations of responses to Q1, Q2A and Q2B was created. Station numbers were manually assigned based on responses to Q2A and Q2B. The table was merged with another table containing information from VHA Site Tracking (VAST) system to obtain VISN, parent facility name, number, city and state. Issues that were addressed:

Responses from the same location of site visit (survey question 2a) with 2 different VISN responses (survey question 1).

Survey responses to question 2b : o The station numbers entered did not always match the facility/clinic name o Station numbers entered may have had transposed characters o Missing station numbers o Spelling variation in facility/clinic names

The facility crosswalk was systematically and manually validated.

Numeric responses of 999, 9999, and >= 99999 to questions 44, 48, and 52 were assumed to imply Do Not Know. Do not know responses were counted but were excluded from the calculation of the summary statistics for each question.

All free-text responses were grouped to a response category=OTHER for questions with defined categorical responses and Other (Fill-in) response option. This applied to questions 20, 22, 30, 37, and 57.

24

A.2 Qualitative Responses Several items permitted verbal explanation to accompany a categorical (i.e., yes/no) response. An initial qualitative analysis comments was performed to provide a fuller understanding of the scope and nature of problematic activities pertaining to patient scheduling and electronic waitlists and to complement the quantitative analysis of categorical response items. We analyzed the unstructured interviews to assess the extent to which scheduling procedures may not have followed national policy. Narrative data recorded by site visit teams from interviews with 1,962, non-clinician respondents (clinicians didnt provide narrative responses) representing 260 clinic sites were coded. Analysis A team of 14 coders reviewed narrative survey fields and coded for 7 scheduling practices: i. Using the date patient wants to be seen as the desired date ii. Using the date the provider orders as the desired date iii. Routinely entering only an available date as desired date (no input from Vet) iv. Routinely entering only an available date as desired date in negotiation with the

Veteran v. Changing the desired date after it has been entered (i.e., altering data for a non-

clinical reason) vi. Threats or coercion to follow scheduling practice vii. Punishment or retribution for scheduling practice or voicing concerns For respondents that were coded positive for v, vi or vii, an observation we independently coded by one of two coders, who then reviewed findings and agreed on the final set of coded data. For findings on obstacles to appropriate scheduling, one coder reviewed narrative data from a site visit item on obstacles to appropriate scheduling, and compiled a list of the most common major obstacles. Limitations These were site visit data, not a respondent survey. The site visit teams for the Phase Two visits had less training and guidance due to logistical constraints and so potentially may exhibit more variability in data collection. The data were filtered through the members of the site audit teams and were generally not verbatim quotations. Thus, many responses likely lack much of the context provided by the respondents during the visit. Because respondents were generally not expressly asked about the specific behaviors enumerated in this report (e.g., whether they felt coerced or threatened), the absence of the behavior being reported cannot be considered to be equivalent to the respondent reporting the behavior was absent. Moreover, respondents may have self-censored, or may have responded to questions in order to be helpful even when they had incomplete information (helpful respondent bias). For reports of potentially fraudulent practices, which have the most significant

25

implications for the personnel involved, in most cases the reports are from a single individual at the clinic setting. Because of the limited time available, an initial analysis was performed only on question #12 (Do you feel you receive instruction from the facility to enter a desired date other than the date a Veteran asks to be seen?) for which there were 2,086 categorical responses, for which additional explanation was provided by 270 respondents (12.9%) (coded as Question #13). Data preparation occurred in several steps:

A copy of the excel data set was created for qualitative data analysis to ensure integrity of the original data. Purposive sampling was used to identify respondents who described problematic activities pertaining to patient scheduling and electronic waitlists (EWL) using Yes to question 12; A separate excel spreadsheet was created that contained all of the survey data fields for the identified subset (Q12-Yes). Dichotomous and numerical fields were retained but hidden for ease of coding.

This excel data set was analyzed using an inductive content analysis approach to identify emergent categories. Coding was done within Excel by adding columns for each identified category and pasting relevant quotes within the corresponding category. Operational definitions were developed and included as a footer in the spreadsheet. The initial set of codes included 10 (Details re. Q 12, Scheduling practice descriptions, Training, Instructed, Perceived Deviation from protocol, Concern re. lying, Acknowledged lying, Reading between the lines, Saving emails, Expectations and Staff Empowered).

26

Appendix B: Sites Visited by Day of Access Audit B.1 Phase I Sites Visited

Trip # May 12 May 13 May 14 May 15 May 16

1 Manchester, NH

White River Junction, VT

Togus (Augusta), ME

Bedford, MA Boston (Jamaica), MA

2 Boston (West Roxbury), MA

Boston (Brockton), MA

Newington, CT Providence, RI None

3 Syracuse, NY Bath, NY Canandaigua, NY

Rochester, NY Buffalo, NY

4 Albany, NY HV HCS Montrose, NY

HV HCS Castle Point, NY

NYH HCS Bronx, NY

NYH HCS St Albans, NY

5 NYH HCS Brooklyn, NY

NYH HCS, Manhattan

Northport, NY NJHCS East Orange, NJ

Brick, NJ

6 Erie, PA Butler, PA Pittsburgh (HD), PA

Pittsburgh (Uptn), PA

Altoona, PA

7 NJHCS Lyons, NJ

Wilkes-Barre, PA

Allentown, PA Lebanon, PA Horsham, PA

8 Clarksburg, WV

Martinsburg, WV

Washington, DC Richmond, VA Salem, VA

9 Philadelphia, PA

Wilmington, DE

Baltimore, MD Perry Point, MD Loch Raven, MD

10 Asheville, NC Mountain Home, TN

Middle Tenn (York), TN

Middle Tenn (Main), TN

Memphis, TN

11 Raleigh, NC Durham, NC Winston-Salem, NC

Salisbury, NC Charlotte, NC

12 Fayetteville, NC

Myrtle Beach, SC

Goose Creek, SC

Charleston, SC Savannah, GA

13 Greenville, SC Columbia, SC Augusta, GA Dublin, GA Atlanta, GA

14 Birmingham, AL

Tuskegee, AL Montgomery, AL Pensacola, FL Mobile, AL

15 Tallahassee, FL

Lake City, FL Gainesville, FL Jacksonville, FL Ocala, FL

16 Daytona Beach, FL

The Villages, FL

Orlando (Old), FL

Orlando (New), FL

Lake Nona, FL

17 Tampa, FL New Port Richey, FL

Bay Pines, FL Bradenton, FL Sarasota, FL

18 Cape Charles, FL

Sunrise, FL Miami, FL West Palm Beach, FL

Viera, FL

19 Cleveland, OH Parma, OH Lorain, OH Canton, OH Youngstown, OH

20 Dayton, OH Toledo, OH Cincinnati, OH Chillicothe, OH Columbus, OH

27

Trip # May 12 May 13 May 14 May 15 May 16

21 Ft Wayne, IN Marion, IN Indianapolis, IN Danville, IN Peoria, IL

22 Saginaw, MI Battle Creek, MI

Ann Arbor, MI Toledo, OH Detroit, MI

23 Iron Mountain, WI

Appleton, WI Tomah, WI Madison, WI Milwaukee, WI

24 Chicago, IL Crown Point, IN

Hines, IL Lovell FHCC, IL None

25 Saint Louis, MO

Evansville, IL Marion, IN Popular Bluff, MO

Mount Vernon, MO

26 Kansas City, KS

Leavenworth, KS

Topeka, KS Columbia, MO Wichita, KS

27 Jackson, MS New Orleans, LA

Alexandria, LA Baton Rouge, LA

None

28 Shreveport, LA Little Rock, AK North Little Rock, AK

Beaumont, TX Houston, TX

29 Oklahoma City, OK

Tulsa, OK Muskogee, OK Ft Smith, OK Fayetteville, AK

30 Dallas, TX Bonham, TX Ft Worth, TX Waco, TX None

31 Harlingen, TX McAllen, TX Corpus Christie, TX

None None

32 San Antonio, TX

Temple, TX Austin, TX None None

33 San Diego, CA Mission Valley, CA

Prescott, AZ Mesa, AZ Phoenix, AZ

34 Amarillo, TX Big Spring, TX None None None

35 Tucson, AZ El Paso, TX Albuquerque, NM

None None

36 Portland (East), OR

Portland (Main), OR

Port Angles, WA Seattle, WA Tacoma, WA

37 Portland (West), OR

Eugene, OR Roseburg, OR White City, OR None

38 Cheyenne, WY Denver, CO Colorado Springs, CO

None None

39 Walla Walla, WA

Boise, ID None None None

40 Grand Junction, CO

Salt Lake City, UT

None None None

41 San Francisco, CA

Palo Alto, CA Livermore, CA San Jose, CA Fresno, CA

42 Redding, CA Reno, NV Sacramento, CA Martinez, CA None

43 Oceanside, CA Loma Linda, CA

Long Beach, CA West Los Angeles, CA

Sepulveda, CA

44 Las Vegas Las Vegas Las Vegas (NE), Las Vegas Las Vegas

28

Trip # May 12 May 13 May 14 May 15 May 16

(Main), NV (NW), NV NV (SW), NV (SE), NV

45 Ft Meade, SD Hot Springs, SD

None None None

46 Ft Harrison, MT Billings, MT Sheridan, MT None None

47 Grand Island, NE

Lincoln, NE Omaha, NE Des Moines, IA Iowa City, IA

48 Fargo, ND Sioux Falls, ND

St. Cloud, MN Minneapolis, MN

None

49 Brick, NJ East Orange, NJ

St. Albans None None

50 Ocala, FL Lake Nona, FL Viera, FL None None

51 None None None Anchorage, AK None

52 None None None None San Juan, PR

53 None None None None Honolulu, HI (Not Completed)

29

B.2 Phase Two Sites Visited (Label of Y indicates the site visit was completed).

Trip # Day 1

Day 2

Day 3

Day 4

Day 5 1 Williston, ND Minot, ND Dickinson, ND Y Bismarck, ND Y

2 Grafton, ND Y Grand Forks, ND Y Jamestown, ND Y

3 Bemidji, ND Y Hibbing, MN Y Ely, MN Y Superior, WI Y

4 Fergus Falls, MD Y Alexandria, MN Montevideo, MN Brainerd, MN Y

5 Faith, SD Y Pierre, SD Y Isabel, SD Y

6 Aberdeen, SD Y McLaughlin, SD Eagle Butte, SD Y Watertown, SD Y

7 Mission, SD Y Winner, SD Y Wagner, SD Y O'Neill, NE Y

8 Newcastle, WY Pine Ridge, SD Gordon, NE Rapid City, SD Y

9 Hayward, WI Y Rice Lake, WI Y Maplewood, WI Y Chippewa Valley, WI

Y Ramsey, MN Y

10 Mankato, MN St. James MN Spirit Lake, IA Rochester, MN Y Albert Lea, MN Y

11 Sioux City, IA Y Norfolk, NE Y Carroll, IA Y Fort Dodge, IA Y

12 Mason City, IA Y Decorah, IA Y Waterloo, IA Marshalltown, IA

Knoxville, IA

13 Dubuque, IA Cedar Rapids, IA Y Coralville, IA Y Bettendorf, IA

14 Sterling, IL Galesburg, IA Ottumwa, IA Quincy, IL

15 North Platte, NE Y Holdrege, NE Y Bellevue, NE Shenandoah, IA Y

16 Mount Vernon, WA

Y Bremerton, WA Y Bellevue, WA Y Lake City, WA Y Port Angeles, WA

Y

17 Chehalis, WA Y Warrenton, OR Y Federal Way, WA

Y West Linn, OR Y

18 The Dales, OR Y Yakima, WA Y Richland, WA Y Wenatchee, WA Y Boardman, OR Y

19 Newport, OR Y Salem, OR Y Bend, OR Y Burns, OR

20 No longer a trip

21 Kalispell, MT Y Libby, MT Y Coeur d'Alene, ID

Lewiston, ID

22 Enterprise, OR LaGrande, OR Y Caldwell, ID Y Mountain Home, ID

Y Twin Falls, ID

30

Trip # Day 1

Day 2

Day 3

Day 4

Day 5 23 North Bend, OR Brookings, OR Y Grants Pass, OR

24 Yreka, CA Eureka, CA Klamath Falls, OR

25 Ukiah, CA Y Clearlake, CA Y Chico, CA Y Yuba City, CA Y McClellan Park, CA

Y

26 Susanville, CA Y Auburn, CA Y Fallon, NV Y Gardnerville, NV Y

27 Santa Rosa, CA Fairfield/Travis, CA

Mare Island, CA

28 Winnemucca, NV

29 Oakland, CA Y SF Downtown, CA

San Bruno, CA Fremont, CA Y Monterey, CA/Capitola, CA

30 Stockton, CA Central Valley, CA

Modesto, CA Y Sonora, CA Y

31 Oakhurst, CA Y Atwater/Merced, CA

Y Tulare, CA Y

32 Santa Maria, CA Y Santa Barbara, CA

Y Oxnard, CA Y San Luis Obispo, CA

Y Bakersfield, CA Y

33 Antelope Valley, CA

Y Victorville, CA Y Rancho Cucamonga, CA

y

34 East Los Angeles, CA

Gardena, CA

35 Whittier/Santa Fe Springs, CA

Y Cabrillo, CA Y Anaheim, CA Y Corona, CA Y

36 Santa Ana, CA Y Laguna Hills, CA Y Murrieta, CA Y Palm Desert, CA Y Yuma, AZ Y

37 Escondido, CA Y Chula Vista, CA Y Imperial Valley, CA

Y Oceanside, CA Y Mission Valley, CA

38 Cutbank, MT Y Great Falls, MT Y Glasgow, MT

39 Salmon, ID Missoula, MT Y Anaconda, MT Y Bozeman, MT Y

40 Lewiston, MT Glendive, MT Miles City, MT

41 Powell, WY Y Gillette, WY Y Casper, WY Y Riverton, WY Y

31

Trip # Day 1

Day 2

Day 3

Day 4

Day 5 42 Pocatello, ID Y Roosevelt, UT Rock Springs, WY

43 Ogden, UT Y W Salt Lake Valley, UT

Y Orem, UT

44 Fort Collins, CO Greeley, CO Golden, CO Y Aurora, CO Y

45 Sidney, NE Y

46 Montrose, CO Durango, CO Y Farmington, NM Y

47 Lamar, CO Y La Junta, CO Y Pueblo, CO Y Alamosa, CO Y Raton, NM Y

48 Kingman, AZ Y Lake Havasu, AZ Y Flagstaff, AZ Y

49 Show Low, AZ Y Globe, AZ Safford, AZ

50 Cottonwood, AZ Payson, AZ Phoenix (Thunderbird), AZ

Y Surprise, AZ SE Gilbert, AZ/Anthem, AZ

51 Casa Grande, AZ Y Green Valley, AZ Y Sierra Vista, AZ Tucson, AZ (NW) Tucson, AZ (SE)

52 Childress, TX Y Lubbock, TX Y Stamford, TX Abilene, TX

53 San Angelo, TX Y Big Spring, TX Y Odessa, TX Y Fort Stockton, TX

54 Gallup, NM Y Espanola, NM Y Las Vegas, NM Y Santa Fe, NM Y Dalhart, TX Y

55 Clovis, TX Hobbs, TX Artesia, TX Alamogordo, TX Y Las Cruces, TX Y Eastside El Paso, TX

56 Rio Rancho, NM Truth or Consequence, NM

Y Silver City, NM

57 Del Rio, TX Y Laredo, TX Y

58 Polk Street Annex Clinic

Y Sherman, TX Y Bridgeport, TX Y Denton, TX Y Greenville, TX Y Bonham, TX Y

59 Brownwood, TX Y Granbury, TX Y Tyler, TX Y Palestine, TX Y Sherman, TX Y

60 New Braufels, TX

Cedar Park, TX Y College Station, TX

Y La Grange, TX Y

61 Beeville, TX Y Seguin, TX South Bexar County, San Antonio, TX

Y Pecan Valley, San Antonio, TX

Y Victoria, TX Y

32

Trip # Day 1

Day 2

Day 3

Day 4

Day 5 62 LaSalle, IL Y Joliet, IL Y Manteno, IL

(Kankakee) Y

63 Oak Lawn, IL Y Chicago Heights, IL

Y Auburn Gresham, IL

Y Lakeside, IL Evanston, IL

64 Freeport, IL Y Rockford, IL Y Aurora, IL McHenry, IL Elgin, IL

65 Janesville, WI Y Union Grove, WI Y Kenosha, WI Y

66 Baraboo, WI LaCrosse, WI Beaver Dam, WI Y Cleveland, WI LaCrosse #2, WI

67 Loyal, WI/Clark County, WI

Wisconsin Rapids, WI

Green Bay, WI

68 Rhinelander, WI Y Wausau, WI Y Menominee, WI Y

69 Ironwood, MI Y Hancock, MI Y Marquette, MI Y Sault St. Marie, MI

Y Manistique, MI

70 Bangor, ME/Lincoln, ME

Y Caribou, ME Y Calais, ME Y

71 Tilton, NH Y Conway, NH Y Rumford, ME Lewiston, ME Bingham, ME

72 Portland, ME Y Saco, ME Y Somersworth, NH

Y Portsmouth, NH Haverhill, MA

73 Littleton, NH Y Burlington, VT Y Rutland, VT Y Bennington, VT

74 Brattleboro, VT Keene, NH Fitchburg, MA Y Greenfield, MA Y Pittsfield, MA Y

75 Lowell, MA Y Gloucester, MA Y Lynn, MA Y Causeway St, Boston, MA

Y Quincy, MA Y

76 Middletown, RI Y New Bedford, MA

Y Oak Bluff, MA Hyannis, MA Y Plymouth, MA

77 New London, CT Y Willimantic, CT Y Winsted, CT Y Waterbury, CT Y Danbury, CT Y

78 Plattsburgh, NY Malone, NY Y Massena, NY Y Saranac Lake, NY

Westport, NY

79 Troy, NY Y Clifton Park, NY Y Glen Falls, NY Y Fonda, NY Y Schenectady, NY Y

80 Rome, NY Y Watertown, NY Y Oswego, NY Y Auburn, NY Y Freeville, NY Y

81 Bainbridge, NY Binghamton, NY Elmira, NY Mansfield, PA Coudersport, PA

82 Niagara Falls, NY Lockport, NY Y Lackawanna, NY Y Springville, NY Y Dunkirk, NY Y

83 Wellsville, NY Y Olean, NY Y Jamestown, NY Y McKean, PA Warren, PA Y

33

Trip # Day 1

Day 2

Day 3

Day 4

Day 5 84 Catskill, NY Y Kingston, NY Y Sayre, PA Y Williamsport, PA Y Columbia

County, PA

85 Tobyhanna, PA Y Pottsville, PA Y Reading, PA Frackville, PA Y Northampton County, PA

Y

86 Camp Hill, PA Y York, PA Y Lancaster, PA Spring City, PA Springfield, PA

87 Ft. Indiantown Gap, Annville, PA

Fort Dix, NJ Y Camden, NJ Y Atlantic County, NJ

Gloucester, NJ Y

88 Cumberland County, NJ

Y Kent County, DE Cape May County, DE

Y Sussex County, DE

Cambridge, MD Y

89 Ft. Howard, MD Y Ft. Meade, MD Y Greenbelt, MD Y Glen Bernie, MD Y

90 Ft. Belvoir, VA Y Southeast, DC Y S. PG County, MD

Y Charlotte Hall, MD

Y Pocomoke City, MD

Y

91 Fort Detrick, MD Y Hagerstown, MD Y Cumberland, MD

Y Petersburg, WV Y Stephens City, VA

Y

92 Ashtabula, OH Y Crawford, PA Y Venango, PA Y Clarion County, PA

Dubois, PA

93 Mercer County, PA

Y Lawrence County, PA

Y Beaver County, PA

Y Cranberry Township, PA

Y Armstrong County, PA

94 State College, PA

Y Westmoreland, PA

Washington County, PA

Fayette County, PA

Belmont County, OH/Johnstown, PA

Y

95 Monongalia County, WV

Y Wood County, WV

Y Braxton County, WV

Tucker County, WV

Franklin, WV Y

96 Morristown, NJ Piscataway, NJ Y Hamilton, NJ Y Tinton Falls, NJ y Elizabeth, NJ Y

97 Paterson, NJ Port Jarvis, NJ Y Goshen, NJ Y Monticello, NJ Y

98 White Plains, NY New City, NY Y Carmel, NY Y Poughkeepsie, NY

Y Pine Plains, NY y

99 Newark, NJ Hackensack, NJ Yonkers, NY Harlem, NY Jersey City, NJ

100 Staten Island, NY Chapel Street (NYC), NY

Valley Stream, NY

Y Sunnyside(NYC), NY

Stamford, CT Y

101 East Meadows, NY

Y Bay Shore, NY Patchogue, NY Riverhead, NY Y

34

Trip # Day 1

Day 2

Day 3

Day 4

Day 5 102 Lynchburg, VA Y Charlottesville,

VA Y Staunton, VA Y Harrisonburg,

VA Y Fredericksburg,

VA Y

103 Greenbrier, WV Y Tazewell, VA Y Wytheville, VA Y Danville, VA Emporia, VA

104 Virginia Beach, VA

Y Elizabeth City, NC (Albemarle)

Y Goldsboro, NC Y Wilmington, NC Y Greenville, NC

105 Hamlet, NC Y Robeson, NC Y Jacksonville, NC Y Morehead City, NC

106 Hickory, NC Y Rutherfordton, NC

Y Franklin, NC Y Blairsville, GA Y Oakwood, GA Y

107 Rock Hill, SC/Sumter, SC

Y Spartanburg, SC Y Anderson, SC Y Florence, SC Y Aiken, SC Y

108 Orangeburg, SC/Trident, SC

Y Beaufort, SC Y Brunswick, GA Valdosta, GA Hinesville, GA Y Waycross, GA

109 Columbus, GA Y Perry, GA Y Albany, GA Dothan, GA Baldwin County, FL

110 Macon, GA Y Newnan, GA Ft. McPhearson, GA

Y Stockbridge, GA Carrollton, GA

111 Lawrenceville, GA

Y Smyrna, GA Y Rome, GA Y Athens, GA

112 Jasper, AL Y Gadsden, AL Y Decatur, AL Y Shoals Area, AL Y Huntsville, AL Y

113 Columbus, MS Y Smithville, MS Y Kosciusko, MS Meridian, MS Hattiesburg, MS

114 Natchez, MS Y Macomb, MS Bogalusa, LA Slidell, LA Hammond, LA Y

115 St. John, LA Y Houma, LA Y Franklin, LA Y Lafayette, LA Baton Rouge, LA Y

116 Jennings, LA Y Lake Charles, LA Y Ft Polk, LA Y Natchitoches, LA Y Monroe, LA

117 Longview, TX Texarkana, TX El Dorado, AR Greenville, MS Y Pine Bluff, AR Y

118 Richmond, TX Texas City, TX Y Galveston, TX Y Lake Jackson, TX Y

119 Katy, TX Y Tomball, TX Y Conroe, TX Y Charles Wilson, TX

Y

120 Hot Springs, AR Y Mena, AR Y Hartshorne, OK Ada, OK

121 Altus, OK Y Lawton, OK Y Wichita Falls, TK Ardmore, OK

122 Stillwater, OK Y Enid, OK Y Blackwell, OK Y Vinita, OK Jay, OK

35

Trip # Day 1

Day 2

Day 3

Day 4

Day 5 123 Harrison, AR Y Branson, MO Y Mountain

Home, AR Y Mt. Vernon, MO Y

124 Conway, AR Y Russellville, AR Y Ozark, AR Y Searcy, AR Y

125 Liberal, KS Dodge City, KS Hays, KS Salina, KS Hutchinson, KS

126 Emporia, KS Chanute, KS Parsons, KS Ft Scott, KS Nevada, KS

127 Junction City, KS Y Ft Riley, KS Seneca, KS St. Joseph, KS

128 Cameron, KS Y Kirksville. MO Y Excelsior Springs, MO

Y

129 Wyandotte, KS Y Lawrence, KS Y Garnett, KS Paola, KS Belton, KS Y

130 Warrensburg, MO

Y Sedalia, MO Y Jefferson City, MO

Y Mexicon, MO Y

131 Lake of Ozarks, MO

Y Marshfield, MO Y Ft Leonard Wood, MO

Y St. James, MO Y Salem, MO

132 Bellville, MO St Louis CBOC, MO

Y St Charles, MO Y Washington, MO

Y Farmington, MO

133 Mayfield, KY Y Sikeston, MO Y Paragould, AR Pocahontas, AR Y West Plains, AR

134 Paducah, KY Y Cape Girardeau, MO

Y Carbondale, IL Y Mount Vernon, IL

Y Harrisburg, IL Y

135 Hanson, KY Y Owensboro, KY Y Evansville, IL Y Vincennes, IL Y Effingham, IL Y

136 Martinsville, IN Y Bloomington, IN Y Terra Haute, IN Y Lafayette, IN Y

137 Springfield, IL Y Decatur, IL Y Mattoon, IL Y

138 Muncie, In Y Peru, IN Y Goshen, IN Y South Bend, IN Y

139 Benton Harbor, MI

Y Muskegon, MI Y Cadillac, MI Y Claire, MI Y Lansing, MI Y

140 Gaylord, MI Cheboygan, MI Traverse City, MI Alpena, MI Oscoda, MI/Grand Rapids, MI

141 Pontiac, MI Y Yale, MI Y Bad Axe, MI Y Grayling, MI Y Flint, MI Y

142 Jackson, MI Lima, OH Springfield, OH Y Richmond, IN Y

143 Middletown, OH Y Hamilton, OH Y Dearborn, IN Y Bellevue, KY Y Florence, KY Y Lawrenceburg, IN

Y

36

Trip # Day 1

Day 2

Day 3

Day 4

Day 5 144 Clermont

County, OH Y Georgetown, OH Y Portsmouth, OH Athens, OH Lancaster, OH

145 Grove City, OH Y Newark, OH Y Marietta, OH Y

146 Marion, OH Y Mansfield, OH Y Sandusky, OH Y McCaferty, OH Y

147 Zanesville, OH Y Cambridge, OH Y New Philadelphia, OH

Y East Liverpool, OH

Y

148 Ravenna, OH Warren, OH Y Painesville, OH Akron CBOC, OH

149 St Augustine, FL Y Palatka, FL Y Orange City, FL Y St. Marys, FL

150 Kissimmee, FL Y Leesburgh, FL Y Inverness, FL Clermont, FL Y Lecanto, FL Y

151 Lakeland, FL Y Zephyrhills, FL Y Brookville, FL Y Palm Harbor, FL Y St. Petersburg, FL

152 Sebring, FL Okeechobee, FL Port Charlotte, FL

Naples, FL Y

153 Hollywood, FL Y Homestead, FL Y Key Largo, FL Y Key West, FL

154 Miami (CBOC), FL

Pembroke Pines, FL

Y Deerfield Beach, FL

Y Boca Raton, FL Y Del Ray Beach, FL

Y

155 Ft. Pierce, FL Vero Beach, FL Stuart, FL Y Port St. Lucie Y

156 Helena, AR Y Memphis South, TN

Y Byhalia, MS Y Jonesboro, AR Y Covington, TN Y

157 Jackson, TN/Savannah, TN

Y Dyersburg, TN Y Dover, KY Clarkesville, TN Hopkinsville, KY

158 Maury County, KY

Y Tullahoma, TN Y McMinnville, TN Y Coopersville, TN Y

159 Morristown, TN Roane County, TN

Y Sevierville , TN Rogersville, TN

160 Bowling Green, KY

Y Grayson County, KY

Y Ft Knox, KY Y Newberg, KY Y

161 Shivery, KY DuPont, KY Y New Albany, IN Y Scott County, KY Y Carroll County, KY

Y

162 Charleston, WV Y Prestonsburg, Y Norton, VA Bristol, VA

37

Trip # Day 1

Day 2

Day 3

Day 4

Day 5 KY

163 Gallipolis, OH Y Lenore, WV Y

164 Biloxi VAMC Panama City, FL Y Eglin ARB Y Marianna, FL Y

165

166 Springfield, MA Worcester, MA Y Farmington, MA Y

167 Selma Y Monroeville, AL Y Ft. Rucker Y Guntersville Y

168 Colville, WA Y Republic, WA Y Ponderay, ID Y Tonasket, WA Y

169 St. George, UT Y Pahrump, CA Y

170 NE 410, San Antonio, TX

NW 410, San Antonio, TX/SW Military Clinic, San Antonio, TX

North Central, San Antonio, TX

Y Valcones Heights, San Antonio, TX

Shavano Park, San Antonio, TX

Y

171 Oxford, AL Y Childersburg, AL Y Bessemer, AL Y

172 Hazard, KY Y Berea, KY Y Somerset, KY Y Morehead, KY Y

173 Scotts Bluff, NE Y

172 - San Juan

Arecibo, SJ Y Ponce, SJ Y Mayaguez, SJ Y Guayama, SJ Y Ceiba, SJ Y

173 - Haw

VA Pacific Islands, Honolulu, HI

Y Hilo, HI Y Kona, HI Y Ewa Beach, HI (Leeward CBOC)

Y Maui, HI Y

Kauai, HI (173) Y Kahului (173) Y Batavia, NY Y Kerrville, TX Newington, CT Y Total By

Day 124 131 114 94 54

Complete Total

517

Total to Visit

746

38

Appendix C: Phase One Sites Determined to Require a Further Review

C.1 Phase One Sites Requiring Further Review

The table below provides a list of sites determined to require a further review based on

assessment of site team reports. Overall, 81 (37%) of the sites from the 216 sites

visited in the Phase One Access Audit require further review. This initial

assessment of sites requiring further review is based on a review of qualitative

responses by front-line staff to questions contained in site audit reports. The listing of

these sites should be understood as a preliminary step, and further actions will be

taken after the determination of the extent of issues related to scheduling and

access management practices.

VISN

Facility Name

Requires Further Review

1 1 VA Central Western Mass, MA Yes

2 1 Boston (Brockton), MA Yes

3 3 New Jersey Health Care System (HCS) Lyons Campus, NJ

Yes

4 3 Castle Point Campus Hudson Valley VA HCS, NY Yes

5 4 Clarksburg VA Medical Center (VAMC), WV Yes

6 4 Philadelphia VAMC, PA Yes

7 4 Pittsburgh University Drive, PA Yes

8 4 Wilmington VAMC, DE Yes

9 4 Altoona, PA Yes

10 4 Erie, PA Yes

11 4 Lebanon VAMC, PA Yes

12 4 Horsham/Willow Grove Community Based Outpatient Clinic (Philadelphia), PA

Yes

13 5 Martinsburg VAMC, WV Yes

14 5 Washington, DC Yes

15 6 Raleigh CBOC, NC Yes

16 6 Richmond VAMC, VA Yes

17 6 Charlotte, NC Yes

18 7 Charleston, SC Yes

19 7 Myrtle Beach SC Yes

20 7 Savannah, GA Yes

21 7 Dublin, GA Yes

22 7 Montgomery, AL Yes

23 7 Tuskegee, AL Yes

24 7 Augusta, GA Yes

39

VISN

Facility Name

Requires Further Review

25 7 Columbia, SC Yes

26 7 Atlanta, GA Yes

27 8 Gainesville, FL Yes

28 8 Sarasota CBOC, GA Yes

29 8 San Juan, PR Yes

30 8 Bay Pines, FL Yes

31 8 Lake City, FL Yes

32 9 Nashville Main Campus, TN Yes

33 9 Memphis, TN Yes

34 9 Chattanooga CBOC, TN Yes

35 10 Cleveland, OH Yes

36 10 Cincinnati, OH Yes

37 11 Ann Arbor HCS, MI Yes

38 11 Indianapolis, IN Yes

39 11 Danville, IN Yes

40 12 Hines VAMC, IL Yes

41 12 Peoria, IL Yes

42 12 Adam Benjamin Jr. Clinic (Crown Point) Yes

43 12 Madison VAMC, WI Yes

44 15 Wichita, KS Yes

45 15 Leavenworth, KS Yes

46 15 Marion, IL Yes

47 16 New Orleans, LA Yes

48 16 Shreveport, IL Yes

49 16 Alexandria, LA Yes

50 16 Mobile, AL Yes

51 16 Houston, TX Yes

52 16 Baton Rouge, LA Yes

53 16 Pensacola, FL Yes

54 17 South Texas Veterans HCS (San Antonio), TX Yes

55 17 Central Texas HCS (Temple), TX Yes

56 17 Dallas, TX Yes

57 17 Fort Worth, TX Yes

58 17 Corpus Christi Outpatient Clinic and PACT Annex, TX Yes

59 17 Harlingen (Texas Coastal Bend HCS), TX Yes

60 17 McAllen (Texas Coastal Bend HCS), TX Yes

61 17 Central Texas Health Care System Austin, TX Yes

62 18 Big Springs (West TX VA Medical Center), TX Yes

40

VISN

Facility Name

Requires Further Review

63 18 New Mexico VA Health Care System (Albuquerque), NM

Yes

64 18 Prescott, AZ Yes

65 19 Fort Harrison, MT Yes

66 19 Billings (CBOC), MT Yes

67 19 Grand Junction, CO Yes

68 19 Cheyenne, WY Yes

69 19 Colorado Springs CBOC, CO Yes

70 20 Spokane, WA Yes

71 20 VA Puget Sound HCS (Seattle Division), WA Yes

72 20 VA Puget Sound HCS (American Lake Division), WA Yes

73 20 Walla Walla VAMC, WA Yes

74 20 Portland VAMC (Vancouver Campus), WA Yes

75 20 Portland VAMC (Oregon Campus), OR Yes

76 20 Roseburg, OR Yes

77 21 Livermore VAMC, CA Yes

78 22 Sepulveda (Los Angeles), CA Yes

79 22 Las Vegas (Main), NV Yes

80 22 Las Vegas Southwest (CBOC), NV Yes

81 23 Minneapolis HCS, MN Yes

C.1 Phase Two Sites Requiring Further Review

The table below provides a list of sites determined to require a further review based on

assessment of site team reports. Overall, 31 sites in the Phase Two Access Audit

require further review. This initial assessment of sites requiring further review is based

on a review of qualitative responses by front-line staff to questions and comments

contained in site audit reports. The listing of these sites should be understood as a

preliminary step, and further actions will be taken after the determination of the

extent of issues related to scheduling and access manage

41

VISN Facility Name

Requires Further Review

1 4 Westover, WV Yes

2 4 Wood County, WV Yes

3 6 Virginia Beach, VA Yes

4 6 Raleigh CBOC, NC Yes

5 6 Elizabeth City, NC Yes

6 6 Wilmington, NC Yes

7 7 Smyrna/Austell, NC Yes

8 8 Eglin, FL CBOC Yes

9 8 Jacksonville, NC CBOC Yes

10 8 San Juan, PR Yes

11 9 McMinnville, TN Yes

12 9 Dupont, KY Yes

13 9 Ft. Knox, KY Yes

14 10 Richmond, OH CBOC Yes

15 10 Wenatchee, OH CBOC Yes

16 11 Muskegon, MI CBOC Yes

17 11 Lansing, MI CBOC Yes

18 12 Joliet, IL CBOC Yes

19 12 Great Lakes, IL CBOC Yes

20 12 Kenosha, WI Clinic Yes

21 12 Janesville Clinic Yes

22 15 West Plains, MO Yes

23 16 Gulfport, MS VAHCS Yes

24 16 Hot Springs, AR Yes

25 20 South Sound CBOC (Chihalis, WA) Yes

26 21 Yuba City, CA Yes

27 22 Escondido, CA CBOC Yes

28 22 Imperial Valley, CA Yes

29 23 Rapid City, SD CBOC Yes

30 23 Prairie Health: Faith, Isabel and, Eagle Butte CBOCs, SD

Yes

31 23 Rochester, MN Yes

42

Appendix D: Further Data Tables

This section contains certain additional tabular reports from the Access Audit

questionnaires. It is noteworthy that there are many opportunities (highlighted in yellow)

to increase the consistency of desirable practices within VA facilities

D.1 Customer Service Emphasis

Responses

% "Yes"

% Facilities

with at least 1

Response "Yes"

% Facilities with >25% Responses

"Yes"

% Facilities with >75% Responses

"Yes"

Q35. Are patients on the missed opportunity list called to remind them of upcoming appointments?

2,893 57 140 (100 %)

127 (91 %) 35 (25 %)

Self Rating of Customer Service (1=poor, 2=fair, 3=good, 4=very good,

5=excellent) Responses Mean Score

% Facilities with mean rating 3 or

greater

% Facilities with mean rating less

than 3

Q24. Please rate yourself on customer service

2,925 4.6 140 0

Q25. Please rate your facility on customer service

2,909 4.1 139 1

Respondents indicated that other specific barriers existed to offering Veterans timely access to care.

* indicates mandatory questions

Responses % "Yes"

% Facilities

with at least 1

Response "Yes"

% Facilities

with >25%

Responses "Yes"

% Facilities

with >75%

Responses "Yes"

Q18. Are there other obstacles to being able to provide Veterans timely access to care?

2,921 51 140 (100%)

129 (92%)

20(14%)

43

D.2 Supervision for Frontline Staff

* indicates mandatory questions Responses % "Yes"

% Facilities

with at least 1

Response "Yes"

% Facilities

with >25% Responses

"Yes"

% Facilities

with >75% Responses

"Yes"

Q29. Does your supervisor periodically check your work?

3,006 79 140 (100 %)

140 (100 %)

91 (65 %)

Q31. Do you receive any feedback?

2,354 94 140 (100 %)

140 (100 %)

136 (97 %)

Responses

Daily

Weekly

Monthly

Every 6 mo

Annually

Don't know

other

Q30. How often does your supervisor check your work? (PERCENT)

1539 30% 26% 15% 4% 1% 16% 7%

D.3 Training for Frontline Staff A large proportion of staff appear to have received training on the scheduling policy, but 50% of schedulers could not recall when their last training had occurred, and, for 14% of schedulers, the training appears to have occurred in anticipation of the audit.

* indicates mandatory questions

Responses % "Yes" % Facilities

with at least 1

Response "Yes"

% Facilities

with >25% Responses

"Yes"

% Facilities

with >75% Responses

"Yes"

Q21. Have you received training on the scheduling policy at your facility?

3,052 96 140 (100 %)

140 (100 %)

138 (99 %)

44

Responses

Within last

week

Within last

month

Within last 6 mo

Within last year

More than a year ago OTHER

Q22. If yes, when was the last training completed? (PERCENT)

1927 14% 0% 0% 23% 14% 50%

D.4 Improvement of Scheduling Practices

* indicates mandatory questions

Responses % "Yes"

% Facilities

with at least 1

Response "Yes"

% Facilities

with >25% Responses

"Yes"

% Facilities

with >75% Responses

"Yes"

Q32. Has anything been done at your facility to improve the scheduling process including entry of desired date?

2,950 59 140 (100 %)

137 (98 %) 18 (13 %)

D.5 Regular review by Clinic Managers of Clinic Operations and Access Data.

* indicates mandatory questions

Responses % "Yes"

% Facilities

with at least 1

Response "Yes"

% Facilities

with >25% Response

s "Yes"

% Facilities with >75% Responses

"Yes"

Q56. * Do you review clinic operations data (i.e. the Access Index or the like information) at regular team meetings?

259 71% 87% 86% 58%

45

Responses Daily Weekly Monthly

Every 6 mo Annually Other

Q57. If you review clinic operations data, how often is it reviewed? (PERCENT)

183 13% 35% 36% 5% 0% 12%

46

Appendix E: Questionnaire

47

48

49

50

51

52

53

54