nanpa oversight working group 2000 nanpa performance review june 18, 2001 pat caldwell, chair

NANPA Oversight Working Group

2000 NANPAPerformance Review

June 18, 2001Pat Caldwell, Chair

June 18, 2001 2000 NANPA Performance Review 2

New EnvironmentChanges to NANPA’s functions and responsibilities in 2000.

• First and Second NRO Orders

• COCUS evolves into NRUF

• State delegated authority

• Increased State activity in NPA relief planning

• New criteria for code requests (e.g. facilities readiness, utilization thresholds)

• Greater regulatory demand for information


Performance Evaluation ProcessNOWG data gathering, analysis and feedback

• Solicit feedback via surveys and NOWG operational reviews

• Analyze input• Develop conclusions and recommendations• Review with the FCC• Preview results with NANPA• Present report to the NANC


Input AnalyzedSurvey responses, operational review and observations

• NANPA’s Annual Report• NANPA operational reviews and documentation• Performance Feedback Survey responses

– State Commissions 16

– Industry 26

– Non-US Acknowledgements


Response ComparisonsFewer responses than in the past

NANPA Annual Performance Review Volume of Responses

50

68

2617 14 16

0

20

40

60

80

1998 1999 2000

US Industry

State Regulators


AnalysisNOWG Activities and Process

• Contacted originator for clarification if survey responses seemed contradictory or unclear

• Aggregated analysis of survey ratings - Quantitative• Aggregated analysis of survey comments- Qualitative • NOWG observations and concerns• Recommended steps for improvements


Criteria for Assessment of CommentsCategories used for analysis

NOWG developed chart to ensure analytical consistency of survey ratings.

Note: This chart was not sent

to survey respondents.

Table 1

Satisfaction Rating Scale

Satisfaction Rating

Used when

EXCEEDED

Exceeded performance standards for position overall. Exceeded performance even in the most difficult and

complex parts of the position, including taking on responsibility for extra or unique tasks.

Decisions and recommendations were always sound and often in unstructured, non-routine areas of position responsibilities.

MET

Met the performance standards for the position. No improvement or very little is needed in order to be

considered fully successful in all aspects of the position.

Performance was competent and reliable. Decisions and recommendations were sound in routine

areas, and were generally sound in the less structured, non-routine areas.

SOMETIMES MET

Did not always complete or consistently meet one or more of the performance standards for the position.

Did not always complete assignments on time or comprehensively, therefore, did not thoroughly meet required standards of performance.

Improvement is required in the areas where deficiencies are noted.

Performance is below reasonable expectations given length of time in position.

DID NOT MEET

Did not meet the performance standards for the position.

Position objectives were not met. Performance was unreliable and assignments were

submitted late, incomplete, or not at all. Decisions and recommendations were not sound.

There is a need for to demonstrate immediate improvement in performance in the areas where deficiencies were noted.

N/A Not Applicable


Criteria for Assessment of CommentsCategories used for analysis

NOWG developed chart to maintain analytical consistency throughout the survey comments analysis.

Table 2 Performance Categories for Qualitative Analysis

Communications/Responsiveness

NANPA’s ability to communicate effectively in a timely manner consistent with regulatory directive, requirements and guidelines

Guidelines/Requirements/ Regulatory Directives

NANPA’s contractual obligation to comply with state and federal regulatory directives, requirements and guidelines by performing all functions agreed to in a manner that comports to the intent and quality expectations.

Staffing

Met the level of staffing required to perform all the NANPA functions, and that all personnel have the necessary expertise and education to perform the job.

Technical Expertise and Analysis

Sufficient tools and knowledge applied by NANPA personnel in the performance of their functions.

Web Site

Electronic repository that provides general access to NANP information for regulators, industry and the public. Key features include the status of the various numbering resources, timely updates, NANPA contact information.

Tactical

Safeguarding of NANPA’s obligation as unbiased and impartial steward of NANP resources without the advocacy of positions that will create economic collateral for their parent company at the expense of the industry and regulators (their clients).


ConclusionNANPA 2000 Annual Performance Review resulted in “MET” rating

The NOWG has determined that the NANPA’s performance resulted in a “Met” rating, which the NOWG determined means the following:

Met the performance standards for the position. Outside the areas which may have been improved if CAS had been delivered,

little improvement is needed in order to be considered fully successful in all aspects of the position.

Performance was competent and reliable. Decisions and recommendations were sound in routine areas, and were generally

sound in the less structured, non-routine areas.

The reader should be aware that the NOWG’s performance evaluation of “Met” should not be used as a comparison to the1999 performance review conclusion of “Above Average”. The NOWG changed the rating scale and any comparison of scales may give the reader an incorrect interpretation of NANPA’s performance. In addition it should be noted that the NANPA’s “Met” rating was achieved during substantial changes in 2000.


FindingsSummary of survey verbatims and NOWG observations

• Annual Report– Vast improvement with all requirements met except NANP exhaust

• Communications / Responsiveness– Prompt, courteous and professional

– Need to improve notification and provide longer transition periods when changing processes or requirements

• Guidelines / Requirements– Excellent job of maintaining confidentiality

– Need to conduct a full review of all information on Part 1 applications to avoid multiple re-submissions

– Need to improve Part 4 record management and reporting


Findings (con’t)Summary of survey verbatims and NOWG observations

• Technical / Analysis– Demonstrated improved expertise

– Need to improve familiarity with local conditions

– Need more proactive communications during NPA Relief meetings

– More practice resolution of contentious issues

– Need consistent interpretation/clear direction from NANPA regarding (new/developing) FCC rules and guidelines

Tools• The Document Distribution System received accolades

• Lack of information on CAS implementation and deployment date

• Forecasting model/analysis approach unclear and unknown


Findings (con’t)Survey verbatims and NOWG observations

• Staffing– Helpful, responsive and willing to assist– Need to improve response consistency with information provided by

different NANPA personnel

• Web– Contains useful and valuable information– Needs timely updating with new/additional information– Somewhat difficult to navigate

• Tactical– Continued reports of confusion regarding responsibilities of NeuStar

employees who sometimes perform functions as NANPA – Changes to administration procedures lack clear written justification

and documentation


RecommendationsAnnual Report

• NANP exhaust forecast must be included to meet the baseline criteria per Requirements Document.

• NANPA has the option to include any additional information.

Some suggestions that may improve the report are:1. Caribbean numbering authority contacts and web sites to the same

extent as the US and Canada2. NPA maps3. Information about the Binder of Decisional Principles and perhaps

the index listing (see last bullet item under Section 7 of this Report) 4. Add an Index


RecommendationsCommunications / Responsiveness

•NPA Relief Planning Letters need to be kept current on the web site and reflect changes, when changes occur.•NPA Relief Planners require training to better their facilitations skills, and ability to manage the consensus process during meetings and conference calls.•NANPA must provide notification when modifying or implementing a new process giving its customers advanced notice of the change and its effective date.•NANPA must revisit its practices on updating individual telephone voice messages to ensure announcements are updated in a timely manner.


Recommendations Guidelines / Requirements / Regulatory Directives

Both NPA Relief Planners and CO Code Administrators must ensure that they maintain uniformity, and consistency in their treatment and responses to their customers.

NPA Relief Planners need additional knowledge and training on local conditions, e.g. dialing plans.

NANPA needs to institute a records tracking and management system to ensure that the duplicative efforts caused by misplaced Part 4 forms are avoided in the future.

NANPA must make the FCC and NANC aware of concerns related to NPA relief activity.

NANPA must proactively facilitate interpretation and resolution when there are conflicts between rules, requirements and/or guidelines.


RecommendationsStaffing and Technical / Analysis

Staffing Continue enhancing the training and development program, meeting

facilitation skills and customer service for NANPA personnel.

Technical / Analysis NANPA needs to improve on the NPA relief forecasting accuracy. If guidelines are inadequate, NANPA has an obligation to

communicate the issue and provide a solution to the appropriate body for review.

NANPA must continue to ensure that all CO Code administrators and NPA Relief Planners apply the industry guidelines in the same manner and eliminate inconsistent interpretations.

(Note: NOWG acknowledges that states, under delegated authority, are permitted to direct NANPA to operate in a manner that does not comport with national guidelines.)


Recommendations Tools and Web Site

Tools NANPA must provide NANC with their rollout schedule for CAS by the end of

June such that the system will be available by September 1, 2001 NANPA must educate the industry and regulators about forecasting tools and

assumptions. They should also be using all available data, including detailed historical demand and forecasts, to make forecasts the highest possible quality.

Web Site Provide for more frequent updates to information on the web site. Provide for electronic assistance to those customers who are unable to navigate

the site. Post more detailed NANPA organizational chart and contact information,

including contact information for NeuStar employees who provide support to NANPA.


Recommendations Overall

NANPA needs to review entire resource applications for completeness, and advise of all errors, before suspending or rejecting an application.

There were issues raised in the 1999 performance review that came out again in 2000.

• CAS • Consistency

• NANPA vs. NeuStar

• Annual Report


Next Year’s Review In the interest of continuous NOWG improvement

• Eliminate competing NANPA surveys during response period

• Distribute survey forms earlier

• Simplify survey format and analysis

• Define rating scale

• Document a process for managing multiple submissions from individual company’s

nanpa oversight working group 2000 nanpa performance review june 18, 2001 pat caldwell, chair

Documents