1

EHR Response to the Report from the Committee of Visitors for the Division of Human Resource Development

November 2017

INTEGRITY AND EFFICIENCY OF PROGRAM PROCESSES AND MANAGEMENT

Briefly discuss and provide comments for each relevant aspect of the programs’ review process and management. Comments should be based on a review of proposal actions (awards, declinations, returns without review, and withdrawals) that were completed within the fiscal years under review. Provide comments for each program being reviewed and for those questions that are relevant to the programs under review. Quantitative information may be required for some questions. Constructive comments noting areas in need of improvement are encouraged. I. Questions about the quality and effectiveness of the programs’ use of the merit review process. Please answer the following questions about the effectiveness of the merit review process and provide comments or concerns in the space below the question.

QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCESS

YES, NO,

DATA NOT AVAILABLE, or

NOT APPLICABLE

1. Are the review methods (for example, panel, ad hoc, site visits) appropriate? Comments:

• The review process uses a mixture of panel reviews, drawing in ad hoc reviews and site visits when needed.

• The use of site visits to review and assess projects' unique components or program officers' questions/reservations about projects was appropriate.

• Recommendation: We encourage the use of expert ad hoc and mail reviews for highly technical and/or very specific research proposals where appropriate. Response: We agree that this approach is important, and HRD program officers will continue to employ it when they believe it is appropriate. We have found ad hoc reviews to increase the quality of merit review across all programs, and ad hoc reviews are especially useful, as the Committee notes, when a proposal involves highly technical or specific research topics.

• Recommendation: In instances of ad hoc and triaged proposal reviews, we recommend that additional debriefing feedback is provided to support future successful proposal reviews.

YES

2

Response: Program officers will strive to reference both panelists’ findings and ad hoc reviewers’ findings in the written PO Comments that they provide to PIs. Similarly, when programs choose to use the “triage” process, which eliminates a panel discussion and the resulting panel summary, program officers will highlight the most significant areas for improvement in their communications with PIs. We will also continue to encourage PIs to speak with the program officer about possible future proposals.

• Recommendation: We encourage expanding the use of site visits for

large, comprehensive, and/or institution-wide grants, as well as for capacity-building and less senior institutions.

Response: HRD programs will continue the regular practice of conducting site visits to the extent that budgets and staffing constraints allow, and we will ensure that large and comprehensive projects, as well as less senior institutions, are included. It is worth noting that the CREST program utilizes both site visits and reverse site visits, and the HBCU-UP and LSAMP programs provide technical assistance visits to institutions with little or no NSF funding as a pre-proposal activity. The TCUP and ADVANCE programs conduct regular post-award site visits to assess the progress of projects, as well as pre-award site visits to address the recommendations of reviewers.

Data Source: EIS/Type of Review Module 2. Are both merit review criteria addressed

a) In individual reviews? b) In panel summaries?

c) In Program Officer review analyses?

Comments: • Merit review criteria were more well-defined in panel summaries than

individual reviews, which led the COV to wonder whether the review criteria and review charge are clear enough to the individual reviewers.

• The panel summaries seemed to be able to draw out the distinction between the review criteria.

• The influence of the PO and/or panel chair on reviewers' attentiveness to the merit review criteria was clear – some programs seemed to have prepared their reviewers very well. An experienced chair can be very supportive of novice reviewers; for instance, AGEP was a strong example of this.

• Recommendation: A clearer definition and distinction is needed between broader impacts and intellectual merit – consider providing examples for how each relates both to student support programs as well as research focused programs.

YES

3

Response: In education R&D proposals, the distinction between features that constitute "intellectual merit" and features that constitute "broader impacts" is not always clear-cut. In proposals, PIs are expected to lay out the particular features they view as addressing intellectual merit and the particular features they view as addressing broader impacts. Reviewers may agree or disagree with the PI's interpretation. In the orientation webinars that HRD programs conduct for reviewers before they start reviewing proposals, program officers discuss examples of intellectual merit and broader impacts—including the fact that one reviewer may consider an issue more relevant to intellectual merit, while another may think the same issue is more relevant to broader impacts. NSF has developed a brochure (Perspectives on Broader Impacts [NSF 15-008]), a website (https://www.nsf.gov/od/oia/special/broaderimpacts/), and a video for reviewers, which define both merit review criteria and highlight many examples of broader impacts in proposals but which avoid being overly prescriptive about how both PIs and reviewers should approach the criteria. The division will incorporate these resources consistently across all programs and will include other examples in the briefings that are given to reviewers, so that they will have a richer context for interpreting the criteria in the set of proposals they are examining.

• Recommendation: Provide reviewers with a template that includes strengths and weaknesses under each criterion so that the reviewers are prompted and the expectations are clearer. Response: HRD programs ask reviewers to address intellectual merit and broader impacts separately in both individual reviews and panel summaries, in accordance with NSF's standard instructions and templates (in FastLane) for proposal review. Although we appreciate the Committee’s recommendation to provide reviewers with a more structured template (and we realize that some NSF programs do so), at the present time the division is hesitant to be overly prescriptive about the format of reviews. However, we will monitor this issue in the future, especially after incorporating other indicated changes in the orientation of reviewers, and we will add structure to improve the usefulness of reviews as needed.

• Recommendation: Intersectionality is a criterion outlined in some of

the solicitations; therefore, more attention should be paid to intersectionality in reviews. The COV strongly encourages that intersectionality be included as an explicit review criterion to make sure that it is addressed.

Response: Intersectionality is increasingly important in research on inclusion and academic/career success of underrepresented groups. When a program requires attention to intersectionality in proposals, we will list it as an “additional solicitation-specific review criterion” in the relevant solicitation (for example, the current ADVANCE solicitation [NSF 16-594] takes this approach), or program officers will instruct reviewers to address it explicitly as a component of a proposal’s intellectual merit or broader impacts.

4

Data Source: Jackets

3. Do the individual reviewers giving written reviews provide substantive comments to explain their assessment of the proposals? Comments:

• Overall, yes, strengths and weaknesses were noted; the reviews were well written and provided support for the assessment of each proposal.

• Although many were very thorough, others were short and thin.

• Recommendation: The COV encourages POs to do more to prepare and guide reviewers on the NSF review process, expectations, and reviewer responsibilities. Response: HRD programs regularly use pre-panel webinars, written instructions, and briefings at the beginning of panel meetings to guide reviewers. The Committee’s recommendation is timely as we attempt to grow our reviewer pool, and we will look for every opportunity to improve our reviewer orientation process, especially for inexperienced reviewers. NSF has recently launched a “reviewer orientation pilot” program as one means of providing consistent guidance to reviewers across the Foundation. HRD will explore whether materials from this program can be used to enhance its own reviewer orientations.

• Recommendation: To ensure consistency across HRD, we recommend developing a review template and instructions, and providing examples of informative and thorough reviews that draw upon models and best practices used by successful programs.

Response: As indicated in the response to the second recommendation under Question I.2 above, we currently prefer to use NSF’s standard instructions and the templates in FastLane, which reflect NSF policy regarding the merit review process. In orientations for reviewers, we stress that reviewers should give appropriate attention to both NSF merit review criteria in both individual reviews and panel summaries. Because of the variety and complexity of proposals across HRD programs, we also find it problematic to provide model reviews. However, during the orientation webinars for reviewers, we will pay greater attention to describing ingredients of strong, informative reviews.

Data Source: Jackets

YES

4. Do the panel summaries provide the rationale for the panel consensus (or reasons consensus was not reached)? Comments:

• Overall, yes. • Intellectual Merit and Broader Impacts criteria were addressed every

time, but the solicitation-specific criteria sometimes received no comments.

YES

5

• AGEP and ADVANCE were particularly thorough and a strong case for the panel consensus was made in the jackets reviewed.

• Recommendation: In cases where the PO analysis and recommendation go beyond what can be attributed to individual reviews, there should be a clear discussion and rationale for this decision. Response: We agree. Program officers will strive to mention in their review analyses all of the relevant factors that influenced their decision, especially factors not mentioned in the panel summary and reviews. As a standard practice, we will also try to ensure that PO comments (which go to the PI) capture the main reasons for a declination decision, with an emphasis on any points not included in the panel summary and reviews. As we review our programs’ procedures and orient new program officers, we will emphasize the importance of this issue.

Data Source: Jackets 5. Does the documentation in the jacket provide the rationale for the award/decline decision? [Note: Documentation in the jacket usually includes a context statement, individual reviews, panel summary (if applicable), site visit reports (if applicable), program officer review analysis, and staff diary notes.] Comments:

• In general, the COV was very impressed by the ability of POs to summarize and draw conclusions from the individual reviews and panel summaries, as well as their ability to solicit appropriate additional information from the applicant, when required – such as in instances where the proposal was on the borderline between award and decline.

• Documentation to declined PIs was very thorough and will be useful to support future proposals.

• Post-panel review questions found in correspondence as well as responses from the applicants were quite thoughtful overall.

• Recommendation: In cases where the PO has a strong and important

opinion that could benefit from a second expert opinion, we encourage HRD to make use of expert ad hoc reviews. Response: HRD supports the use of ad hoc reviewers and other experts, including program officers in HRD and other divisions. We will stress this option as a good practice.

Data Source: Jackets

YES

6. Does the documentation to the PI provide the rationale for the award/decline decision? [Note: Documentation to PI usually includes context statement, individual reviews, panel summary (if applicable), site visit reports (if applicable), and, if not otherwise provided in the panel summary, an explanation from the program

YES

6

officer (written in the PO Comments field or emailed with a copy in the jacket, or telephoned with a diary note in the jacket) of the basis for a declination.] Comments:

• Yes, the documentation to the PI identified strengths and weaknesses of the proposal, which the COV thought would be useful for project implementation in the case of awards as well as future proposal submissions in the case of declines.

Data Source: Jackets 7. Additional comments on the quality and effectiveness of the Program/Division use of merit review process:

• We applaud the strong leadership in the review process by program officers and encourage consistency across the Division.

• Overall, very good use of the merit review process.

YES

7

II. Questions concerning the selection of reviewers. Please answer the following questions about the selection of reviewers and provide comments or concerns in the space below the question.

SELECTION OF REVIEWERS

YES , NO, DATA NOT

AVAILABLE, or NOT

APPLICABLE

1. Did the Division make use of reviewers having appropriate expertise

and/or qualifications? Comments:

• HRD does a good job of finding a diverse set of well-qualified reviewers. • The COV noted the importance of balancing panels with an appropriate

mix of both experienced and novice reviewers to ensure that substantive comments and feedback are provided to applicants while also making sure that there is a robust pool of new reviewers to inject fresh ideas and perspective from the field and academia into the review process.

• We also noted the importance of having researchers/evaluators on panels.

• Diverse fields of study were well represented among panelists. • Reviewers who had specific expertise in content/domain were speaking

up on those areas in reviews, which adds credibility to the review process.

• Diversity of institutions and persons on panels was impressive and appropriate for the programs.

• The COV believes that inclusion of diverse and varied perspectives is important to the quality of the merit review process. The COV noted that within the HBCU-UP portfolio, the reviewers who reported their race were all African American/Black. Similarly, of the ADVANCE reviewers who reported their race, the majority were white. The COV supports inclusion and diversity across panels, including the inclusion of white males and persons with disabilities, who appear to be underrepresented.

COV Recommendations:

• Recommendation: While we recognize that there are two competing values – one to draw in the requisite expertise to provide high quality reviews, and another to continue to diversify and draw in novice reviewers – and we applaud NSF’s efforts to continue to make strides in each of these areas, we strongly encourage NSF to find new ways to continue to improve and pull in novice reviewers from junior faculty pools to increase diversity and opportunities for mentorship.

Response: All HRD programs strive for the type of balance that the Committee describes. Through outreach and referrals, we will make an effort to expand our pool of new reviewers, especially junior faculty.

• Recommendation: Consistent with the HRD Vision – a well-prepared and

competitive U.S. workforce of scientists, technologists, engineers,

YES

8

mathematicians, and educators that reflects the diversity of the U.S. population – we urge proactive, intentional, and continued vigilance across all HRD programs to maintain proportional demographic representation of institutions and individual reviewers.

Response: HRD programs strive for that goal to the extent possible. Within each HRD program, we will pay close attention to the demographic representation of institutions and reviewers. In striving for appropriate diversity, program officers must also consider the geographical distribution of the proposing institutions. For example, the TCUP, HBCU-UP, and CREST programs target (by design) institutions that are clustered or are otherwise not evenly distributed across all 50 states.

• Recommendation: The whole Division could benefit from pulling from reviewer pools across programs that are successful in recruiting diverse reviewers in areas of interest/need where the program has a gap (e.g., ADVANCE could pull from LSAMP).

Response: The division has actually followed this practice regularly, and we will continue to do so. HRD program officers often ask for reviewer recommendations from HRD program officers in other programs, as well as program officers in other divisions.

• Recommendation: Specifically with respect to ADVANCE, CREST, and

HBCU-UP, include more male reviewers of all backgrounds, with a sensitivity to where they are on the diversity continuum and their understanding of the importance of capacity-building initiatives at MSIs. Response: We will pay close attention to the recruitment of a diverse group of male reviewers in the indicated programs. During the period that the COV examined, the CREST and HBCU-UP programs had more male than female reviewers (approximately 73% male for CREST, and approximately 55% male for HBCU-UP). The ADVANCE program has recently increased male participation on proposal-review panels and site visit teams and has had success recruiting men of color, in particular. However, this issue remains a challenge, especially given the demographics of experts in gender equity. For example, for a recent ADVANCE Partnership panel, 11 women were invited and 10 accepted, whereas 11 men were invited and only 3 accepted. The program has expanded the range of expertise sought for its panels in order to recruit more men, and will continue to do so. All three programs make concerted efforts to recruit male reviewers of diverse backgrounds for panels. Because the demographic information that we can obtain is limited, we can usually gauge only approximately “where [reviewers] are on the diversity continuum.” We will look more closely at the balance of reviewers from MSIs vs. non-MSIs.

• Recommendation: Consider including training on cultural competency

and implicit bias in panel reviewer orientation where it is not already done.

Response: We will cover these interrelated issues in our orientation sessions for reviewers. The “reviewer orientation pilot” program that NSF

9

has recently developed for NSF-wide use has a good module on implicit bias, which we will consider using with HRD reviewers.

• Recommendation: POs across all programs should be more intentional

in selecting reviewers from diverse institution types – specifically, community colleges, tribal colleges, and other MSIs – across the Division.

Response: We agree that recruiting reviewers who represent a diverse array of institutions is essential, given the diverse community of institutions that HRD serves. Program officers will be mindful of this issue as they select reviewers. It is worth noting that at least half of HBCU-UP and TCUP reviewers have come from HBCUs and TCUs, respectively, and many ADVANCE, CREST, and LSAMP reviewers have come from MSIs.

• Recommendation: Within programs, special effort should be made to

make sure that reviewers participate in reviewing proposals for programs in which they or their institutions might not normally participate, but for which they have content/domain knowledge or a unique perspective to contribute.

Response: We will include this strategy as one element of being intentional about recruiting reviewers from a diverse array of institution types.

Data Source: Jackets

2. Did the Division recognize and resolve conflicts of interest when appropriate?

Comments:

• Yes, in instances where an unidentified COI was found during panels, the reviewer was excused from the discussion of that proposal.

Data Source: Jackets

YES

3. Additional comments on reviewer selection:

• It was noted that women of color were not well represented on

ADVANCE panels, which is of particular concern especially when criteria like intersectionality appear in the calls for proposals.

• Diversity of reviewers is important to the integrity of the review process, yet it was difficult to assess the diversity of the reviewers due to the limited amount of demographic information reported by reviewers. The COV wondered whether NSF can do more to encourage reviewers to report this information.

• Only 2.5% of the reviewers were from the 2-year college community. Given the importance of that type of institution in the pipeline, the COV felt that that this share is insufficient. When we are talking about broadening participation, we feel this institution type should be better represented.

YES

10

• Recommendation: To reduce the barriers that could keep some individuals from participating in the panel review process, consider making better use of remote (virtual) reviews. Consider training chairs to make sure that they know how to best include remote participants in discussions.

Response: During the past few years, NSF has used virtual panels more and more. They bring both benefits and challenges. One of the benefits is to enable participation by individuals who might not be able to travel to participate in an in-person panel. All HRD programs currently provide panelists with the option of participating virtually. We will continue to accommodate virtual panelists. In programs that utilize panel chairs, program officers will remind the chairs to include virtual panelists effectively in the panel discussions.

• Recommendation: In striving to include reviewers from diverse institution types, it is important to note that the institutional culture/climate at some institutions does not support the involvement of more junior faculty on panels. We recommend that NSF consider innovative approaches to attract and support the participation of junior faculty members as well as community college faculty and administrators (e.g., providing travel stipends up front to reduce the burden on the institutions and participants, conducting outreach to presidents and other institutional leadership to garner buy-in and support, etc.).

Response: Program officers and the division’s leadership often have opportunities to talk with department chairs, deans, provosts, presidents, and other administrators. During those conversations, we will make a point to mention our continual need for new reviewers, along with the benefits that faculty members gain by serving as reviewers—in particular, the knowledge of the characteristics of successful proposals and the experience of how groups of reviewers actually think about proposals when they discuss them. In addition, as mentioned earlier, we will actively recruit junior faculty and community college faculty to serve as reviewers. It is not likely that we can change travel and reimbursement policies to allow the advance of payments before services are rendered by panelists, but NSF will review this suggestion. We realize that requiring panelists to cover the costs of their lodging and meals until they are reimbursed after the panel meeting creates a hardship for some, and junior faculty are most likely to be affected. As one way to alleviate this burden, we will investigate whether it is possible to process reimbursements faster after panel meetings.

• Recommendation: The COV encourages HRD to continue its efforts to select reviewers who are representative of the applicant pool. Response: We agree about the importance of recruiting reviewers who are representative of the applicant pool, and we will continue our efforts to do so. (The responses to the recommendations under Question II.1 [pages 7–9] also address this issue.)

11

• Recommendation: The COV encourages HRD to continue and expand representatives from various types of institutions (including MSIs and 2-year institutions) across ALL six programs.

Response: We agree about the importance of recruiting reviewers from a diverse array of institutions. See the response to the sixth recommendation under Question II.1 (page 9).

• Recommendation: While experienced reviewers are enormously

valuable to review panels, it is also important to expand the diversity of perspective and opinion and to incorporate new ideas into the dialogue. HRD should be intentional about balancing panels with both experienced reviewers (who can mentor) and novice reviewers (who can interject new ideas into the discussion and review). This balance was found to be enormously beneficial to the review process where it was noted to exist.

Response: We agree about the importance of recruiting a mix of experienced and novice reviewers, both to inject fresh perspectives and to build the reviewer pool for the future. See the response to the first recommendation under Question II.1 (page 7).

• Recommendation: To support inclusiveness and diversity, consider

making better use of new awardees and competitive applicants in general as reviewers.

Response: We agree with this approach. Although we frequently invite investigators on recent awards, as well as more experienced investigators, to serve as reviewers, we will pay particular attention to the potential value of this pool for achieving diversity on panels.

12

III. Questions concerning the management of the programs under review. Please comment on the following: MANAGEMENT OF THE PROGRAM UNDER REVIEW

1. Management of the Division and its programs. Comments:

• Each program has formalized management plans and logic models that help to structure the management of each program portfolio

• The management plans address issues and concerns germane to their unique roles within the HRD portfolio.

• The COV applauds HRD for its division-wide strategic plan and accomplishments report, which responds to recommendations made by the EHR Advisory Committee. These documents are critical to ensuring a common vision for program success.

• We also applaud the programs for their clear use of their program management plans in the implementation and prioritization of program activities and initiatives.

• Some of the logic models do not appropriately account for external factors and/or are too narrowly focused. There is not always clear alignment between the problems/issues, activities, and how they will result in intended outcomes. In addition, it is not clear that the outcomes within the logic models make use of the best available science/evidence.

• We applaud the focus on external evaluations being required in grant projects. • When reviewing the AGEP program management documentation, the COV was concerned

by the reduction and uncertainty of the budget combined with the elimination of solicitation tracks and the message that this sends to the community.

• Recommendation: Develop an HRD-wide logic model which: o maps the program/Division goals to the outcomes o maps the program/Division activities to the intended short, medium and longer term

outcomes o indicates the measures that will be used to determine the degree to which outcomes

are achieved o provides the theory and/or the research to justify expectations that specific activities

will lead to specific outcomes o illustrates how each program plays a role in contributing to the program/Division

outcomes and meeting the program/Division overall objectives

Response: We appreciate the Committee’s interest in having a coherent picture of HRD’s programs and how they complement each other. We believe that we have largely addressed this interest in other documents. Namely, HRD’s Strategic Plan articulates the division’s vision, mission, and strategic goals; and before developing the Strategic Plan, HRD developed, with input from external scholars, a theory of change, which provides an explanatory framework for the division’s activities. Both are referenced on HRD’s website at https://nsf.gov/ehr/hrd/about.jsp. We agree that it could be an informative exercise to try to develop a logic model to accompany the Strategic Plan and the theory of change, although it might be challenging to create a logic model for an organization consisting of multiple programs, each of which has unique components and different goals. We will consult experts in EHR’s Evaluation and Monitoring Working Group about the feasibility of a logic model for the division. They regularly assist the division in designing and revising logic models for individual programs. Another option is to consult an external evaluator.

13

• Recommendation: In the logic models, we recommend more explicit recognition of the role

of institutional leadership in broadening participation.

Response: Program staff will address this issue the next time the logic models are revised. (Each program’s logic model is reviewed every time the relevant solicitation is revised.) In addition, when HRD’s Strategic Plan is updated, we will ensure that it calls attention to the role of institutional leadership in broadening participation.

• Recommendation: The COV encourages each program to make better use of program

monitoring data to determine whether the programs are achieving their outcome goals as well as to inform internal interim reviews and strategic planning activities. These activities will assist HRD in making midstream corrections that support programs in meeting outcome goals and may also help HRD assess whether the data being collected is most useful to supporting the achievement of goals and objectives.

Response: Currently, the CREST and LSAMP programs utilize monitoring systems, and efforts are underway to analyze the data with respect to the programs’ outcome goals and portfolios. For the four remaining programs, which do not have monitoring systems, HRD is conducting an inventory of data in NSF’s internal systems to assess its usefulness for program monitoring. The results will guide decision-making about future program monitoring systems.

• Recommendation: Logic models should be consistent across the Division and should

include clearly labeled sections – i.e., goals and assumptions.

Response: Program staff will address this issue the next time the logic models are revised. (Each program’s logic model is reviewed every time the relevant solicitation is revised.)

• Recommendation: Logic models lack uniformity and clarity and could benefit from additional

rigor. For instance, inputs and outcomes fail to take into consideration environment/context – i.e., as a baseline, what would have happened without any intervention. The logic models would be more valuable if the input included baseline numbers with regard to the population being served/targeted. And the outcomes numbers should be reported in relation to the baseline numbers, also taking into account the context/environment (e.g., demographic trends). There is very rich potential here for the use of NCSES data such as the Science & Engineering Indicators. For instance, according to the most recent S&E Indicators, postdoctorates in the biological sciences have been steadily declining over the past 10 years. You could use data like this to easily provide more substantive support for the success of your postdoctoral programs. In addition, the NCSES data sets are widely regarded as very good baseline data across the federal government, and we encourage HRD to make better use of this data.

Response: This is an excellent suggestion, and the program staff will address it the next time the logic models are revised. (Each program’s logic model is reviewed every time the relevant solicitation is revised.) In particular, we will ask EHR’s Evaluation and Monitoring Working Group, which advises the revision of the logic models, to consider the general issue of incorporating baseline data. In the past, when appropriate baseline data has been available, HRD programs have used it to revise their goals and objectives as expressed in new solicitations and logic models. The CREST program employed NCSES data to develop the CREST-Postdoctoral Research Fellowship track.

• Recommendation: Rotators comprise approximately one third of NSF’s workforce. Because

issues of broadening participation are historically long-term issues that require committed

14

champions and change agents over the long term, NSF may want to consider the role of continuity in leadership to broaden participation so that “champions” and “change agents” in this area do not turn over.

Response: We agree that continuity in leadership is important. At present, every HRD program is led by a career (“permanent”) program officer. Although HRD’s senior leadership as well as program leadership will change over time, the division’s practice is to link every rotator with a permanent employee to ensure that the division’s fundamental mission of broadening participation continues over time. In this way, critical knowledge is preserved and shared. We also recognize that champions and change agents in the community are equally beneficial. Expertise in broadening participation is one factor that is considered when HRD rotators are invited to work at NSF.

• Recommendation: The COV was excited to see additional staff allocated to the LSAMP and ADVANCE programs. Across programs, as proposal volume increases, staffing should respond to this need.

Response: HRD’s leadership will periodically revisit staffing levels in the programs in light of changing workload, and will make adjustments as resources permit.

• Recommendation: The COV recommends the continued use of post-award site visits as a management tool.

Response: We agree about the value of post-award site visits, and HRD programs will continue to conduct them within budgetary and staffing constraints.

2. Responsiveness of the Division to emerging research and education opportunities. Comments:

• As a whole the HRD portfolio of programs responds to important national needs and provides support to broaden participation across the STEM pipeline.

• We are seeing a shift in the demographics across institutions of higher education. There are now significant populations of URM students at predominantly white universities, not just at HBCUs, HSIs, etc. The COV wondered whether this portfolio of programs is designed to reach those students as well.

• With respect to TCUP in particular, there is an inherent tension between traditional values and culture, and innovation. This is a challenge for the program. The program could be well suited to blur traditional research boundaries, for example the Food-Energy-Water nexus, and thus address socially and culturally relevant problems/solutions that resonate with the community.

• With respect to CREST in particular, there were proposals in which the investigators were working on new frontier research. Consider adding, as a review criterion for renewal proposals that the reviewers have to evaluate how the research being proposed will position the center for research directorate funding.

• Recommendation: Exceptional faculty who, in addition to their research expertise, also have strong teaching skills, and who are able to encourage diverse students to pursue degrees and careers in STEM fields, are critical to a robust, diverse, and well-prepared professoriate of the future. The COV recommends prioritizing training of STEM faculty with pedagogical strategies needed to effectively teach and mentor diverse populations of students.

15

Response: Many HRD awards to institutions focus on helping STEM faculty to effectively mentor diverse student populations. This is a common concern when developing, implementing, and testing models to increase the success of historically underrepresented minority undergraduate and graduate students, postdoctoral fellows, and faculty. Besides continuing this focus on high-quality mentoring in funded projects, HRD will leverage the Division of Undergraduate Education’s Improving Undergraduate STEM Education (IUSE: EHR) program, which funds many projects to spread effective pedagogical practices.

• Recommendation: We recommend specifically targeting support to faculty from underrepresented groups, regardless of institution type. There is no program that supports faculty from groups underrepresented in STEM fields, especially women of color. While the intent is to reach them through HBCU-UP, TCUP, ADVANCE, etc., that logic assumes that the faculty at those funded institutions are in fact from underrepresented groups as well, which is not necessarily the case. Response: Although HRD’s grants go to institutions, not individuals, most HRD programs do strive to engage underrepresented faculty. Because these are Federal programs, they cannot be exclusive with regard to the individuals they assist. We note that HBCU-UP encourages and funds faculty development, reaching large numbers of faculty from groups underrepresented in STEM fields. AGEP’s awards to institutions focus on historically underrepresented minority faculty in STEM. The most recent ADVANCE solicitation (NSF 16-594) supports a focus on intersectionality with regard to institutional and cultural barriers that differ among women. As HRD’s programs evolve, we will look for other effective ways to serve faculty from underrepresented groups in all institution types.

• Recommendation: We encourage AGEP to more explicitly address pedagogy and teaching

skills needed to encourage and support the next generation of diverse STEM learners and teachers once they enter the professoriate. And consider adding a postdoctoral track, especially in fields where a postdoc is a necessary step to the professoriate.

Response: We agree that preparing faculty to teach effectively is an important issue. The AGEP program is shifting its focus from graduate students and postdocs to early career faculty members. Hence a postdoctoral track is not tenable. In future AGEP solicitations, we will encourage professional development related to the pedagogy and teaching skills that young faculty will need to succeed in their careers.

• Recommendation: We encourage an increased emphasis on intersectionality as an emerging research opportunity.

Response: We recognize the growing importance of intersectionality in broadening participation research. The most recent ADVANCE solicitation (NSF 16-594) added a program-specific review criterion on intersectionality, requiring each proposal to “address the intersection of gender with other characteristics … such as race, ethnicity, disability status, foreign-born and foreign-trained status, sexual orientation, and faculty appointment type.” The most recent AGEP solicitation (NSF 16-552) encourages proposers to “study the intersection of race and ethnicity with gender, disability, socioeconomic status and other demographic characteristics that may impact doctoral degree completion, postdoctoral training and faculty career progression.” In those programs, program officers are discussing the concept of intersectionality in webinars on proposal preparation and pre-panel orientation webinars for reviewers. Other programs with research emphases will call attention to the opportunity for research on intersectionality as new solicitations are developed.

16

3. Program planning and prioritization process (internal and external) that guided the

development of the portfolio. Comments:

• The COV applauds the HRD all hands meetings and recommends using this forum to talk about cross-program initiatives and strategy and cross-pollination.

• Although disability is included as a priority in HRD’s strategic plan, and since some of the work in this area has been distributed across the Foundation, there is a near absence of disability-related research or projects among the proposals reviewed.

• Recommendation: With regard to renewal proposals and experienced PIs, we encourage HRD to consider raising the review bar higher to encourage the pursuit of higher level goals and outcomes from more senior members of the broadening participation community and push for greater returns from continued investments. Response: We agree about setting a higher review standard for projects that have had several years of support from NSF. HRD programs do not generally support renewal proposals. Instead, PIs are expected to submit new proposals, and when they submit them to the same program track, the proposed project must be different from the previously funded project and must show accomplishments from the prior support. Such proposals must also include a description of how (if at all) the proposed new work relates to the previous project. Programs (such as LSAMP and HBCU-UP) that receive these types of proposals do set higher expectations for recommended awards.

4. Responsiveness of Program/Division programs to previous COV comments and recommendations.

Comments:

• Overall, yes, programs and the division did a great job at using the prior COV

recommendations to make important changes – for instance, the LSAMP program addressed the staffing issue noted by the prior COV.

• However, the prior COV for the HBCU-UP program recommended an increase in staffing due to the increase in proposals. We did not see this addressed during this period under review.

• The COV also noted that the following recommendation from 2014 ADVANCE COV report was not yet acted upon: “The ADVANCE program will seek the advice of the EHR Evaluation and Monitoring Working Group and will investigate the feasibility and resources required for a sophisticated program impact evaluation of the type that the COV recommends.” Program staff noted that this was currently in progress and will be funded in 2017. Response: Increasing workload is an issue across many NSF programs, and limited resources often make it a challenge to adjust staffing levels in a timely manner. The HBCU-UP program currently has 1.5 full-time equivalent (FTE) career employees and a program officer under the Intergovernmental Personnel Act (IPA) who is 100% allocated to the program. When the prior COV looked at the program’s staffing, it had only 1.25 FTE career employees and no IPA program officer. To address the workforce shortage, HRD has employed several additional program officers on a part-time basis.

17

IV. Questions about Portfolio. Please answer the following about the portfolio of awards made by the programs under review.

RESULTING PORTFOLIO OF AWARDS

APPROPRIATE,

NOT APPROPRIATE, OR DATA NOT

AVAILABLE

1. Does the program/Division portfolio have an appropriate balance of awards across disciplines and sub-disciplines of the activity? Comments:

• In general, yes. However, relatively few projects focus on the social, behavioral and economic sciences, particularly on the underrepresentation of certain minority groups in some fields, e.g., economics, geography, etc.

• In regard to the balance of awards across programs, a growing population of potential underrepresented students would suggest these programs should be growing. However, budgets are generally stagnant.

Data Source: EIS/Committee of Visitors Module. From the Report View drop-down, select the Funding Rate module to see counts of proposals and awards for programs. The Proposal Count by Type Report View will also provide a summary of proposals by program.

APPROPRIATE

2. Are awards appropriate in size and duration for the scope of the

projects? Comments:

• We commend HRD for changing the size and duration as needed to remain responsive to changing priorities and needs.

• Yes, funding levels are generally appropriate in size and duration. • As always, some of these projects will need more time to reach their

full potential, but if NSF can fund further requests down the line, the structural limitations for how long grants can be funded is not a problem.

• The duration (3 to 5 years) is appropriate – for example allowing centers/programs to become fully operational and subsequent outcomes to be observed. A thorough review of accomplishments in later years then provides a basis for performance-based renewals.

Data Source: EIS/Committee of Visitors Module. From the Report View drop-down, select Average Award Size and Duration.

APPROPRIATE

3. Does the program/Division portfolio include awards for projects that are innovative or potentially transformative?

APPROPRIATE

18

Comments: • Yes. Again, the COV has identified areas and opportunities for

growth, but overall, the level of transformative potential is high. • Awards reflect innovations in both the research and policy domains.

They also have the potential to transform practices in colleges and universities that have the critical role of workforce development. In this regard, both ADVANCE and HBCU-UP have been particularly important in advancing innovations and transformative actions to broaden participation. In both programs, the COV identified several examples of projects that have been transformative at the institutional level.

• Many awards are innovative at the individual and institutional level in terms of creating new strategies or applying existing strategies to increase URM participation and success.

• Recommendation: The COV noted some instances where scaling or replication of best practices is warranted. We encourage HRD to solicit and support more projects engaged in these types of efforts. Response: In its Strategic Plan and in its programs, HRD encourages the development and spread of best practices. Some HRD programs (LSAMP is a notable example) have tracks and resource centers designed to support the scale-up of practices, along with associated research. In addition, many awardees from HRD programs have begun to move their practices toward scaling and replication by submitting proposals to NSF’s INCLUDES program, which was launched in early 2016. In most HRD programs, however, current program budgets are not sufficient to support an emphasis on scale-up efforts, which are relatively expensive.

Data Source: Jackets

4. Does the Program/Division portfolio include inter- and multi-disciplinary projects?

Comments:

• Yes. For example, there is evidence of HRD co-funding from and to the research directorates.

• The vast majority of proposals reviewed were interdisciplinary.

• Recommendation: We encourage expanded intra-division and cross-directorate co-funding activities with HRD programs. Response: We are pleased to note that there are ongoing intra-divisional co-funding activities between HBCU-UP and CREST; ADVANCE and TCUP; and LSAMP and TCUP. Within EHR, there are increasing opportunities to expand the co-funding of grants in HRD programs. For example, NSF’s FY 2018 Budget Request to Congress calls for DUE’s Improving Undergraduate STEM Education (IUSE: EHR) program and NSF Scholarships in STEM (S-STEM) program to collaborate with HRD’s LSAMP program on the issue of enhancing the persistence of students from low-income and underrepresented groups, and HRD and DUE are also collaborating to design a new

APPROPRIATE

19

Hispanic Serving Institutions (HSI) program and will share the budget. HRD programs also receive co-funding from the Established Program to Stimulate Competitive Research (EPSCoR). In many cases, HRD and other directorates collaboratively fund related activities at particular institutions through separate grants. For example, TCUP collaborates with the directorates for Engineering, Geosciences, and Social, Behavioral, and Economic Sciences; AGEP and HBCU-UP collaborate with the Directorate for Mathematical and Physical Sciences; CREST collaborates with the Directorate for Engineering; HBCU-UP led an NSF-wide effort to support research at HBCUs in conjunction with other NSF research directorates; and LSAMP collaborated with other directorates as mentioned in the response below. However, in view of the distinctive characteristics of HRD programs and the targeted institutions with which HRD works, it is not likely that the amount of co-funding that other NSF units outside EHR contribute to grants in HRD programs will increase substantially from historical levels.

• Recommendation: The COV was surprised that LSAMP did not receive any co-funding from other directorates. We encourage additional outreach to secure co-funding for this important multi-disciplinary program.

Response: Over the past few years, LSAMP has engaged in complementary funding activities with the Directorate for Biological Sciences, the Directorate for Mathematical and Physical Sciences, and the Established Program to Stimulate Competitive Research (EPSCoR). The majority of these activities have involved student support (a critical issue in the LSAMP program), providing the means for LSAMP students to participate in research in the United States and abroad. The program expects this type of collaboration with other NSF directorates to continue. LSAMP program officers will look for opportunities to secure co-funding for LSAMP awards, especially when those projects advance particular research interests of the other directorates. However, we believe the opportunities are limited.

• Recommendation: We encourage HRD and the research directorates

to more proactively support CREST with increased co-funding.

Response: As mentioned above, CREST has received co-funding from the HBCU-UP program and has collaborated with the Directorate for Engineering to fund related activities. We expect those relationships to continue. Although an increase in direct co-funding from other directorates seems unlikely, CREST program officers will continue outreach to other directorates and will, in particular, look for additional opportunities to engage in complementary funding of related activities.

Data Source: If co-funding is a desired proxy for measuring inter- and multi-disciplinary projects, the Co-Funding from Contributing Orgs and Co-Funding Contributed to Recipient Orgs reports can be obtained using the EIS/Committee of Visitors Module. They are available as selections on the Report View drop-down.

20

5. Does the program portfolio have an appropriate geographical distribution of Principal Investigators? Comments:

• Compared to NSF as a whole, yes. This is a strength of HRD, and there appear to be proposals from and grants to institutions across the U.S. However, the data, as presented in the module, do not allow an assessment of whether or not within a particular community (Tribal Colleges, HBCUs, HSIs, etc.) there is a broad representation or whether only a few primary institutions are contributing to the bulk of awards.

• Different programs seem to be more heavily weighted toward the East or West. For instance, CREST seems to be weighted more toward the East, which may not appropriately account for the growing number of HSIs in the West.

• In other programs, awards are disproportionately concentrated in certain jurisdictions – e.g., DC has 50% more awards than CA and 11 jurisdictions have no current HRD awards.

• Despite scarce resources, HRD awards are more geographically and institutionally diverse than those in other divisions in NSF, and on this measure, HRD can serve as a model for other divisions.

• Recommendation: Consider comparing the geographical distribution of awards in HRD with the distribution in EHR and the research directorates (R&RAs). Response: We agree that this sort of comparison could be useful, and we will do it as we prepare future analyses of HRD’s portfolio of awards. As noted earlier, the clustering of awards in some HRD programs reflects the particular geographic distribution of the institutions targeted by the programs.

• Recommendation: HRD should provide the percentage of each type

of MSI that is funded and an assessment of how well the funded institutions within each type represent the whole population within that type.

Response: NSF compiles data and provides an annual report to Congress detailing NSF’s funding to MSIs. That data can be broken down by type of MSI and by NSF directorate, division, and program. We will include a breakdown for HRD in the briefing materials for the next COV. We are not sure whether we can find data that would address “how well the funded institutions within each [MSI] type represent the whole population within that type,” but we will ask our analysts to look into that issue.

Data Source: EIS/Committee of Visitors Module. Select Proposals by State from the Report View drop-down.

APPROPRIATE

6. Does the Program/Division portfolio have an appropriate balance of awards to different types of institutions?

APPROPRIATE

21

Comments: • Overall, the balance of awards seems reflective of the higher

education landscape in that some were to public, private, large, small, comprehensive, and baccalaureate institutions.

• The overall demographics of the U.S. are becoming more diverse in general and in higher education in particular. Therefore, the portfolio of awards should continue to seek full representation of institution types as a way of addressing URM students in each institution type (e.g., Predominantly White Institutions, HBCUs, HSIs, TCUs, community colleges).

• While different types of institutions are funded, the community college sector is underrepresented, especially considering its critical importance to enhancing the pipeline of future students, faculty, and the workforce, particularly for underrepresented minorities.

• The COV also noted that there were a limited number of ADVANCE proposals from MSIs.

• Recommendation: Given the large number of URM students enrolled in community colleges, HRD is encouraged to continue to reach out to and engage community colleges, and to increase proposal submissions as well as awards to those institutions. Response: We agree with the Committee about the importance of engagement with community colleges. HRD has made significant advances during the past few years. In collaboration with other NSF units, HRD has developed a series of Dear Colleague Letters (DCLs) that encourage proposal submissions by Hispanic-serving two-year colleges:

• NSF 14-064, “Stimulating Research on Effective Strategies in Undergraduate STEM Education at Two-Year Hispanic Serving Institutions”

• NSF 14-065, “Announcement of Effort to Broaden the Participation of Students in Two-Year Hispanic Serving Institutions in Science, Technology, Engineering, and Mathematics (STEM)”

• NSF 15-063, “Announcement of Effort to Broaden the Participation of Students in Two-Year Hispanic Serving Institutions in Science, Technology, Engineering, and Mathematics (STEM)”

• NSF 15-078, “Stimulating Research on Effective Strategies in Undergraduate STEM Education at Two-Year Hispanic Serving Institutions”

• NSF 16-094, “Strengthening Transfer of Students from Two-year Hispanic-serving Institutions to Four-year STEM Programs”

Although most of these DCLs were published during the period examined by the COV, they were not reported to the COV because they were collaborative efforts with the Division of Undergraduate Education (DUE) and other NSF directorates. An example of an award that emerged from those DCLs is Award HRD-1450661 to Science Foundation Arizona for the “KickStarter Program for Hispanic-Serving Community Colleges,” which is providing technical assistance to improve the competitiveness of those colleges for NSF

22

funding. Recently, HRD and DUE developed another DCL (NSF 17-092, “Improving Undergraduate STEM Education in Hispanic Serving Institutions [HSIs]”) to solicit ideas for the design of an HSI program, which will focus heavily on community colleges. In FY 2017, the AGEP program made three awards to associate degree-granting institutions. In addition, the HBCU-UP program has provided funding to many community colleges, and the LSAMP program continues to support the Bridges to the Baccalaureate alliances, which are led by community colleges.

• Recommendation: Conduct outreach to MSIs on opportunities within the ADVANCE program, and consider adding MSIs as a targeted priority in future solicitations.

Response: The ADVANCE program’s Partnership and Adaptation tracks are intended to increase opportunities for the broad spectrum of institutions, including MSIs. All MSIs and community colleges are invited to submit to all of the ADVANCE tracks. Program officers will conduct outreach to MSIs whenever possible. We are not likely to add MSIs as a targeted priority in future solicitations, especially since the program has added an emphasis on research on intersectionality, which cuts across institution types.

• Recommendation: Consider broadening the base of institutions in the

CREST portfolio, i.e., adding “emerging” institutions with respect to their research capacity. Response: See the response to the fourth recommendation under Question IV.11 (page 29).

• Recommendation: Several areas for increased program attention

have been identified and could be addressed: community college engagement, support for students and faculty with disabilities, and the advancement of URM faculty in different institution types.

Response: See the responses to the Committee’s recommendations concerning these three areas elsewhere in this document.

Data Source: EIS/Committee of Visitors Module. Select Proposals by Institution Type from the Report View drop-down. Also, the Obligations by Institution Type will provide information on the funding to institutions by type.

7. Does the program/Division portfolio have an appropriate balance of awards to new and early-career investigators? NOTE: A new investigator is an individual who has not served as the PI or Co-PI on any award from NSF (with the exception of doctoral dissertation awards, graduate or post-doctoral fellowships, research planning grants, or conferences, symposia and workshop grants.) An early-career investigator is defined as someone within seven years of receiving his or her last degree at the time of the award.

APPROPRIATE

23

Comments:

• We applaud HRD’s efforts in this area. • The portfolio included 21% early career, 26% midcareer, and 52%

late career PIs. • Many of the first-time PIs partnered with an experienced investigator.

Data Source: EIS/Committee of Visitors Module. Select Funding Rate from the Report View drop-down. After this report is run, use the Category Filter button to select New PI for the PI Status filter or New Involvement (PIs & coPIs) = Yes.

8. Does the Program/Division portfolio include projects that integrate research and education? Comments:

• Yes. Significant integration of research and education is reflected in several projects.

• This is especially true in some of the programs – e.g., CREST, where new research equipment is used for student projects or even lab courses.

• The COV liked the fact that the "Broadening Participation Research" tracks across solicitations encourage work, and research, on the science and social science of broadening participation.

Data Source: Jackets

APPROPRIATE

9. Does the program portfolio have appropriate participation of underrepresented groups1? Comments:

• For the most part; however, there may be concern about a limited number of PIs from underrepresented racial/ethnic groups in several HRD programs.

• Although disability is included as a priority in HRD’s strategic plan, it is not clear that the portfolio of awards reflects this priority.

• A specific focus on faculty of color is lacking.

• Recommendation: HRD is encouraged to accelerate its work on the intersectionality of race/ethnicity and gender, as well as other intersectionalities, particularly in the ADVANCE program. Response: As mentioned earlier, the ADVANCE program has added an explicit requirement for research on intersectionality to its solicitation, and the AGEP program’s solicitation now encourages attention to intersectionality. AGEP also made awards that examine the intersectionality of race/ethnicity and gender in FY 2016 and FY 2017. HRD has begun discussions on intersectionality within the

APPROPRIATE

1 NSF does not have the legal authority to require principal investigators or reviewers to provide demographic data. Since provision of such data is voluntary, the demographic data available are incomplete. This may make it difficult to answer this question for small programs. However, experience suggests that even with the limited data available, COVs are able to provide a meaningful response to this question for most programs.

24

division and within NSF via a lunch-time discussion series, and program officers will continue to emphasize the topic with proposers and panelists. HRD’s Data Scientist organized a working group of American Association for the Advancement of Science (AAAS) Fellows (who are currently working at NSF) to address this issue, among others, in preparation for an HRD presentation at the “Understanding Interventions that Broaden Participation in Science Careers” conference. The efforts of this working group will inform future HRD activities.

• Recommendation: HRD is also encouraged to expand participation of

persons with disabilities.

Response: HRD program officers will consider ways of increasing attention to persons with disabilities in the programs it currently manages. HRD program officers manage ongoing awards focusing on persons with disabilities that were made under the previous Research in Disabilities Education (RDE) program, as well as related awards in the directorate-wide EHR Core Research (ECR) program (which absorbed the RDE program and several other programs) and the NSF-wide INCLUDES program. Because this COV did not examine those programs (which are not primarily managed by HRD), the COV did not get a full picture of HRD’s attention to persons with disabilities.

• Recommendation: There should be a more explicit focus on faculty of

color, especially female faculty of color, across program areas.

Response: See the response to the second recommendation under Question III.2 (page 15).

Data Source: EIS/Committee of Visitors Module. Select Funding Rate from the Report View drop-down. After this report is run, use the Category Filter button to select Women Involvement = Yes or Minority Involvement = Yes to apply the appropriate filters.

10. Is the program/Division relevant to national priorities, agency mission, relevant fields and other constituent needs? Include citations of relevant external reports. Comments:

• The HRD Vision states that “HRD envisions a well-prepared and competitive U.S. workforce of scientists, technologists, engineers, mathematicians, and educators that reflects the diversity of the U.S. population.” Furthermore, “HRD's mission is to grow the innovative and competitive U.S. science, technology, engineering and mathematics (STEM) workforce that is vital for sustaining and advancing the Nation's prosperity by supporting the broader participation and success of individuals currently underrepresented in STEM and the institutions that serve them.”

• The Division’s programs strongly reflect the priority of developing the U.S. workforce – a critical national priority and 21st century challenge. The anticipated population demographics of the future indicate that this workforce development must focus on segments of the population

APPROPRIATE

25

that are underrepresented in the STEM disciplines. Thus programs that support inclusiveness, like the HRD programs, are relevant to the national priorities and the agency mission to sustain a world class workforce in STEM.

• The COV feels very strongly that the programs of HRD have made huge strides in broadening the participation of underrepresented groups in the sciences. Our enthusiasm is based on the composite of our prior experiences as well as the data provided for this specific COV. However, the HRD-wide COV should not be a substitute for activities that attempt to establish progress, impact, efficacy, and potential weaknesses of these outstanding programs. We were struck by the slim amount of material for our review that dealt with program impact or evaluation of any kind. As much as we love the programs, it seems that data ought to be available that can demonstrate efficacy and value to NSF and to the nation.

• The portfolio is attentive to numerous recent national reports that cite the need to advance diversity in the STEM workforce:

o Johnson, D.R. (2011). “Women of Color in Science, Technology, Engineering, and Mathematics (STEM).” New Directions for Institutional Research, 2011(152), 75-85.

o Moon, N.W., Todd, R.L., Morton, D.L., and Ivey, E. (2012). Accommodating Students with Disabilities in Science, Technology, Engineering, and Mathematics (STEM). Atlanta, GA: Center for Assistive Technology and Environmental Access, College of Architecture, Georgia Institute of Technology.

o National Academy of Sciences, National Academy of Engineering, and Institute of Medicine (2007). Rising Above the Gathering Storm. Washington, DC: The National Academies Press.

o National Academy of Sciences, National Academy of Engineering, and Institute of Medicine (2010). Rising Above the Gathering Storm, Revisited: Rapidly Approaching Category 5. Washington, DC: The National Academies Press.

o National Research Council (2011). Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press.

o National Research Council (2012). Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. Washington, DC: The National Academies Press.

o Rothwell, J. (2013). The Hidden STEM Economy. Washington, DC: Brookings Institution Metropolitan Policy Program.

o Rutledge, J.C., Carter-Veale, W.Y., and Tull, R.G. (2011). “Successful PhD Pathways to Advanced STEM Careers for Black Women.” In H.T. Frierson and W.F. Tate (Eds.), Beyond Stock Stories and Folktales: African Americans’ Paths to STEM Fields (pp. 165-209). Bingley, UK: Emerald Group Publishing Limited.

o Excelencia in Education and the United Negro College Fund (2014). Black and Brown: Institutions of Higher Education. Washington, DC.

o Santiago, D. (2012). Finding Your Workforce: The Top 25 Institutions Graduating Latinos in Science, Technology,

26

Engineering, and Math (STEM) By Academic Level 2009-2010. Washington, DC: Excelencia in Education.

o Corbett, C., and Hill, C. (2015). Solving the Equation: The Variables for Women’s Success in Engineering and Computing. Washington, DC: American Association of University Women.

o National Academy of Engineering and National Research Council (2012). Community Colleges in the Evolving STEM Education Landscape: Summary of a Summit. Washington, DC: The National Academies Press.

o Annual reports of the Congressionally mandated Committee on Equal Opportunity in Science and Engineering (CEOSE).

o Committee on STEM Education, National Science and Technology Council (2013). Federal STEM Education 5-Year Strategic Plan.

o Thomas, N.R., Poole, D.J., and Herbers, J.M. (2015). “Gender in Science and Engineering Faculties: Demographic Inertia Revisited.” PLoS ONE 10(10).

o Jaggars, S.S., Fink, J., Fletcher, J., and Dundar, A. (2016). A Longitudinal Analysis of Community College Pathways to Computer Science Bachelor’s Degrees. Mountain View, CA: Google Inc. Retrieved from http://goo.gl/Eiz33G.

o Harmon, N. (2012). The Role of Minority-Serving Institutions in National College Completion Goals. Washington, DC: Institute for Higher Education Policy.

o Google Inc. and Gallup Inc. (2016). Diversity Gaps in Computer Science: Exploring the Underrepresentation of Girls, Blacks and Hispanics. Retrieved from http://goo.gl/PG34aH.

• Recommendation: With more URM students and faculty attending

and teaching at non-MSIs, additional attention and resources should be paid to finding creative ways to impact URM populations at non-MSIs. Response: As the Committee recognizes, some HRD programs have a Congressional mandate to support specific types of MSIs. But examples of awards focusing on underrepresented students and faculty at non-MSIs can be readily found in the ADVANCE, AGEP, LSAMP, and TCUP programs, and the solicitations of all HRD programs are supportive of broadly applicable, innovative approaches for engaging students and faculty from groups underrepresented in STEM fields. In accordance with the missions of the individual programs, HRD staff will continue to look for appropriate ways to impact URM faculty and students at non-MSIs.

• Recommendation: A focal point and driver for the agency should be

supporting translational and transformative research on the participation of underrepresented groups in STEM – students, leaders, researchers, etc. – and how to encourage the use of research-based best practices across fields/sectors in innovative ways and new contexts. Specific attention should be paid to the research supporting groups and institutions facing barriers to participation with federal STEM research and education programs.

27

Response: The division will continue to provide guidance for the Broadening Participation and Institutional Capacity component of the EHR Core Research (ECR) program, which spans all four divisions in EHR. That is the primary locus at NSF for the type of research described in the Committee’s recommendation. Minority-serving institutions are encouraged to explore research topics and workshops that support capacity building, with a special focus on developing faculty to carry out STEM education research. The HBCU-UP program features two tracks that focus on research on broadening participation of underrepresented groups: Broadening Participation Research Projects and Broadening Participation Research Centers.

• Recommendation: HRD should expand its focus on institutionalization and institutional change (specifically faculty retention) to build and sustain leaders who are willing to invest in the sustainability of projects and programs – for example, ADVANCE at the University of Michigan, University of Washington, Montana State University, University of Wisconsin, Louisiana State University, and Jackson State University.

Response: As the Committee notes, institutionalization and institutional change are critically important to the success of ADVANCE projects. The TCUP and HBCU-UP programs also emphasize institutional transformation by supporting the research infrastructure and faculty research at institutions. Sustainability of best practices is generally an important component in all HRD programs. We will look for additional ways to stress the importance of sustained institutional change in all HRD programs.

Data Source: Jackets 11. Additional comments on the quality of the projects or the balance of the portfolio:

• It is important that HRD continue to lead broadening participation initiatives for the Foundation. However, if the nation is to achieve its goals in building a competitive workforce of the future, every directorate must be involved as a partner.

• Our perception is that there are many well-intentioned dominant groups that seek guidance to broaden participation. A successful example of this is the ADVANCE Partnerships for Learning and Adaptation Networks (PLAN) track awardee “Advocates and Allies” at North Dakota State University, who are moving the needle. The dissemination grants are invaluable.

• Recommendation: In order to expand the programs’ portfolios, we encourage HRD to intensify outreach and technical assistance on the proposal preparation and review process to institutions and PIs who have not historically been successful in securing funding from NSF. One strategy might be to hold NSF Days at institutions that have not historically been well-funded. Response: We share the Committee’s interest in utilizing NSF Days and similar outreach events to reach potential PIs and institutions that

APPROPRIATE

28

have not had much involvement with NSF. NSF Days were held recently at Spelman College and Delaware State University (two HBCUs), and HRD program officers participated in both events. We will consult the NSF staff who coordinate the NSF Day events to determine how minority-serving institutions can increase their role as hosts for and participants in these important regional events. The HBCU-UP program organized NSF Day-like activities at Jackson State University and Bowie State University. In addition, program officers have hosted webinars on NSF programs and proposal writing for faculty at MSIs, and program officers have given similar presentations to groups of faculty from MSIs at various meetings. HRD program officers will continue to organize events like these to the extent that their schedules will allow. The division has also funded some knowledgeable grantees to provide technical assistance to less experienced institutions. Examples include:

• Award HRD-1313529, QEM Network, “Leadership Development and Technical Assistance to TCUP Faculty and Students” — provides workshops for faculty and staff at TCUP institutions to increase their familiarity with NSF funding opportunities, NSF's proposal review process, strategic planning and sustainability strategies for projects, financial management, and other topics.

• Award HRD-1450661, Science Foundation Arizona, “KickStarter Program for Hispanic-Serving Community Colleges” — provides technical assistance to improve the competitiveness of Hispanic-serving community colleges in Federal grant programs in STEM fields.

• Award HRD-1458467, QEM Network, “Workshops for Improving Grant Writing Competitiveness for Faculty at Historically Black Colleges and Universities” — provides workshops to over 250 STEM faculty at HBCUs and other MSIs to encourage successful proposals to the HBCU-UP program and the Major Research Instrumentation (MRI) program.

• Award HRD-1741718, QEM Network, “Workshops to Broaden the Participation of HBCU STEM Faculty in NSF’s Education Research-focused Programs” — provides two workshops and a series of webinars to faculty at HBCUs to enhance their ability to prepare proposals that incorporate effective pedagogies and culturally sensitive interventions from STEM education research.

We will look for other opportunities to harness outside resources to provide valuable mentoring and expand the community of grantees.

• Recommendation: In a previous COV report, there was a suggestion

that CREST institutions be paired with Engineering Research Center (ERC) proposers. The COV encourages HRD to move forward with this recommendation.

Response: Several CREST Centers have established partnerships with NSF Centers such as the Engineering Research Centers, the Science and Technology Centers, and the Materials Research Science and Engineering Centers. The CREST program will continue to explore and establish partnerships with its sister centers programs throughout NSF.

29

• Recommendation: Internally, POs should conduct outreach across

the Foundation so that POs in the research directorates know more about how they might collaborate and co-fund HRD programs and broadening participation projects in general.

Response: HRD program officers have already begun this educational effort through several “brown bag” discussion sessions, such as the Broadening Participation Journal Club and the Investing in Diversity seminar series. HRD will continue to sponsor these discussions, which so far have been well-attended by staff from all parts of NSF.

• Recommendation: Institutions with emerging research capacity, which show promise of becoming competitive, might benefit from an ADVANCE Catalyst-like track within CREST to build additional capacity.

Response: In outreach efforts, CREST program officers will pay special attention to “emerging” institutions to ensure that those institutions are encouraged and prepared to submit proposals to NSF. Since 2003, the CREST solicitation has included the HBCU-RISE opportunity, which attempts to build the research capacity of HBCUs (see NSF 03-579 and subsequent solicitations). Many “emerging” institutions have been added to the CREST portfolio through the program’s HBCU-RISE component, which has a capacity-building function analogous to the ADVANCE program’s Institutional Transformation Catalyst track. Although we realize that HBCU-RISE is limited to HBCUs, it is not likely that CREST will add another (broader) track in the near future.

30

OTHER TOPICS 1. Please comment on any Division or program areas in need of improvement or gaps (if any)

within program areas.

• Areas for improvement include: additional attention to persons with disabilities within HRD portfolios; intersectionality; HBCUs, HSIs, and Tribal Colleges in the ADVANCE portfolio; HBCUs and HSIs in the AGEP portfolio; men faculty of color; SBE sciences in areas where men and women of color are underrepresented; and community colleges across all HRD programs.

• Women of color, in particular, inadvertently become invisible when programs focus exclusively on women or minorities. Consider paying close attention to this group.

• NSF needs to be careful to disaggregate historically underrepresented populations of interest appropriately and to monitor the extent to which proposals target specific underrepresented demographic groups (e.g., address variation of ethnicity within racial groups), in order to ensure that multiple demographic groups are reached by its programs.

2. Please provide comments as appropriate on the performance of the Division and its programs in

meeting goals and objectives that are not covered by the above questions.

• Goals and objectives for each program can be confusing/inconsistent or thin – going through each new announcement left us wondering, “What are you trying to do and what are the outcomes?”

• Recommendation: Recognizing that logic models are still under construction, the COV strongly recommends that work continue on these models in such a way that they reflect a level of consistency across programs and a careful alignment among problems, activities, and outcomes. External advice and consultation may be helpful in this regard, including input from the EHR Advisory Committee. Response: As noted earlier, HRD will revisit the program-specific logic models in collaboration with experts in EHR’s Evaluation and Monitoring Working Group.

• Recommendation: Critical self-reflection is very important to the planning and management

of programs and to strengthening the clarity and transparency of outcomes. We encourage HRD to continue to engage in critical self-reflection.

Response: HRD has held all-hands meetings, scientific staff discussions on various aspects of program management, program planning meetings with HRD’s leadership, and HRD staff retreats and will continue to do so, because these are important opportunities for self-reflection. In addition, one HRD staff meeting each month has been dedicated to scientific discussions relevant to the division’s work.

3. Please identify agency-wide issues that should be addressed by NSF to help improve the

performance of the Division or its programs.

• We applaud the efforts of HRD – including the huge impact programs such as LSAMP and ADVANCE have had on broadening participation.

• HRD has been a resource for NSF-wide programs, and we applaud that HRD provides broadening participation expertise to other divisions, provides suggestions for reviewers with appropriate expertise, and provides opportunities for joint funding across NSF and other agencies, both inside and outside the government.

• The proportion of the population from minority groups is expected to grow, while the total population of working-age adults does not continue to grow. The success of minority

31

groups in STEM will be central to our success as a nation. Therefore, we will soon face challenges as a nation to conduct, apply, and implement research on how to best educate and employ historically underrepresented demographic groups. It cannot be overemphasized that the time for investment in research, policy, and practice about broadening participation is now. This is because the demographic change is taking place quickly, yet making the necessary structural transformations to respond to this change is likely to take longer.

• Recommendation: There is concern that budgets are flat but the number of Minority-Serving Institutions that are eligible for programs such as CREST is increasing. There needs to be general and broad recognition that the growth of the Latino sector of our population is huge and that engaging that sector in STEM is a national imperative. Response: The division acknowledges the important demographic changes occurring in the United States, particularly with the Hispanic and Latino populations. See the response to the first recommendation under Question IV.6 (pages 21–22), which describes HRD’s participation in Dear Colleague Letters focusing on Hispanic-serving institutions and NSF’s proposed new HSI program.

• Recommendation: NSF needs to intensify and accelerate broadening participation as a

priority across all divisions, with HRD underscoring its leadership role, developing and validating new models and strategies, and moving things forward.

Response: HRD recognizes its special role across the Foundation in underscoring the importance of broadening participation as a national priority in both fundamental research and workforce development. The division’s Strategic Plan reflects this role. With EHR as a co-lead directorate, the NSF-wide INCLUDES initiative (https://www.nsf.gov/news/special_reports/nsfincludes/) was launched in early 2016 as a multi-stage, multi-year effort to address the challenge of broadening participation in STEM nationwide. HRD’s lead role in this initiative is a testament to the division’s rich history and expertise in this area. In addition, as mentioned earlier, HRD program officers have already initiated outreach activities inside NSF through a variety of “brown bag” seminars, such as the Broadening Participation Journal Club and the Investing in Diversity seminar series. HRD staff plan to lead many more such activities.

• Recommendation: The agency is strongly encouraged to stay up-to-date and ensure

consistency across the Foundation regarding appropriate terminology in reference to various social identities. For example, indigenous people of Alaska are called Alaska Natives, not Alaskan Natives, and this particular terminology was noted incorrectly across eJacket, the eJacket COV Module, forms, and solicitation documents – with the exception of the TCUP, CREST, and AGEP solicitations.

Response: We appreciate the Committee’s observation, which will be relayed to others at NSF also. We will attempt to ensure that terminology in HRD documents is consistent and conforms with Federal standards and generally accepted usage.

• Recommendation: NSF should better articulate the focus on the role of and

intersectionality between multiple identities in STEM success, including the identities of race and disability, as well as gender. We believe that it is important to make sure that the inclusion of particular dimensions does not get “lost” when it is assumed that these dimensions are embedded in different programs.

Response: See the response to the fourth recommendation under Question III.2 (page 15).

32

• Recommendation: The other directorates should be encouraged to propose strategies to achieve the inclusive participation that NSF advocates.

Response: NSF INCLUDES, for which EHR is a co-lead directorate and HRD is the lead managing division, was intentionally designed as an NSF-wide initiative and aligned with goals and objectives in NSF’s Strategic Plan. All NSF directorates have a voice in the activities that are encouraged, a role in the operation, and a stake in the outcomes. Through HRD’s regular programs and outreach within NSF, the division also provides incentives to other directorates to explore strategies for achieving inclusive participation. For example, HRD hosts discussions on diversity through its Investing in Diversity internal seminar series and the Broadening Participation Journal Club. The AGEP program offers an opportunity to PIs with awards from NSF’s Directorate for Mathematical and Physical Sciences (MPS) whereby those PIs can request supplements for graduate students pursuing STEM research. The TCUP program has formed formal partnerships with three other directorates. Other HRD programs have co-funded projects in other directorates for increasing the participation of underrepresented minorities in STEM.

• Recommendation: Provide incentives for program officers to be more involved in

proposal development – perhaps webinars devoted to sharing information on successes of other efforts, best practices that may be adapted to proposed projects, and resources to assist in proposal implementation.

Response: HRD program officers already conduct a great deal of outreach to provide advice to potential proposal-writers. They give presentations and proposal-writing workshops around the country. During site visits, they not only ensure that funded projects are meeting their objectives but also offer guidance to PIs preparing new proposals. As mentioned earlier, the HBCU-UP program organizes its own version of NSF Days, which provide technical assistance to aspiring and long-time PIs at HBCUs to sharpen their skills for submitting proposals to programs throughout NSF. Program officers from other directorates participate in these events along with HBCU-UP program officers. HRD program officers also host webinars to share information about particular programs and about proposal-writing in general. They will continue all of these practices within staffing and time constraints and will look for ways to extend the practices to larger audiences. For example, improvements in videoconferencing technology have enabled effective webinars, a low-cost solution for interactive sessions that can reach hundreds of people at once across the nation. See also the response to the first recommendation under Question IV.11 (pages 27–28).

• Recommendation: Perhaps each of the program solicitations should require plans for institutionalization and sustainability of projects – next steps – so that a project would not be a one-time, short-term activity of an individual investigator.

Response: Currently, the solicitations for ADVANCE, AGEP, HBCU-UP, and LSAMP require proposals to include plans for sustainability and institutionalization, at least for some program tracks. These plans are typically shorter components within the proposal’s maximum 15-page Project Description, but they are important components of a successful proposal.

4. Please provide comments about major gaps or significant overlaps among the programs in the

Division.

• We did not find any overlaps; although we did identify areas/gaps that should be strengthened, particularly: community college engagement, support for students and

33

faculty with disabilities, and the advancement of URM faculty of color, especially women, in different institution types, including Minority-Serving and Predominantly White Institutions.

5. Please provide suggestions and comments on the approach and methods (presented during the

overview of programs) for assessing the impacts of HRD programs using CREST as a pilot for the study.

• Both HRD and NSF as a whole need to continue and accelerate the use of analytics to

research the impact of programs and demonstrate the results of investments; we applaud the “Deep Dive” effort.

• The CREST pilot is a good start at mining the extensive data that NSF has collected in years of annual and final reports from projects funded via these programs.

• We feel that understanding the participation of MSIs in multiple programs is of particular importance. For example, one could ask: If an institution develops a successful undergraduate program, is it more likely to then go on to develop a graduate program?

• Recommendation: We recommend making data sets more available to the public.

Response: NSF collects data through program-specific monitoring systems (for some programs, such as CREST and LSAMP) and project reports from PIs for the purpose of program management and program evaluation, not research. These data often concern the characteristics of individuals and are protected by the Privacy Act. It is usually costly and time-consuming to clean large datasets and ensure that no individuals can be identified when the data are analyzed. Because of this issue and because the data are not intended for research, NSF tends not to clean and release its project participant-related data. NSF primarily uses the data in internal reports and analyses and in program evaluations, and NSF sometimes publishes analyses and summaries (including evaluation reports) that address external interests.

• Recommendation: Let conceptualization of goals drive data collection and analysis and

subsequent assessment of progress toward goals. Response: The goals in HRD’s Strategic Plan were conceived with the understanding that the selection of data to explore outputs and outcomes would necessarily come subsequently. We will continue to improve our performance in this area. EHR’s Evaluation and Monitoring Working Group provides advice to the division on performance assessment.

• Recommendation: Be explicit about how knowledge is transferred across programs and

projects, as well as across the Foundation. We encourage NSF to continue to generate data and disseminate information broadly to show results and drive knowledge generation and sharing initiatives.

Response: We appreciate the Committee’s interest in generating data on activities in HRD’s programs and funded projects and sharing this widely. We believe that some of this interest is addressed in the annual report that NSF produces in response to three Executive Orders: Historically Black Colleges and Universities; Educational Excellence for Hispanic Americans; and the White House Initiative on Asian Americans and Pacific Islanders. In addition, NSF provides an annual report to Congress on support to Hispanic-serving institutions. For use within NSF, the Foundation has established an internal SharePoint site on broadening participation, which contains data and reports for all agency staff to use in program development and tracking program activity related to minority-serving institutions. With regard to grantees’ dissemination of data from their NSF-funded research, NSF has a policy on the dissemination and sharing of research results

34

(https://www.nsf.gov/bfa/dias/policy/dmp.jsp) and a policy on public access to results of NSF-funded research (https://www.nsf.gov/news/special_reports/public_access/). NSF requires PIs to share with other researchers, at no more than incremental cost and within a reasonable time, the primary data, samples, physical collections, and other supporting materials created or gathered in the course of work under NSF grants, and NSF expects grantee organizations to encourage and facilitate such sharing. In 2011, NSF began requiring every proposal to include a “Data Management Plan” describing how the proposal will conform to NSF’s policy on the dissemination and sharing of research results. In 2016, NSF implemented a “public access policy” covering articles in peer-reviewed journals, juried conference papers, and data that result from NSF funding. This policy requires that PIs deposit a copy of their (NSF-funded) peer-reviewed journal publications or juried conference papers in NSF’s Public Access Repository, and requires that they make data and software available as described in their proposal’s Data Management Plan.

• Recommendation: Ensure a careful linkage between ongoing data analytics efforts, division

goals, and anticipated outcomes.

Response: We continue to improve our performance in this area. In 2016, HRD created a new position for a Data Scientist and hired a person who has training in both social science research methods and evaluation. The Data Scientist works with HRD program officers and other staff to conduct analyses directly linked to both program and division goals. Our data analytics efforts are intended to provide evidence for HRD’s decisions and future program evaluations, as well as an increased understanding of program- and division-level outcomes.

• Recommendation: Take steps to align logic models and program monitoring data systems –

ensuring that measurable outputs and outcomes are clearly articulated in each and that there is a careful alignment of identified problems/situations with activities and outcomes.

Response: We will revisit the alignment of the program-specific logic models with monitoring systems and will make adjustments and clarifications where needed. EHR’s Evaluation and Monitoring Working Group assists us with these issues, in coordination with the division’s Data Scientist.

• Recommendation: Take steps to improve grantee compliance with changes to data

monitoring systems by being more explicit in solicitations regarding what is expected from a data collection standard point and by making sure to follow up on the data management plans included within the proposals.

Response: In addition to NSF’s guidelines for preparing the Data Management Plan that is required in each proposal, each directorate, including EHR, has published supplementary guidance that illustrates features of Data Management Plans relevant to research in particular disciplines or disciplinary clusters. In outreach and in meetings with prospective PIs, HRD staff will stress the importance of the Data Management Plan and will discuss the content that is generally appropriate for HRD programs.

6. Please provide comments on any other issues the COV feels are relevant.

• HRD personnel have done a tremendous job maximizing the impact of each program’s budget, despite inadequate budgets and heavy workloads.

• The COV commends HRD for its pilot study because we are at a critical time where we need to identify what has been accomplished to date and what has worked well, for whom, and in what context. That data is critical to these questions and scaling up efforts that work.

35

7. NSF would appreciate your comments on how to improve the COV review process, format and report template.

• This COV supports the use of division-wide rather than program-specific COVs, as division-

wide COVs enable the review of individual programs and also allow the review of the portfolio of programs across the entire division, build coherence, and provide the opportunity to align individual programs more directly to the strategic objectives of EHR and the Foundation at large. The advantage of the HRD-wide COV is the overall picture it gives us and the synergies, themes and holes that are identified. However, a downside of the HRD-wide COV is the limited attention that can be paid to the individual programs and the dependence of the COV on the previous experience individual COV members have with each of the individual programs.

• All of the materials provided were very useful, especially the management plans and cheat sheet.

o Consider presenting data and information in a more user-friendly format. o The table “Define the need – address the challenge” was very helpful – we could

imagine this being much more detailed to assist COV members in visualizing the coverage of the different programs.

• It was very difficult to find data and information required to answer some of the questions. The data sources noted for individual questions within the annotated COV Report Template did not always match the data file names. We recommend that the guidance in the template be revised to reflect the locations and names of documents in the eJacket COV Module.

• The step-by-step instructions regarding what to read first, second, etc., were extremely helpful.

• It would be helpful to receive a 5 or 10 minute overview of each of the programs up front. • Ensure that each COV has a member of the target group on the committee to represent that

group (maybe a professional society committee chair, vendor, think tank or the like) and to learn more about the attractiveness of the programs to the targeted group.

The Committee of Visitors is part of a Federal advisory committee. The function of Federal advisory committees is advisory only. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the Advisory Committee, and do not necessarily reflect the views of the National Science Foundation. SIGNATURE BLOCK:

______________________________________ Francisco C. Rodriguez Chair For the Committee of Visitors for the Division of Human Resource Development