a research cyberinfrastructure strategy

A Research Cyberinfrastructure Strategy for the CIC: Advice to the Provosts from the Chief Information Officers

May 2010


2 | P a g e

Table of Contents

Executive Summary .............................................................................................. page 3

Introduction ..................................................................................................... pages 4-5

What is Cyberinfrastructure?.......................................................................... pages 5-7 How Does Campus Cyberinfrastructure Interact with National and Global Cyberinfrastructure? …………………………………………………………... pages 7-10 What Best Practices Can Be Identified for Campus Cyber- infrastructure?....................................................................................... pages 10-15 What Specific Actions and Investments are Timely Now?..................................................................................................... pages 15-18 How Can CIC Be Leveraged for Better Value from Cyberinfrastructure Investment?.................................................................................................pages 18-20 Notes……………………………………………………………………………………………….………pages 21-22

CIC Chief Information Officers

Sally Jackson University of Illinois at Urbana-Champaign

Klara Jelinkova University of Chicago

Ahmed Kassem University of Illinois at Chicago (retired May 1, 2010)

Brad Wheeler Indiana University

Steve Fleagle University of Iowa

Laura Patterson University of Michigan

David Gift Michigan State University (Chair)

Steve Cawley University of Minnesota

Mort Rahimi Northwestern University

Kathy Starkoff Ohio State University

Kevin Morooney Pennsylvania State University

Gerry McCartney Purdue University

Ron Kraemer University Wisconsin-Madison


3 | P a g e

Executive Summary

Prominent researchers in many fields have written about the disciplinary imperatives behind investment in cyberinfrastructure for research and education. In “A Research Cyberinfrastructure Strategy for the CIC,” we provide a CIO perspective on how to respond to these imperatives, including ideas about how to monitor the changing environment and continuously adjust to it. Our goal should be to enable scholarship at the cutting edge of every discipline, while getting as much value as possible from every dollar spent on cyberinfrastructure. The CIC campuses are very richly endowed with cyberinfrastructure resources but can be even more effective by adopting good practices that support greater coordination at every level:

Plan. Share (at the highest level possible). Design funding models that promote scholarship and stewardship. Rely on user governance. Conduct cyberinfrastructure impact analysis.

Over the long run, these practices should help our institutions produce more scholarship per dollar invested. In the near term, the perspectives underlying these recommended practices allow us to identify six high-priority enhancements of campus cyberinfrastructure that will be necessary if we are to meet the present needs of researchers and the expectations of funding agencies. We should all currently be investing in:

Preparing for “federated identity management” and other enabling technologies for virtual organizations.

Maintaining state-of-the-art communication networks. Providing institutional stewardship of research data. Consolidating computing resources while maintaining diverse architectures. Expanding cyberinfrastructure support to all disciplines. Exploring cloud computing to manage long-term financial commitments.

The CIC’s well-established pattern of IT-related collaboration puts our institutions in a position to contribute more together than the sum of our individual contributions. We recommend leveraging this strength to amplify the value of every dollar of cyberinfrastructure investment along with the following Provostial actions:

Build interest among other Provosts in federated identity management. Review recruiting and retention practices that affect campus

cyberinfrastructure. Continue supporting competition as a consortium. Influence federal spending rules.


4 | P a g e

Introduction

Meet Donna Cox, the director of Illinois’ eDream Institute. A leading figure in digital arts, Professor Cox has not simply appropriated digital media for artistic expression but has engaged deeply with the question of what art is for. She collaborates with computational scientists to help them gain insight into their data through visual representation, and her related outreach activities (such as exhibitions, documentaries, and animated media) aim to make these insights fully accessible to a broader public. In other words, she has actually redrawn her discipline’s boundaries to include creation of scientific understanding and meaningful translation of that understanding for nonscientific audiences.i Meet Ruth Stone, Professor of Ethnomusicology and director of Indiana’s Institute for Digital Arts and Humanities. Professor Stone has led multiple grants that are creating new tools and skills for humanities research. The thousands of hours of video she and her team have collected through field work from all over the world requires massive digital storage systems, and the data must be made available to other scholars to code for use in education and research. Her pioneering work on video software was recently adopted by the Kelley School of Business for use with their online courses.ii As we think about how we invest in technology to support research, education, and outreach, we need to think beyond stereotyped images of computational research. We need to think about people like Donna Cox and Ruth Stone: new kinds of scholars doing new kinds of scholarship.

Along with our most inventive scientists and engineers, these new kinds of scholars require access to a staggering array of advanced resources: high performance computing systems that can model enormously complex processes, huge data sets formed from instruments generating countless observations, advanced visualization tools that help create meaning from otherwise unfathomable complexity, sophisticated telecommunications networks capable of moving large streams of data and supporting synchronized distance interaction, and collaboration support platforms that allow formation of virtual organizations drawing experts from many fields working all over the globe. Though only a small number of really creative people generate this level of demand on an institution’s support structure, many others have special needs of some kind. The perfect poster child for 21st Century scholarship is one whose work cuts across intellectual disciplines and depends on assembling a complete cross-section of what is now known as cyberinfrastructure.

Science and scholarship drive the growth of cyberinfrastructure, often through grants to individuals and teams, because cyberinfrastructure components are vital to cutting-edge research. But external funding for a project never provides all that is needed by the project. Cyberinfrastructure is heavily “layered,” and institutions bear

Wikipedia defines

cyberinfrastructure as

“the new research

environments that

support advanced data

acquisition, data

storage, data

management, data

integration, data

mining, data

visualization and other

computing and

information processing

services over the

Internet.”

http://en.wikipedia.org/wiki/Data_acquisition

http://en.wikipedia.org/wiki/Data_acquisition

http://en.wikipedia.org/wiki/Data_storage_device

http://en.wikipedia.org/wiki/Data_storage_device

http://en.wikipedia.org/wiki/Data_management

http://en.wikipedia.org/wiki/Data_management

http://en.wikipedia.org/wiki/Data_integration

http://en.wikipedia.org/wiki/Data_integration

http://en.wikipedia.org/wiki/Data_mining

http://en.wikipedia.org/wiki/Data_mining

http://en.wikipedia.org/wiki/Data_visualization

http://en.wikipedia.org/wiki/Data_visualization

http://en.wikipedia.org/wiki/Computing

http://en.wikipedia.org/wiki/Service_(economics)

http://en.wikipedia.org/wiki/Internet


5 | P a g e

the responsibility for continuously building layers that researchers and their funding agencies expect. If we want to produce as much creative work as possible for every dollar spent, we must respect and leverage these layers, planning each new investment to contribute as much as possible to, and cohere as completely as possible with, a continuously advancing global cyberinfrastructure for research and education.

What is Cyberinfrastructure?

Cyberinfrastructure includes data networks, computational facilities and computing resources, large data sets and tools for managing them, and specialized software applications including middleware.iii Arguably, it also includes people: technicians involved in creating and operating all of these resources and specialized consultants who assist researchers in using the resources.iv Some of these resources are owned by individual campuses and serve the researchers at that campus; many others are shared across campuses and made available through competitive allocation processes. Cyberinfrastructure encompasses many elements (see figure below) of what is normally thought of as research computing or research technology, but extending far beyond any physical or organizational boundary.

Figure adapted from Educause-Live presentation by Curt Hillegas.v


6 | P a g e

Networking. Data networking for research and education is more complicated than just being connected to the Internet. The typical arrangement for a research university is a campus network connected to a regional Research & Education network that connects to a national R&E network (such as Internet2, National LambdaRail, or ESNET). The CIC is heavily invested in all layers of R&E networking. Individually and as a consortium, we have attempted to exert influence on the direction of national R&E networking, sometimes succeeding and sometimes failing. Computing. Every research university must somehow address the needs of its researchers for high performance computing (known to all computational scientists as HPC). HPC is not defined by a specific set of system characteristics, but by position with respect to the leading edge: HPC is not restricted to the "top of the pyramid," but nor is it inclusive of all supercomputing. For example, very small computer clusters in a researcher's lab are probably not HPC resources; however, these small clusters, which are very common at all of our institutions, play an important part in an institutional strategy and must be considered while planning how to meet needs at the leading edge. When we talk about investment in HPC, we are talking about investing in capabilities, not just buying objects. HPC does take place on hardware, in some sort of operating environment, but in addition, HPC requires appropriate software applications and code development, technical consulting and support, and often, training for the computational scientist. Buying a machine is definitely not the same as investing in HPC; a credible HPC strategy will involve balanced buildout of a whole complex system. Data. New technologies have greatly amplified the amount of data being collected, analyzed, and saved for re-use. All disciplines need access to data storage, and beyond that, all disciplines need data curation and stewardship. In many fields, the most important digital assets are large datasets that must be accessed remotely. High-profile examples of large datasets held within our institutions include the Hathi Trustvi collection of digitized library holdings and the data to be transmitted by the Large Synoptic Survey Telescope currently being constructed in Chile.vii Of course our researchers also require access to datasets produced and stewarded elsewhere, and in these cases the researcher’s home institution is responsible for the infrastructure support needed to gain access. Applications. The applications required for 21st Century scholarship go far beyond number-crunching. Some of the most dramatic breakthroughs in the disciplines are being achieved through advances in visualization, making this area an important priority for investment. Staffing. Virtually all cyberinfrastructure-enabled research depends on skilled technical staff, often PhD-trained in the discipline. As noted above, this is particularly true of high performance computing, which requires special programming for parallel processing. “Blue Waters” is the world’s first petascale computer, being built at Illinois with an NSF award that involves all of the CIC institutions as partners. To take advantage of the Blue Waters architecture, researchers will need significant improvements in software code. Part of the preparation for Blue Waters is to train a


7 | P a g e

new generation of programmers, which the CIC is doing through its Virtual School for Computational Science and Engineering.viii Beyond research programming, there is an increasing need for new kinds of expertise in helping researchers negotiate the complexity of the sociotechnical environment.ix

How Does Campus Cyberinfrastructure Interact with National and Global Cyberinfrastructure?

As illustrated in the accompanying figure,x research support depends on “stacked” technology components. The stack differs from discipline to discipline, and even from researcher to researcher. Ideally, every researcher should experience all

resources needed for his or her research as a single fully-integrated system. Very important background knowledge for every Provost is how campus cyberinfrastructure components interact with components above the campus level. NSF and many higher education organizations are deeply interested in the relationships among research support resources at all levels.xi The network as experienced by the researcher is, in nearly every case, the campus network connected to a regional network connected to a national backbone connected to other worldwide networks; the better the alignment among the providers of each link, the better the experience for the

researcher. We are very privileged in the CIC to have nearly seamless relationships between the campuses and the CIC-controlled OmniPoP, a regional connector to Internet2, the primary national research and education network backbone (headquartered at Michigan, with Network Operations Center managed by Indiana). We have had a more troubled relationship with National LambdaRail, a national network backbone from which we withdrew in 2008 after a decision by the National LambdaRail Board to take on a significant commercial partnership. StarLight, a primary point of connection of US networks with global research and education networks, is operated by Northwestern. With the ascent of high-speed networking, computing resources have become nearly place-independent. A researcher in need of high performance computing does not necessarily need direct local access to computing equipment. Many computing resources are made available to the science and engineering community through national labs, national supercomputing centers, and TeraGrid (an NSF-sponsored project coordinating mechanisms for individual HPC resources). Researchers can request allocations on any of the individual computing resources that are part of TeraGrid, matching system characteristics to their computational problems, using the TeraGrid User Portal (http://portal.teragrid.org) a snapshot of which is shown on the next page).


8 | P a g e

Illinois, Indiana, and Purdue are TeraGrid resource providers. All of the CIC schools are TeraGrid users, and collectively we account for 20% of all usage and 20% of all researchers with allocations on the TeraGrid (see Table below showing snapshot of CIC usage for one recent quarter). What any individual institution provides to its own researchers should be understood as complementary to what is available through these national resources.

The CIC campuses are rich in computational and communications resources, housing many of the above-campus resources that support scholarship worldwide (NCSA,

TeraGrid, Nanohub, StarLight, Internet2 headquarters, the Global Research Network Operations Center (the “NOC” for both Internet2 and National LambdaRail and many others) as well as resources dedicated to their own researchers. Above-campus resources are critically important, especially in defining the cutting edge, but they do not meet all needs even in the relatively narrow realm of high performance computing. NSF’s Cyberinfrastructure Vision for 21st Century Discovery points toward federal investment in leadership-class resources (typically resulting in an above-campus service) and makes clear that individual research institutions are expected to complement these resources with as much additional capacity as their researchers need. And this capacity should be architecturally diverse, because not all scientific problems lend themselves to the same computational methods.

TeraGrid Usage

2nd Quarter 2009

Institution CPU usage Users

Illinois 632 M 234 Indiana 389 M 19 Penn State 92 M 15 Wisconsin 51 M 26 Michigan State 42 M 22 Minnesota 33 M 18 Chicago 28 M 61 Michigan 23 M 12 Purdue 21 M 43 Northwestern 14 M 37 UI-Chicago 9 M 16 Iowa 8 M 16 Ohio State 3 M 1


9 | P a g e

The accompanying diagram (courtesy of Illinois computer science professor Michael Heath) illustrates one of the most crucial architectural distinctions among HPC systems: whether memory is associated with each processor (“distributed memory”), or globally available to all processors (“shared memory”). The clusters common on all our campuses today are distributed-memory systems, often created by assembling commodity computers and connecting them with a dedicated very high-speed network. Distributed-memory architectures work well for a wide range of

algorithms, in which a part of the algorithm, accompanied by the needed data, is handed off to each processor. However, not all algorithms map easily to a distributed-memory architecture. For example, algorithms whose data-access patterns are random or not easily predicted work best on a shared-memory architecture. Because shared-memory systems are much more expensive than distributed-memory systems, they tend to be national, not campus-specific, resources, even though they may be located on a campus. Distributed-memory systems are less expensive, and they can also be incrementally expanded, so they can be purchased with a small investment and then expanded as needs and resources warrant. Distributed-memory clusters remain the norm at American research universities, and we should plan for this to be so for some time to come, partly because of their cost-effectiveness, but partly for operating advantages associated with not having to share (that is, with being able to command full use of the resource for extended periods of time, without waiting in a queue). This said, there are better and worse ways of managing computing cluster inventory for a campus, a point to which we will return shortly. “High performance” is often contrasted with “high throughput.” High throughout computing (HTC) includes many clusters, but also includes other ingenious ways of expanding total capacity, notably those that aggregate surplus cycles from many heterogeneous and geographically distributed systems. Wisconsin researchers created Condor (http://www.cs.wisc.edu/condor/htc.html), one of the most widely used software programs for aggregating cycles from large numbers of individual systems. A researcher with the right kind of problem can submit jobs to a Condor pool, just as jobs can be submitted to any other system. Because these systems typically lack dedicated network connections and cannot guarantee availability of cycles for scheduling of jobs, they tend to be most usable where the problems involve loosely coupled operations. While Condor pools and other forms of grid computing are important ways to increase capacity, they are not equally well-adapted for all problem types. They do not eliminate the need for investment in the underlying systems, but they do offer the possibility of getting more cycles from any level of investment with little added cost. Most likely, any one campus will have significant numbers of users for each of these various forms of HPC and HTC; it is inadvisable to structure decisions about HPC and HTC around majority rule, because this is not a mere matter of taste or brand


10 | P a g e

preference but a matter of matching architecture to problem structure. In sum, every institution should begin its planning for campus cyberinfrastructure investments with the most comprehensive picture possible of the national and global cyberinfrastructure to which any new resource will add, and with the richest possible understanding of problem diversity within the institution’s own mix of research programs. While a campus should not assume that all needs can be served by above-campus resources, neither should a campus duplicate the capabilities of these resources. For HPC in particular, the goal should not be to have the biggest or fastest supercomputer, but to extend the capabilities of the institution's researchers, building campus resources that complement and connect nicely to above-campus resources.

What Best Practices Can be Identified for Campus Cyberinfrastructure?

Attempting to describe best practices for managing campus cyberinfrastructure is a little like attempting to describe best practices for faculty recruitment or retention: There is a changing competitive landscape that is also highly variable by discipline, and specific tactics that give the first adopter a competitive edge may become huge liabilities for higher education as a whole once everyone tries to use the same (formerly advantageous) tactic. The best current thinking in cyberinfrastructure centers on the growing importance of coordination and cooperation, and on the crippling cost of trying to gain competitive advantage through accumulation of material. We recommend five practices we consider to be well-justified, even though they lack the experience base to be described as industry best practices. Good Practice 1: Plan. Maintaining a viable campus cyberinfrastructure is an ongoing process of responding to the co-evolution of technology and the scholarship it enables. This requires context-sensitive planning. If no other planning framework has yet taken hold on the campus, planning for cyberinfrastructure can be modeled loosely on the processes many institutions use for creating long-term facilities master plans. This planning model is appropriate because it does not outline actions to take in a predetermined

Five Good Campus Practices for Managing Cyberinfrastructure

1. Plan.

2. Share (at the highest level possible).

3. Design funding models that promote scholarship and stewardship.

4. Rely on user governance.

5. Conduct cyberinfrastructure impact analysis.


11 | P a g e

order but focuses on keeping many independently motivated projects aligned with an overall direction for the institution. For facilities, individual building projects constantly amend the campus master plan. The role of the master plan is to assure that, in addition to being evaluated programmatically, new buildings are evaluated for their fit with an overall aim and for their impact on other future possibilities. We can think of a cyberinfrastructure master plan as a methodology a campus uses for planning to meet the ever-changing IT needs of its researchers. This methodology would include continuous projection of technology directions and disciplinary trends (analogous to projection of shifts in student enrollment and student demographics), and it would include continuous updating of a campus cyberinfrastructure “map” to track significant changes in the environment and expose the preparation needed to accommodate these changes. The campus CIO should be the steward of this cyberinfrastructure master plan as the CFO is normally the steward of the facilities master plan. Some universities have such plans, but we know from in-depth conversations with peers across the nation that many research universities do not plan their cyberinfrastructure the way they plan for other institutional assets; they simply leave it to individual researchers to meet their own needs with little constraint, little commitment, and little or no review, from the campus. Researchers with high-end computational needs are assumed to be capable of attracting external funding, and when they do so, they are often left to manage the computational resources within their own labs, often without prior assessment of either the adequacy of power and communications infrastructure and without review of financial impact on the campus. This prevailing practice is becoming less viable over time: Letting cyberinfrastructure grow primarily from successful funding proposals enriches our campuses, but it also places stress on other elements of campus IT and adds needless operating cost that can accumulate to very large numbers in any university with significant research activity. Good Practice 2: Share. Share at the highest possible level, especially for high performance computing. As we have pointed out, coherent connection to national and global cyberinfrastructure is far more important today than is having individual control over resources. Sharing is not only a good way to control costs, but also a way to improve the resource itself. Resources that become more valuable as more people adopt them include communication platforms (where increased adoption means that each individual can reach more people), software (where large communities of coders and users can lead to much more rapid and sustainable development), and large community-curated datasets (where the more people using the dataset, the more layers of information build around the data). Some resources actually lose value if they become too locally inflected; for example, value is diminished for datasets, applications, and services that use idiosyncratic methods of controlling access. One resource type particularly receptive to sharing is cluster computing. Clusters


12 | P a g e

maintained independently within individual labs are very common, and federal agencies encourage the proliferation of clusters (inadvertently, through how they make awards for computing equipment). Campus leadership also encourages researchers to build individual clusters through common practices in the awarding of startup funds. The clusters themselves are quite affordable at the level of individual labs, but they can have huge unanticipated costs to institutions if allowed to proliferate all over campus. When located in a researcher’s lab space rather than in purpose-built data centers, computing clusters not only take up more volume but also tend to increase energy costs and other costs of operation. In response to these issues, new models of cluster computing have emerged in which computational nodes are contributed to large shared systems by many individual researchers. In these new models, each researcher has direct control over his or her own resources, but when the resources are not being used by their “owner” they revert to a pool that can be used by others. This is not only beneficial for the contributors, but also far more benign for the institution. Purdue has been particularly forward in developing models for shared cluster computing, and reception by faculty has been very positive. Earlier we introduced a distinction between distributed-memory architectures and shared-memory architectures. Shared-memory systems are, as we pointed out earlier, much more expensive than clusters relative to their performance. While many institutions still invest in such systems, the largest and most powerful shared-memory systems are (and should be) shared across institutions at national or consortial supercomputing centers. A good institutional strategy will include many forms of sharing with many levels of aggregation. HPC facilities may be shared by a few researchers, perhaps doing related work, or may be shared at the institutional level, or at the multi-institutional level among several cooperating universities, or at national or even international levels. Later we will make a specific recommendation for creating shared clusters at the campus or near-above-campus level, for any institution that has not already done so, and we will make another recommendation for looking at cloud computing as a form of demand aggregation. Institution-level investments can also make it easier for researchers to use national-level resources, and one of these is preparing to manage login credentials in a less cumbersome way. Good Practice 3: Design Funding Models That Promote Both Scholarship and Stewardship.

While most research universities now have funding models that work adequately for maintaining their data networks, fewer have well-worked-out models for other components of campus cyberinfrastructure such as HPC, data storage and management, and specialized technical support. This is a serious concern, because a funding model is not just a way to pull in the money needed to operate the resource, but also a system of constraints within which individual researchers must reason about how best to spend. Some funding models encourage spending that serves all interests well, and other funding models encourage spending with negative


13 | P a g e

unintended consequences. Most university CIOs now believe that legacy funding models for HPC are responsible for significant fragmentation of campus cyberinfrastructure. Traditionally, NSF and other funding entities have provided ready support to investigators to purchase their own HPC resources, and campus leadership has willingly allocated space for these resources. Besides leading to proliferation of small HPC facilities tucked into spaces that lack the power and cooling necessary to support HPC systems, this way of doing business also leads to fragmentation of computing operations, to ineffective management of research support workforce, and to higher risks in security and management of research data. Very commonly, project-specific computing resources use graduate students or post-docs as system administrators, diverting them from the science they should be doing. From an institutional perspective, this is undesirable for an additional reason: It works at cross-purposes with developing a stable professional competence in managing HPC resources. The Coalition for Academic Scientific Computing and the Educause Campus Cyberinfrastructure Working Group issued a joint report in 2009 urging that research universities and federal funding agencies work toward policies that encourage sharing to replace policies that encourage fragmentation.xii One approach that is already known not to work very well is to fund HPC resources centrally and charge fees for service. For one thing, this model limits usage to disciplines with ready access to research funds, closing out disciplines with few funding sources. Because the true cost of operating lab-level resources is often concealed from the researchers by hidden subsidies, a service offered at a fee set to recover true cost will look too expensive to researchers (even though it is in fact less expensive to the institution), and this appearance of costliness discourages use of the resource. Still worse, charging by use may have a dampening effect on innovation and on risk-taking with grand-challenge problems. Charging some users (those with grants) and not others (those without) runs afoul of research budget compliance and creates resentment over inequity. Bluntly, fee-for-service models do not encourage sharing; they would have financial advantages for the institution if researchers chose to use them, but researchers avoid them either because they have no discretionary funds or because they perceive the service as too expensive relative to what they could do on their own. Providing centrally funded HPC resources without fees for service solves the problem of providing for researchers without discretionary funds. It does not solve the problem of what to do with additional resources brought to the campus through grant funding. Institutions with centrally funded HPC systems often have large numbers of researcher-operated clusters as well. Buy-in approaches have been piloted at a few institutions (notably Purdue) with generally good results.xiii In a buy-in program, investigators may invest their personal grant funds for HPC in a shared HPC facility, perhaps purchasing a set of computational nodes for which they receive top priority of use (but which may be used by others when the “owners” are not using them). These programs have many


14 | P a g e

advantages for both the institution and the individual researcher. For the researcher, advantages may include system administration resources provided by the HPC center instead of (as is quite common) by a graduate student or post-doc; better security and data protection; access to the HPC network fabric and storage facilities (which researchers may not include when they price their grant budgets); access to shared software made available through the HPC center; access to specialized technical support personnel; and participation in a broader community-of-practice of the HPC users. For the institution, the major advantages are conservation of space and conservation of energy. Importantly, buy-in models can be flexibly managed to include both researcher-purchased resources and campus- or department-funded resources. A base system subsidized from institutional funds may be freely available to all users, while researcher-purchased resources are automatically allocated to their owners on demand. Any buy-in program must be structured to comply with federal research audit requirements regarding the use of grant funds, but there are already several working models at successful research institutions that have passed muster with funding agencies. Good Practice 4: Rely on User Governance. IT Governance refers to “decision rights and accountability framework for encouraging desirable behaviors in the use of IT.”xiv As noted earlier in this paper, many research universities leave spending decisions to anyone with money and hold no one accountable for the overall quality of campus cyberinfrastructure. While it may seem perfectly reasonable to allow researchers to spend their own grant funds as they choose, these expenditures may encumber the campus as a whole with greater expense (e.g., greater energy expense from location of computing clusters in unsuitable academic space). Some form of accountability now seems appropriate to weigh impact on campus against benefit to individual projects. But expecting researchers to give up individual control over resources carries with it an obligation to protect their interests through strong governance methods. Setting priorities among users is an especially critical governance task. This is well understood for shared HPC resources, where decisions have to be made not only about which computational architectures to provide but also about how to assign computing jobs to positions in a queue. Institutional values, goals, and purposes will affect governance in complex ways. For example, an institution may invest in shared HPC systems to increase their availability to all disciplines, or it may do so to support specific research thrusts, and these contrasting purposes will give rise to different ways of prioritizing use. Our sense is that virtually all academic disciplines are now finding HPC to be a useful tool for scholarship, and that most institutions will want to find ways to balance an interest in supporting research across the spectrum of disciplines with an interest in providing adequate support for high end users. Governance and queue/schedule management will need to address both sets of needs.


15 | P a g e

Cyberinfrastructure elements most in need of governance include the data network (including prioritization of investment), the HPC environment (including selection of computational architectures and institutional arrangements for funding and managing them), policies for balancing the massive parallelism needed by some researchers and the throughput needed by the entire community of users, and data stewardship (including curation). Any resource that must be shared among many users must also have some form of user governance to balance competing priorities and reconcile conflicting interests. Good Practice 5: Conduct Cyberinfrastructure Impact Analysis. Cyberinfrastructure impact analysis is to research expenditure what environmental impact analysis is to large development projects: an effort to understand how any proposed investment will affect its context.xv As explained earlier in this paper, the global research and education cyberinfrastructure is complexly layered, with heavy interconnection of computing, data storage, and data communications. For example, a PI’s decision to build a computing facility within an individual department or center may force later investments by the campus in upgraded building networks or even the campus backbone. Cyberinfrastructure impact analysis can be incorporated into many common decision workflows, including creation of new programs, construction of new facilities, establishment of new centers, or acceptance of major grants. The earlier this takes place, the better the opportunities for the institution to control costs. For example, by routinely conducting cyberinfrastructure impact analysis on grant proposals, a campus may be able to produce lower impact ways of accomplishing a researcher’s goal. But at a minimum, this form of analysis allows greater predictability and financial preparation.

What Specific Actions and Investments are Timely Now?

Meeting the needs of researchers and the expectations of funding agencies will require near-term attention to six high-priority enhancements of campus

Six High Priority Investments in Cyberinfrastructure

1. Preparing for “federated identity management” and other enabling technologies for virtual organizations.

2. Maintaining state-of-the-art communication networks.

3. Providing institutional stewardship of research data.

4. Consolidating computing resources while maintaining diverse architectures.

5. Expanding cyberinfrastructure support to all disciplines.

6. Exploring cloud computing to manage long-term financial commitments.


16 | P a g e

cyberinfrastructure: Enhancement 1: Federated Identity Management and Other Enablers of Virtual Organizations. Our researchers, and our students, interact with people and resources all over the world, and they are thus often required to identify themselves through some form of login. Federated identity management is technology for passing credentials from one organization to another, using a trust federation to which the organizations subscribe. To a student or faculty member, federated identity management means that they can log in to remote resources using their home institution credentials. All CIC institutions belong to the trust federation known as InCommon, and all have implemented the technology that enables us to identify our students and faculty to remote resources. We have active projects underway to make our own systems more accessible to researchers and students coming in from other institutions, and we should take additional steps to make our resources highly permeable to people outside the institutions with whom we wish to collaborate. Enhancement 2: State-of-the-Art Communication Networks. Bandwidth is not our only concern; network technology continues to evolve, and although we must be planning years ahead for gradual replacement of old technologies with new technologies, we must also be prepared to respond instantly to the first demands for these technologies. There is still a very long tail in the distribution of need for high-end network resources; who is in the long tail at any point in time is highly unpredictable, so campus networks must be comfortable managing exceptions to whatever their current standards may be. Although we may one day consider outsourcing our data networks to commercial providers, our near-term sourcing strategy for data networking is entwined with regional and national networks dedicated to research and education, to which we must be active contributors. Enhancement 3: Institutional Stewardship of Research Data. Data stewardship means storing and maintaining data for as long as it may be needed and controlling access to it with appropriate policy. We are all accustomed to doing this task for business data, but not for research data. The emerging need for stewardship of research data involves not only storage that is professionally managed, but also curation (things like metadata, collection management, and finding aids). Funding agencies are beginning to require researchers to present plans for long-term data availability. It is expected that data management plans will be required for all proposals NSF-wide before the end of 2010.xvi Leaving this unfamiliar task to individual researchers guarantees that the solutions will be heterogeneous, high-risk, and hard to sustain over time.


17 | P a g e

Enhancement 4: Consolidating Computing Resources. Against a goal of producing as much science as possible for every dollar of investment, shared clusters are far more beneficial than individually operated clusters. As explained above, funding models play a critical role in how broadly cluster resources are shared, and Provosts can exert considerable influence, especially through how faculty startup packages are structured. The common practice of including funds for computing resources in startup packages can have long-term impacts, positive or negative, on the campus as a whole. We recommend looking closely at strategies that guarantee the faculty member access to resources needed for success but also contribute to a sustainable campus resource. For example, instead of committing funds for an independently maintained compute cluster, a startup package could be structured as a commitment of a certain number of cores on a shared system, similar to what Purdue has done to aggregate many researchers’ resources into a shared cluster. It is common on research university campuses for individual researchers to maintain their own computing resources in their own space. This is extremely wasteful of scientific capacity and extremely expensive for the campus. Avoiding needless operating expense means providing consolidated data center space; increasing the yield from all resources is possible through consolidating PI-specific clusters into shared clusters. Absent some compelling scientific rationale for physically distributed computing, all campuses have a compelling financial interest in physical consolidation. Enhancement 5: Expansion of Cyberinfrastructure Support to All Disciplines. We began this primer with two exemplars of 21st Century cyberinfrastructure-enabled work: one an artist, the other a humanist. We chose these examples from many other stellar examples on the CIC campuses to make a point: There are no longer any noncomputational disciplines—just disciplines that have not yet solved the resourcing problems created by scholars who are attempting things that were not possible previously. The launch event for Dream, for example, required network capabilities unprecedented to that point on Illinois’ campus: a network connection good enough to allow physically separated musicians to respond to one another improvisationally in real time. Other disciplines in the arts, the humanities, and the professions are demanding new forms of instrumentation, new computational resources, new access to datasets and collections, and new visualization tools. Investment in cyberinfrastructure is now necessary for all forms of scholarly work. Enhancement 6: Exploration of “Cloud Computing” to Manage Long-Term Cost Commitments. As noted earlier, sharing of cyberinfrastructure resources can occur at the above-campus level, and today’s “cloud computing” resources may fill this role. Cloud computing is computing that happens “in the Internet cloud,” an expression that is generally applied to on-demand resources or services accessed via the Internet.


18 | P a g e

Typically the user of cloud services gets storage or computing time, but has no knowledge of the total size, configuration, or location of the computing facility being used. One of the earliest providers of HPC cloud services has been Amazon, with its Elastic Compute Cloud service. Specialty providers of HPC cloud services (such as R Systems, an NCSA spinoff located in Illinois’ Research Park) cater to even very high end users with services that can be bought “by the drink.” One opportunity that we believe is worth evaluating is the use of cloud services to manage the long-term financial impact of faculty startup packages. At present, it is common to structure startup packages to include funds for purchase of computer equipment, which generally leads to additional unbudgeted costs (for space and for ongoing operation). Including equivalent dollar value to spend on commercial providers of HPC not only provides a faster startup but also allows continuous access over a period of years to state-of-the-art resources instead of rapidly aging resources. This strategy works equally well with an internal cloud or a cloud shared within the CIC, by the way. Cloud HPC services may offer good alternatives to satisfying HPC needs, but attention will have to be paid to a number of contract issues that should not be left to one-off negotiation by individual faculty. These contract issues include assurances by the service provider regarding ownership and security of the data or services; the full cost of the service (considering not just the compute cycles, which are very inexpensive, but also collateral charges such as transport for data, which can be very expensive); and geographical location of the actual service being used (e.g., use of cloud services located outside the U.S. may be a problem for ITAR-subject research data or algorithms). Although due diligence is required for commercial cloud services, we find the general concept attractive enough to be worth an investment in well-conditioned contracts.

How Can CIC be Leveraged for Better Value from Cyberinfrastructure Investment?

Every research institution needs its own cyberinfrastructure strategy to promote its researchers’ ability to contribute at the leading edges of the disciplines. We conclude this paper by pointing out opportunities to leverage the strength of CIC to increase the value of investments we make on each campus. Given that federal funding agencies count on every institution to make significant campus-level investments

CIC Collaborative Opportunities in Cyberinfrastructure Requiring Provost Action

1. Build interest among other Provosts in federated identity management.

2. Review recruiting and retention practices that affect campus cyberinfrastructure. 3. Continue supporting competition as a consortium. 4. Influence federal spending rules.


19 | P a g e

over the shared national cyberinfrastructure, the CIOs believe that we should be looking for institutional and interinstitutional strategies that make the best possible use of every dollar spent. We already regularly benefit from the following practices within the CIO group: Rapid spread of innovation within the CIC. Strategies that succeed at one CIC

school tend to spread rapidly throughout the consortium, not only through the CIOs, but also through the main Technology Collaboration workgroups that comprise a dense social network among our campuses. The shared cluster model is one example; based on Purdue’s success, some other members have already adopted this innovation and others are preparing to do so.

Shared investment in shared infrastructure. CIC already has a well-established practice of shared investment in cyberinfrastructure. We jointly own and operate the OmniPoP, a connector to Internet2 and other R&E networks. An important current initiative is an effort to implement shared distributed storage infrastructure.

Shared contracts and shared contract language. Especially as our campuses

explore cloud services, we may benefit from joint negotiation of both purchase price and contract terms. All higher education institutions share a core of concerns around privacy of personal data, around ownership of intellectual property, and around federal compliance. Our interests are better served by addressing the commercial sector as a unified market segment rather than by attempting to negotiate terms one institution and one vendor at a time.

Influence as a consortium. The CIC CIOs have on occasion been successful in

exerting influence within organizations dedicated to cyberinfrastructure (such as the Internet2 organization). Our most noteworthy success has been in raising the visibility of InCommon and federated identity management.

Coordinated support for research and education. Leveraging previous

collaborative effort around networking, the CIC CIOs were able to rapidly coordinate the videoconferencing technologies needed to run the Virtual School for Computational Science and Engineering.

Several of the recommendations made earlier in this paper would benefit from the direct support of the Provosts. Build interest among other Provosts in federated identity management.

Anecdotally, we know that some IT organizations feel that their campuses are not yet demanding federated identity management. The real benefit from this new technology is in none of us having to manage credentials in ad hoc ways. The CIC Provosts, all of whom enjoy the benefit of being able to log in to a growing list of resources using home campus credentials, can exert positive peer pressure on others through organizations like the AAU and the APLU. The CIOs are very


20 | P a g e

committed to this enabler of interinstitutional collaboration and would happily develop some talking points.

Review recruiting and retention practices that affect campus

cyberinfrastructure. We have argued that including funds for computing clusters in startup packages has negative unintended consequences for each campus, but we are aware that these practices persist because academic leadership believes that they are necessary for competitiveness in recruitment. A deep and candid discussion among an influential group of Provosts might open opportunity for rethinking this wasteful practice. The Provosts could also consider joint statements of support for change in federal agency funding practices, another point of advocacy on which the CIOs could provide content.

Continue supporting competition as a consortium. One recent and dramatic

success has been winning the petascale computing award for the Great Lakes Consortium (CIC plus an expanded set of collaborators). The proposal leading to this award required commitment from all institutions, not from the CIOs but from the academic leadership. We suggest that a natural extension of this would be to pool advanced technical support resources through some form of virtual help desk for researchers, taking advantage of the varied strengths of the individual campuses. A practical step toward this would be a CIC-wide summit meeting, sponsored by the Provosts, to consider whether and how we could pool advanced research technology expertise, engaging the research faculty to identify needs and resources and engaging campus CIOs to plan the enabling structures for sharing.

Influence federal spending rules. Although NSF has great interest in how to

coordinate campus investments with agency investments, certain funding rules and directorate-specific practices work against coordination at the campus level—creating perverse incentives that encourage wasteful use of resources. Campus-specific rules for charging overhead, and for distributing recovered overhead, may create additional perverse incentives at the lab level. Problems in our current model are broadly acknowledged, but so far, no one has been ambitious enough to try to solve this problem on behalf of the whole community. The CIOs recommend developing a CIC posture on the coordination required between institutional and national investment and using this posture to exert influence at the federal level.

The long-term trust relationships among our campuses and well-worn paths to collaboration around technology benefit our researchers directly. We urge the Provosts to consider how we can leverage this strength to amplify our separate and joint influence on the shape, and the overall quality, of national and global cyberinfrastructure.


21 | P a g e

Notes i Donna J. Cox is the Michael Aiken Chair and Director of the eDream Institute at University of Illinois at Urbana-Champaign. To read more about her work at NCSA, see http://avl.ncsa.illinois.edu/. To read more about eDream (Emerging Digital Research and Education in Arts Media), see http://eDream.illinois.edu/. ii Ruth Stone is the Laura Boulton Professor of Ethnomusicology at Indiana University. To read more about her work in folklore and ethnomusicology, see http://www.indiana.edu/~alldrp/members/stone.html. iii See also Klara Jelinkova et al., “Creating a Five-Minute Conversation about Cyberinfrastructure,” Educause Quarterly, Vol. 31, no. 2, 2008. Available online at http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/CreatingaFiveMinuteConversatio/162881. iv Paul Gandel et al., “More People, Not Just More Stuff: Developing a New Vision for Research Cyberinfrastructure,” ECAR Bulletin, issue 2, 2009. Available online at http://www.educause.edu/ECAR/MorePeopleNotJustMoreStuffDeve/163667. v “Building Cyberinfrastructure into a Campus Culture,” (March 30, 2010). See http://net.educause.edu/LIVE108. vi HathiTrust is a consortium of universities, including all of the CIC, formed to digitize library holdings for accessibility and preservation. See http://www.hathitrust.org. vii The Large Synoptic Survey Telescope is a multi-institutional, multi-national resource that aims to gather unprecedented amounts of astronomical data that will enable new science (such as 3D modeling of the universe). The data will be housed at NSCA, in the National Petascale Computing Facility. See http://lsst.org/. viii The Virtual School offers several courses each summer with multiple sites participating by high definition videoconferencing. In 2009, 110 students from 12 CIC institutions attended one or more courses. See http://www.vscse.org/. ix The same point is made by Gandel et al. (note 4). x From Thomas J. Hacker and Bradley Wheeler, “Making Cyberinfrastructure a Strategic Choice,” Educause Quarterly, Vol. 30, no. 1, 2007. Available online at http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/MakingResearchCyberinfrastruct/157439. xi The NSF-wide Advisory Committee on Cyberinfrastructure is currently sponsoring a Campus Bridging Task Force (https://nsf.sharepointspace.com/acci_public/) aimed at understanding how campus cyberinfrastructure can seamlessly connect to national resources. Educause has a working group on campus cyberinfrastructure (http://www.educause.edu/CCI) that has coherence among layers as a standing concern. Internet2 has a history of interest in this area expressed through the work of a Campus Expectations Task Force (http://www.internet2.edu/cetf/). xii “Developing a Coherent Cyberinfrastructure from Local Campus to National Facilities: Challenges and Strategies”, report from a joint workshop of the Educause Campus Cyberinfrastructure Working Group and the Coalition for Academic Scientific Computing hosted by Indiana University in Summer 2008. Report available online at http://net.educause.edu/ir/library/pdf/EPO0906.pdf. xiii For a description of Purdue’s model and a business perspective on its advantages, see “Condo Computing” at


22 | P a g e

http://www.nacubo.org/Business_Officer_Magazine/Magazine_Archives/June-July_2009/Condo_Computing.html. xiv This well-known definition comes from Peter Weill and Jeanne Ross, “IT Governance on One Page,” MIT Sloan Working Paper No. 4517-04 (November 2004). Available online at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=664612. xv A tool for conducting cyberinfrastructure impact analysis on grant proposals has been developed and tested at Illinois, but not yet implemented at any major research university. Testing the tool on several hundred proposals showed significant opportunities for improving the project or reducing the collateral costs to the institution, or both. xvi From remarks at the Spring 2010 Internet2 Member Meeting: Jennifer Schopf, “Update on NSF Programmatic Network Funding,” 26 April 2010; see also http://news.sciencemag.org/scienceinsider/2010/05/nsf-to-ask-every-grant-applicant.html.

a research cyberinfrastructure strategy

Documents