GRID COMPUTING :A PRACTICAL GUIDETO TECHNOLOGYAND APPLICATIONS

AHMAR ABBAS

CHARLESR I V E RM E D I A

a-."

CHARLESRIVERMEDIA, INC.Hingham, Massachusetts

Contents

Preface

xixAcknowledgments

xxiii

1

IT Infrastructure Evolution

11.1 Introduction

21.2 Microprocessor Technology

21.3 Optical Networking Technology

31.4 Storage Technology

61.5 Wireless Technology

71.6 Sensor Technology

91.6.1 Fiber Optic Sensors

91 .6.2 Wireless Sensors

101.7 Global Internet Infrastructure

111.8 World Wide Web and Web Services

141.9 Open-Source Movement

161.10 Conclusion

16

2

Productivity Paradox and Information Technology

192.1 Introduction

202.2 Productivity Paradox

202.3 Return on Technology Investment

202.4 Multi-Story Bureaucracy

212.5 Information Technology Straightjacket

222.6 Consolidation

252.6.1 Server Consolidation

252.6.2 Consolidating Applications

262.6.3 Consolidating Storage

27rrvu

Viii contents

2.7 Outsourcing2.8 Toward aReal-Time Enterprise-Operational Excellence2.9 Conclusion

272829

3

Business Value of Grid Computing

31

3.1 Introduction

323.2 Grid Computing Business ValueAnalysis

333.2.1 Grid Computing Value Element #1 : Leveraging Existing

Hardware Investments andResources

333.2.2 Grid ComputingValue Element#2 : Reducing Operational

Expenses

343.2.3 Grid ComputingValueElement#3 : Creating a Scalable

and Flexible Enterprise IT Infrastructure

353.2.4 Grid Computing Value Element #4 : Accelerating Product

Development, Improving Time to Market, andRaisingCustomer Satisfaction

353.2.5 Grid ComputingValue Element#5 : Increasing Productivity

363.3 Risk Analysis

373.3.1 Lock-in

383.3.2 Switching Costs

383.3.3 Project Implementation Failure

383.4 Grid Marketplace

383.4.1 Grid Taxonomy

393.4.2 Fabric

393.4.3 Middleware

403.4.4 Serviceware

403.4.5 Applications

403.4.6 Grid Service Providers

403.4.7 Grid Applications Service Providers

413.4.8 Grid Consultants

413.5 Conclusion

41

4

Grid Computing Technology-An overview

434.1 Introduction

444.2 History

44

Contents ix

4.3 High-Performance Computing

464.4 Cluster Computing

474.5 Peer-to-Peer Computing

484.6 Internet Computing

494.7 Grid Computing

514.7.1 Peer-to-Peer Networks and Grid Computing

524.7.2 Cluster Computingand Grid Computing

534.7.3 Internet Computing and Grid Computing

534.8 Grid Computing Model

534.9 Grid Protocols

574.9.1 Security: Grid Security Infrastructure

574.9.2 Resource Management: Grid Resource Allocation

Management Protocol

594.9.3 Data Transfer: Grid File Transfer Protocol

604.9.4 Information Services : Grid Information Services

604.10 Globus Toolkit

604.11 Open Grid Services Architecture

614.12 Global Grid Forum

624.13 Types of Grids

644.13.1 Departmental Grids

644.13.2 Enterprise Grids

654.13.3 Extraprise Grids

654.13.4 Global Grids

664.13.5 Compute Grids

664.13.6 Data Grids

674.13.7 Utility Grids

674.14 Grid Networks-WillThere Be Such a Thing as "The Gridnet"?

674.14.1 Grid Network Peering Points

684.15 Grid Applications Characteristics

694.16 Application Integration

704.17 Grid Computing and Public Policy

724.17.1 Sleeper Programs

724.17.2 National Security

72

X Contents

4.17.3 Philanthropic Computing

72

4.18 Conclusion

73

5

Desktop Grids

755.1 Introduction

76

5.2 Background

765.2.1 Cause Computingandthe Internet

765.2.2 Distributed Computing in the Enterprise

77

5.3 Desktop Grids Defined

795.4 The Desktop Grid Value Proposition

805.5 Desktop Grid Challenges

815.6 Desktop Grid Technology-Key Elements to Evaluate

82

5.6.1 Security

825.6.2 Unobtrusiveness

825.6.3 Openness/Ease ofApplication Integration

835.6.4 Robustness

835.6.5 Scalability

835.6.6 Central Manageability

835.6.7 Key Technology Elements-Checklists

845.6.8 KeyTechnology Elements-Summary

855.7 Desktop Grid Suitability-Key Areas for Exploration

865.7.1 Applications

865.7.2 Computing Environment

895.7.3 Culture

905.8 The Grid Server-Additional Functionality to Consider

915.9 Role ofDesktop Grids in an Enterprise Computing Infrastructure

925.9.1 Departmental Grids

925.9.2 Campus Grids

925.9.3 Web Services andBeyond

935.10 Practical Uses ofDesktop Grids-Real-World Examples

945.10.1 Example: Risk Management for Financial Derivatives

955.10.2 Example: Molecular Docking for Drug Discovery

955.10.3 Example: Architectural Rendering

965.11 Conclusion

97

Contents xi

6

Cluster Grids

996.1 Introduction

1006.2 Clusters

1016.2.1 Single System Image

1026.2.2 Single System Environment

1056.3 Industry Examples

1076.3.1 Electronic Design Automation (EDA)

1086.3.2 Bioinformatics

1116.3.3 Industrial Manufacturing

1126.4 Cluster Grids

1146.5 Conclusion

116

7

HPC Grids

1197.1 Introduction

1207.2 Five Steps to Scientific Insight

1217.3 Applications andArchitectures

1227.4 HPCApplication Development Environment

1267.5 Production HPC Reinvented

1287.6 HPC Grids

1317.7 Conclusion

133Acknowledgements

133

8

Data Grids

1358 .1 Introduction

1368.2 Data Grids

1388.3 Alternatives to Data Grids

1398.3.1 Network File System (NFS)

1398.3.2 File Transfer Protocol (FTP)

1408.3.3 NFS over IPSec

1418.3.4 Secure Copy-scp/sftp

1428.3.5 De-Militarized Zone (DMZ)

1438.3.6 GridFTP

1438.3.7 Andrew File System (AFS)

144

Xli contents

8.4 AvakiData Grid

145

8.4.1 Accessing the Data Grid

148

8.4.2 Managing the Data Grid

150

8.5 Data Grid Architecture

153

8.5.1 Grid Servers

153

8.5 .2 Share Servers

154

8.5.3 Data Grid Access Servers (DGAS)

1558.5.4 ProxyServers

1568.5 .5 Failover Servers (Secondary Grid Domain Controllers)

1568.6 Conclusion

157Acknowledgements

157

9

The Open Grid Services Architecture

159

9.1 Introduction

1609.2 An Analogy for OGSA

1619.3 TheEvolution to OGSA

1639.3.1 Grid Computing

1639.3.2 Web Services

1649.3.3 Convergence

1659.4 OGSA Overview

1669.4.1 The OGSA Platform

1679.4.2 OGSI

1689.4.3 OGSA Platform Interfaces

1759.4.4 OGSA Platform Models

1789.5 Building on the OGSA Platform

1799.5.1 WS-Agreement

1809.5.2 Data Access and Integration Services (DAIS)

1819.6 Implementing OGSA-Based Grids

1839.6.1 The Globus Toolkit 3

1839.6.2 GCSF

1859.7 Conclusion

186

Contents X111

10

Creating and Managing Grid Services

18910.1 . Introduction

19010.2 . Services andthe Grid

19010.3 Converting Existing Software

19610.4 Service Discovery

19810.5 Operational Requirements

19910.6 Tools andToolkits

20110.6.1 Globus Toollcit Grid Information Service

20110.6.2 Accessing Grid Information

20310.6.3 Performance Issues with MDS

20610.6.4 Other Information Services andProviders

20610.6.5 Future

20710.7 Support in UDDI

20710.8 UDDI and OGSA

20910.9 UDDIe: UDDI Extensions and Implementation

21010.10 Uses

21810.11 Quality ofService Management

22110.12 Conclusion

222Download

223Acknowledgements

223

11

Desktop Supercomputing : Native Programming for Grids

22511 .1 Introduction

22611 .2 Historical Background-Parallel Computing

22611.2.1 MIMD Computers

22711 .3 Parallel Programming Paradigms

23011 .4 Problems of Current Parallel Programming Paradigms

23311 .5 Desktop Supercomputing : Solving the Parallel

Programming Problem

23411 .6 Desktop Supercomputing Programming Paradigm

23411 .7 Parallel Programming in CxC

23511.8 Parallelizing Existing Applications

23711 .9 Conclusion

237

XIV contents

12

Grid-Enabling Software Applications

239

12.1 Introduction

240

12.2 Grid Computing: Discontinuous Innovation or Massive Yawn?

240

12.3 The Needs of Grid Users

241

12.4 Grid Deployment Criteria

242

12.5 Methods ofGrid Deployment

244

12.6 When to Grid-Enable Software

24512.7 Requirements for Grid-Enabling Software

247

12.8 Grid Programming Tools andExpertise

24712.9 The Process of Grid-Enabling Software Applications

24912.9.1 Analysis

24912.9.2 Application Modifications

25012.10 Grid-Enabling aMainstream Software Application: An Example

25212.10.1 Video Encoding

25312.10.2 TheNeed for Speed

25312.10.3 Current Solutions

25412.10.4 Grid Deployment of Video Encoding

25512.10.5 Requirements for Broad Marketplace Adoption

25512.10.6 Overview ofMPEG 4 Encoder

25612.10.7 Overview of GridIron XLR8

25612.10.8 Distributed ComputingStrategy

25712.10.9 Implementation

25812.10.10 Results

26012.10.11 Next Steps

26212.10.12 Grid-Enabling Video Encoding Summary

26212.11 Conclusion

264

13

Application integration

26713 .1 Introduction

26813.2 Application Classification

26913 .2.1 Parallelism

26913.2.2 Communications

271

Contents XV

13.2.3 Granularity

27113.2.4 Dependency

27113.3 Grid Requirements

27113.3.1 Interfaces

27113.3.2 Job Scheduling

27213.3.3 Data Management

27313.3.4 Remote Execution Environment

27313.3.5 Security

27313.3.6 Gang Scheduling

27413.3.7 Checkpointing and Job Migration

27513.3.8 Management

27513.4 Integrating Applications with Middleware Platforms

27613.4.1 Application Preparation Example

27713.4.2 Issues in Application Integration

27813.5 Conclusion

280

14

Grid-Enabling Network Services

28114.1 Introduction

28214.2 On Demand Optical Connection Services

28314.3 Creating Grid-Enabled Network Services

28414.4 Montague River Grid

28514.5 Montague River Domain

28614.6 Sample API

28814.7 Deployment Example: End-to-End LightPath Management

29214.8 Conclusion

293

15

Managing Grid Environments

295

15.1 Introduction

29615.2 Managing Grids

29615.2.1 Trust

29715.2.2 Identity

29715.2.3 Privacy

29815.2.4 Authorization

299

xvii Contents

15.3 Management Reporting15.3.1 Users15.3.2 Resources15.3 .3 Jobs15 .3.4 Audit Support

15 .4 Monitoring15.4.1 Types of Events15.4.2 Notification Modes

15.5 Service Level Management15.6 Data Catalogs andReplica Management

15.6.1 Data Catalog15.6.2 Replication

15.7 Portals15.8 Conclusion

301301301302302303303304304305305306306307

16

Grid Computing Adoption in Research and industry

30916.1 Introduction

310

16.2 AGlobal Grid Architecture

31216.3 Core Components for Building a Grid

31216.3.1 Distributed Resource Managers

31316.3.2 Portal Software andAuthentication

31416.3.3 The Globus Toolkit 2.0

31416.4 Examples ofResearch andIndustry Grid Implementations

31516.4.1 GlobeXplorer : From Departmental to Global

Grid Computing

31616.4.2 Houston University Campus Grid for Environmental

Modeling and Seismic Imaging

31916.4.3 Ontario HPC Virtual Laboratory

32416.4.4 Canada NRC-CBRBioGrid

32716.4.5 White Rose Grid

33116.4.6 Progress: Polish Research on Grid Environment on

SunServers

33416.5 Conclusion

338Acknowledgements

339

Contents Xvii

17

Grids in Life Sciences

34117.1 Introduction

34217.2 Bioinformatics

34317.2.1 Bioinformatics and High-Performance Computing

34317.2.2 Grid Computing and Bioinformatics

344Bioinformatics-An Excellent Grid Application

34517.2.3 Example Grid Computing: Smallpox Research

34617.3 Computational Chemistry andBiochemistry

34617.4 Protein Modeling

34617.5 Ab Initio Molecular Modeling

34717.5.1 ExampleAb Initio Modeling with Parallel Computing

34817.6 Grid Computing in Life Sciences

34817.7 Artificial Intelligence and Life Sciences

34917.8 Conclusion

349

18

Grids in the Telecommunications Sector

35118.1 Introduction

35218.2 Telcos as Users

35218.2.1 CPU Intensive Application: Network Planning

and Management

35318.2.2 Data Intensive Applications : EDRAnalysis and

DataWarehouse

35618.2.3 A Corporate Grid Platform

35918.3 Telcos as Providers

36118.3.1 Grid Services : ABusiness in the Future Network

36118.4 Conclusion

362

19

Grids in Other Industries

36519.1 Introduction

36619.2 Grids in Financial Services

36619.3 Geo Sciences

36819.4 Manufacturing

36819.5 Electronic Design Automation

37019.6 Entertainment andMedia

370

xviii Contents

19.7 Chemical and Material Sciences

37119.8 Gaming

37119.9 Conclusion

372

20

Hive Computing for Transaction Processing Grids

37320.1 Introduction

37420.2 Hive Computing

37420.2.1 Assumptions

37520.2.2 Overview

37620.2.3 Capabilities

37820.2.4 Programming Model

37920.2.5 Benefits

38220.4 Conclusion

386

About the CD-ROM

389