10/21/2009 1

Protein Explorer: A Petaflops Special-Purpose Computer

System for Molecular Dynamics Simulations

Makoto Taiji, Tetsu Narumi, Yousuke Ohno, Noriyuki Futatsugi, Atsushi Suenaga, Naoki Takada and Akihiko Konagya

High-Performance Biocomputing Research Team, Bioinformatics Group, Genomic Sciences Center, Institute of Physical and

Chemical Research (RIKEN) ACM/IEEE SC2003 Conference

Presented by Jing XueFor Computer Architecture Reading Group

10/21/2009 2

Protein Explorer (PE) System• What is PE system ?

– special-purpose computer system for molecular dynamics simulations particularly protein simulations– PC cluster equipped with special-purpose engines that calculate non-bonded interactions between atoms

• What is MDGRAPE-3 Chip?– A dedicated LSI performs these force calculations

• Why is specialized?– Most time is spent on the calculation of long-range forces

• Gravitational, Coulomb, and Van der Waals forces– Communication-to-computation ratio is 0.25 Gbytes/sec*Tflops

• What is GRAPE?– GRAPE (GRAvity PipE) is a project to develop high-performance competitive special purpose systems– Classical particles (e.g. gravitational N-body problems, Molecular dynamics (MD) simulations)

• What is pioneered?– Delft Molecular Dynamics Processor (DMDP) and FASTRUN processor

• What is the targets of PE system?– High-precision screening for drug design and large scale simulations– Funded by Protein 3000 project started in 2002

10/21/2009 3

MDGRAPE-3 System

10/21/2009 4

What Protein Explorer Calculates

• Two-body forces on i-th particle Fi

Where , .

The vectors ri, rj are the position vectors of the i,j-th particles and is an arbitrary smooth function.

Coulomb forces:

Lennard-Jones potential:

10/21/2009 5

MDGRAPE-3 chip

10/21/2009 6

Software Cost and Performance Estimation

1. Call pe_start_calc2. Perform other force calculations3. Call pe_wait_calc (force)4. Add other forces such as bonding force to ‘force’, then calculates orbits of

particles, and increments a time

1. Get command sequence from host memory2. Write positions, charges, and species of j-

particles to the chip3. Broadcast positions of 40 i-particles to all the

chip4. Issue the calculation command5. Do calculation6. Read results out from the chip7. Transfer result to host memory8. Repeat from 3-7 for all i-particles9. Tell the end of the calculation to the host

10/21/2009 7

Anton, a Special-Purpose Machine for Molecular Dynamics Simulation

David E. Shaw et. al.D. E. Shaw Research

ISCA 2007

Presented by Alok Garg

10/21/2009 8

Force Calculation

• Mechanics force fields on 200,000 molecules/time step

• Total energy– Bonded (linear time complexity)– Non-bonded - O(n2)

• Non-bonded interactions– Range limited interactions– Long-range interactions

• Expresses as a convolution• Computed by: FFT, multiplication, and

inverse FFT

• 73% computation is range limited interactions

10/21/2009 9

Range-Limited Pair-wise Interactions

10/21/2009 10

Parallelization Algorithm

10/21/2009 11

Computation

e.g.A cubical system of 25,000 atomsAtoms in tower: 220Atoms in plate: 430

Total pairs: 94,60023,000 satisfy the selection rule

Each node must import 550 particles

10/21/2009 12

Specialized Hardware

10/21/2009 13

Rest of the Machine

10/21/2009 14

Major Computational Tasks

10/21/2009 15

System Architecture

• Features– Multicast support– Compressed transfer of

sparse data structures

• Push based communication

• Memory accumulate forces for each particle

10/21/2009 16

Performance Results