A Tentative Proposal for ISTORE-2

Winfried W. Wilckewilcke@almaden.ibm.com(408) 927-2139Almaden Research Center

July 18, 2000

Richard C. Boothrcbooth@us.ibm.com(408) 927-1879Almaden Research Center

David A. Pattersonpattrsn@cs.berkeley.edu(510) 642-6587University of California, Berkeley

Underlying Beliefs...• Commodity components are quickly winning the server

wars– Gigabit Ethernet will win everything– x86 Processors– Linux OS will prosper

• Large servers (100-10k nodes) will be quite common - and most are storage centric

• What matters most:– Ease of management, density of nodes and seamless

geographical interconnect

Generations of IStore• IStore = IStore-1: Present UCB Project• IStore-2: Joint Research Prototype

– ~2000 nodes– Split between UCB, IBM and others– Hardware similar to IStore-1– Focus on real applications and management

software– Operational YE 2001

• Follow-on Work

Talk Outline

• Project Goals• Applications• Research Topics• Hardware Architecture• Development Schedule• Working Relationships• Next Steps

Applications&

Research Topics

Candidate Applications• Research Focus

– NOAA Severe Weather Warning (R. Arps, ARC)– Fast Image Recognition (J. Malik, UCB)

• Commercial Focus– Scalable E-business server (IGS) - a must !– Deep Searching of Entire Web; Webfountain (N. Pass)– (tbd) Large Scale Network Attached Server (J. Palmer)– (tbd) Speech Recognition Farms for Phone-based Special Web-

services

NOAA Severe Weather.... Ron Arps

• Doppler Radar enables detection of violent tornadoes and plane crashes due to windshear

• Doubled warning time for residents in Oklahoma during '99 class 5 outbreaks– Goal: 15 minutes avg. warning time in 2004

• Eventually 120 radar sites will be established• Matches well with I-Store characteristics

– Needs scalable local storage/processing plus seamless transfer of data on geographical scale, manageable from one site

WebfountainNorm Pass

• Index entire Web every few weeks– Google, Northernlight index 25%

• 4 TB index => 200 TB in two years• 'Miner' technology demonstrated

– Resumes, Prices, Geospatial,...– Prototype running on a 30 node Linux farm

Software Model

• Users will see a standard Linux farm (shared nothing) programming model– No porting effort for existing Linux farm

applications (except dealing with different versions of Linux, of course)

• The system management functions are only visible to system administrators– Exception are performance monitoring functions

useful for tuning apps

Differences to a Linux Farm• Much higher spatial density of Nodes or ‘Bricks’• Single network protocol (Ethernet) for ALL off-node

communications• Design with geographical distribution in mind • Diagnostic Processors• Lego-like, standardized building blocks

– Regular and relaxed homogeneous• Monitoring Hardware• Measuring of relevant environmental parameters• (New) System Management Language

• AME, SON and RAIN objectives

AME, RAIN and SON

• Three areas of system research to be explored with I-Store

• These three areas are largely independent of each other

AME• Availability

– No single points of failure– Introspection, failover and fast failure– Fast repair by swapping identical blocks

• Maintainability– Homogenous structure– System management language

• Extensibility/Scalability– Shared nothing architecture

RAIN• Redundant Array of Inexpensive Network

(Switches)• Issues to be explored

– Optimal topology – Density/cost of ports, optics vs. copper– Routing algorithms within a machine– Need for TCP hardware acceleration– Performance of Ethernet protocol– Frame sizes– Simplified switches

SON

• Storage Oriented Nodes• Basic Premise of one node=one disk=one

processor– It works in farms, but is it a good general choice?– Is the loss of flexibility (in the ratio of disks per

processor) a good tradeoff for easier management?

Additional Software Research Topics...

• Define AME, RAIN, SON benchmarks• Server Management Language• Parallel Searching of geographically

distributed database• Dynamic Resource Allocation (i.e. Firewalls)• SCSI over TCP/IP (SAN within I-Store)• Storage for mobile users (a’la Ocean Store)

System Management Language

• Define a high-level, interpretive(?) system management language– May use facilities of system OS

• Highly regular I-Store is the first target• Sample Verbs

– allocate, protect, share, map, backup, restore, copy, correlate, display, discover, ping, initialize, report, arm, define(node)....

System Management Language• Should easily describe tasks such as:

– Backup all data located in the Philippines to Colorado (a volcano is about to blow)

– Set alarm if any disk is more than 80% full– Define protected subregions in the system– Display CPU utilization by time and state– Discover present routing topology– Show 3D correlation plot of disk vibration vs brick

temperature vs. actual failure events– .....

Hardware ArchitectureDevelopment Schedule

&Working Relationships

IStore HardwareArchitecture Goals

• Seamless Scalability– O(10,000) AME Storage Nodes– Optimized Storage Brick for Packaging Density

• Geographically Disperse Nodes– Gb Ethernet Connections to WAN Routers

• Storage Brick – Full PME Brick: Processor, Memory, Cache– Gb Ethernet as the Sole Interconnection Fabric– Imbedded Disk with 10s GBytes

IStore HardwareArchitecture Goals (cont.)

• State-of-the-art Intel Processor Memory Element (PME) – 650 MHz Pentium III with 100 MHz System Bus– 256 KB L2 cache– O(512MB) main memory

• State-of-the-art Interconnect Fabric– 1 Gb Ethernet Runtime Network– 10/100 Mb Ethernet Diagnostic Network

• State-of-the-art Disks– 2.5" ~32 GB drive

IStore HardwareArchitecture Goals (cont.)

• Berkeley AME Hardware Management Support– Diagnostic processor– Environmental sensors

• TCP/IP Hardware Accelerator– Class 4: Hardware State Machine

• SCSI over TCP ("iSCSI") Support• Compatible with Standard Ethernet

Switches/Routers

IStore-1Current Berkeley Design

• 80 nodes• AME• 266 MHz Pentium II• Four 100 MB Ethernet Ports/brick• Integrated UPS

IStore-2Deltas from IStore-1

• Geographically Disperse Nodes– O(1000) nodes at Almaden– O(1000) nodes at Berkeley

• Upgraded Storage Brick– Pentium III 650 MHz Processor– Two Gb Ethernet Copper Ports/brick– One 2.5" ATA disk

• User Supplied UPS Support• Standard Ethernet Switches

Follow on Work

• Ethernet Sourced in Memory Controller (North Bridge)

• TCP/IP Hardware Accelerator– Class 4: Hardware State Machine

• SCSI over TCP Support• Integrated UPS

Why an IStore-2 PrototypeIs Interesting

• Storage Bricks– New ratios for MIPS/bandwidth/storage– New level of density

• AME Hardware Support– Seamless scaling– Self maintaining nodes

• It Exists

IStore-2Core Design Team

• IBM (full time)– System Architect: Winfried Wilcke– Lead Designer: Richard Booth– 1 Experienced Hardware Designer: tbd– 3 Designers: tbd

• Berkeley– 6 Graduate Students

IStore-2Development Schedule

• Working Model– 7/00: Agreement in Principle– 8/00: Working Team Membership

• Design– 9/00: Architecture Specification version 1.0– 11/00: Design Workbook version 1.0

• Implementation– 2Q/01: First 3 Nodes Power-up– 3Q/01: O(64) nodes available to users– 4Q/01: O(2000) nodes available to users

IStore-2 Footprint(per 1000 nodes)

• 16 Storage (19") Racks – 64 Storage bricks/rack

• 8 type 1 storage bricks/drawer• 8 storage drawers/rack

– Ethernet switches in rack• 8 Global Ethernet Switch (19") Racks• Requires 600 sq.. ft lab

IStore-2 PlatformRequired Resources

• Staffing– 6 ARC/SSD IBMers– 6 UCB Graduate Students

• Lab Space– 600 sq. ft. lab at Almaden– 600 sq. ft. lab at Berkeley

• Hardware Costs– $3M (mostly 2001 dollars)

IStore-2Working Model

• Jointly Authored Architecture Specification– 1 or 2 Almaden authors– 1 or 2 Berkeley authors

• Design Workbook– Each Core Team Member owns a section

• Weekly Half Day Working Face-to-face Meetings– Alternate between Almaden and Berkeley

• Shared Electronic Documentation• Machine Available -for free- to Users From Either Institution• IP is Handled Like Previous IBM/UCB Projects ??• Fabrication (some design ?) Vendored Out

Next Steps• Continue to Seek Feedback on Proposal• Funding Discussion

– IBM– Berkeley

• Form IBM Team• Begin Regular Working Meetings• Begin Architectural Design