i-cluster acm-sigops jte cluster computing bruno richard research program manager hp labs grenoble
TRANSCRIPT
I-Cluster
I-Cluster
ACM-SIGOPSJTE Cluster
Computing
Bruno RichardResearch Program Manager
HP Labs Grenoble
Page 22October 2001
HP Labs
•Focus on infrastructure, information appliances and e-services
• World’s second largest computer research lab
• 750 researchers in 6 labs globally
• Leading-edge collaborations
• Helps create HP’s IP portfolio
Page 32October 2001
HP Labs Grenoble
•Role
•Deliver technology related to emerging Internet access devices
•Two main research areas:
•PPP
–personalization, profiling, privacy–User-adapted computing
environment•LEA
–Local Environment of Access Devices
–Wireless communities
•I-Cluster
•Compute-intensive services
•Partnership with ID-IMAG, INRIA
Page 42October 2001
Project overview
What is I-Cluster?
A distributed framework of tools that transparently takes advantage of unused network resources and transforms them into compute-intensive services
Project rationale
Support distributed virtual functions utilising unmodified, standard hardware
Learn how cluster devices interact with each other -- potential and limitations
Create environment for development and execution of applications that will use enterprise infrastructure or internet rather than dedicated cluster
Apply knowledge gained from I-Cluster to future products
Page 52October 2001
I-Cluster Value
•Supercomputing-enabled
•Not limited to metacomputing– Cluster computing environment
•Fine-grained problems– Network communications can be stressed
•User compliance
•Self-organized system– Self sustaining
•From federative model to community model– Server-less– Service oriented– No administration– Peer-to-peer system
•Real-world conditions– Massively scales (10000 devices)– No specific hardware required– Heterogeneous environments– Roaming, disconnections
•Applications do not need to be rewritten– But no shared memory
Page 62October 2001
I-Cluster CloudI-Cluster Cloud
Device Device
Device
DeviceDevice
DeviceDevice
I-Cluster CloudI-Cluster Cloud
Device
Device
DeviceDevice
Step 4My service is executed on the cluster within seconds
Device
DeviceDevice
Step 1Request a computing service from the cloud.
“Render Star Wars movie for my PDA”
I-Cluster CloudI-Cluster Cloud
Device
Device
DeviceDevice
Step 2Identify cluster aggregate that fits the required service
Device
DeviceDevice
I-Cluster CloudI-Cluster Cloud
Device
Device
DeviceDevice
Step 3Efficiently distribute the job on identified devices
Device
DeviceDevice
Starwars.avi
Usage exampleUsage example
Page 72October 2001
Research areas
I-Cluster cloud
P2P community
Gathers resources
Discovers network topology
Mode switch
Any PC becomes a cluster machine
Use idle periods
No (lowest) user impact
Match finder
Instantiates a cluster on the cloud
Allocates devices to given jobs
Tetris
Inter-intra task scheduling
Page 82October 2001
I-Cluster cloud
•Devices operate in peer to peer:
•A server answers other’s requests
•A client actively polls others
•Local database: a view of the cloud
•Incomplete
–Some elements are unknown–Elements are forgotten
•Lazy consistency
–Best effort consistency
•Network analysis
•Topology
•Bandwidth, latency
•Congestion analysis
•Very fast convergence
Database
Emitter
Receiver Database
Emitter
Receiver
Database
Emitter
ReceiverDatabase
Emitter
Receiver
Page 92October 2001
Cloud operation: An exampleFull network view
Blue: I-Cluster devicesYellow:
Routers+other
A
B
Page 10
2October 2001
Cloud operation (cont’d)
View from device AA
B
Page 11
2October 2001
Cloud operation (cont’d)
View from device BA
B
Page 12
2October 2001
Cloud operation (cont’d)
Combined viewAfter synchronization
Each device will forget some items
Based on relevance of peers
A
B
Page 13
2October 2001
I-Cluster mode switchEach I-Cluster PC has 2 modes
-User mode- Standard Windows operation
-Cluster mode- Cluster Linux distribution- Sandbox for jobs execution- Secure mode (no user data access)
Idle periods used for cluster computing
-Idleness detection
-Automatic switch between modes
Lowest user impact
-Easy installation
-No change in user partitions
-Low psychological impact
-Automatic transitions
Ease of development
-Easier than other sandboxing technologies
User mode
I-Cluster mode
Off
Switch offSwitch on
AvailableInactivity prediction
ReservedAllocated
RWU
Tentative
Allocation
Job end
User request
Page 14
2October 2001
Hidden cluster distribution
•User hard disk partitions are kept without modification
Page 15
2October 2001
Hidden cluster distribution
•Then anchors are added
•Master Boot Record is changed to a new code
•A hidden zone is used for storing our mode control tools
Page 16
2October 2001
Hidden cluster distribution
•Then a fake partition is created
•Big file in user’s file system
•Contiguous, unfragmented
•System-protected, unwriteable
•This partition will be usable as a boot option by the switcher
Page 17
2October 2001
Match finder
•Called upon user request
•A service is invoked
•User data attached to invocation
•Service requirements available
•Number of processing nodes
•Minimum RAM
•Maximum Network latency
•…
•Allocates a cluster within the cloud
Service request:- 4 nodes required- At least 256 MB
4 machines allocatedJob started there
Page 18
2October 2001
Tetris
•Optimal use of computing resources
•Inter/Intra job scheduling
•Use of job knowledge for intelligent task/resource assignment
•Use of past experiences to improve future scheduling
Agreements between HP-Labs and INRIA
Page 19
2
Tetris
duration
number ofprocessors
Page 20
2
Tetris
duration
number ofprocessors
Page 21
2October 2001
Backup slides
“It is all about power, space and time”Chessmaster Savielly Grigorievitch Tartakower
Page 22
2October 2001
The experimental platform
Page 23
2
TOP500 resultsI-Cluster
Supercomputing with a mainstream
cluster
A TOP500 cluster based on 225 standard hp e-Vectra machines
Partnership with IMAG-ID, INRIA, Intel225 HP e-Vectra:
- Pentium® III- 733 MHz- 256 MB- Standard Ethernet (100 MBps)
76 Gflop/s * as of April 15th 2001(*) Standard LINPACK benchmarkhttp://www.top500.org/