the ibis model as a paradigm for programming distributed systems henri bal [email protected] vrije...
TRANSCRIPT
![Page 1: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/1.jpg)
The Ibis model as a paradigm for
programming distributed systems
Henri [email protected]
Vrije Universiteit Amsterdam
(from Grids and Clouds to Smartphones)
![Page 2: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/2.jpg)
Outline
● History of distributed systems
● Clusters, grids, clouds, networked world
● Programming distributed systems
● Driving applications
● The Ibis system
● Ibis on smartphones
![Page 3: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/3.jpg)
History of the Distributed World – Part
I (1980s)● Networks of Workstations (NOWs)● Collections of Workstations (COWs)● Processor pools
(Amoeba)● Condor pools● (Beowulf) clusters
![Page 4: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/4.jpg)
Amoeba processor pool (Zoo, 1994)
![Page 5: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/5.jpg)
History of the (more) Distributed World – Part
II● Metacomputing (Smarr & Catlett, CACM
1992)● Flocking Condor (Epema, FGCS 1996)● Distributed ASCI Supercomputer (1996 –
?)● Grid Blueprint (Foster & Kesselmann
1998)● Desktop grids, SETI@home (1999)
![Page 6: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/6.jpg)
Design of DAS (1996)
DAS-1 DAS-2 DAS-3 DAS-4
(slide from Andy’s ASCI’97 presentation)
![Page 7: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/7.jpg)
DAS-3
![Page 8: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/8.jpg)
A real (heterogeneous) Grid
![Page 9: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/9.jpg)
History of the (Modern) Distributed World – Part
III● Cloud computing
● Infrastructure as a service● Virtualization
● Mobile computing● Sensor networks● Smart phones
● The Networked World
![Page 10: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/10.jpg)
Problem
● How to write high-performance applications for real-world distributed systems?
● How to integrate many different resources?
![Page 11: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/11.jpg)
Our approach
● Study fundamental underlying problems
● … hand-in-hand with realistic applications
● … integrate solutions in one system: Ibis
Distributed SystemsUseUserr
!
![Page 12: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/12.jpg)
Fundamental problems
● Performance – efficiency on wide-area systems
● Heterogeneity – different systems & APIs● Malleability – resources come and go● Fault tolerance - crashes● Connectivity – firewalls, NAT, etc.● Security – very hard
![Page 13: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/13.jpg)
Case study: spam filters
#1 Dutch Computer scientist
Top security-expert
CONGRATULATIONS!! YOU HAVE WON2.5 MEURO!!
Dear prof. Tanenbaum,
You may not know me,but I’d like to give you2.5 Million Euro to do research. Please come to Brussels to collect the money.
Yours sincerely,
Mr. V.I. Person
![Page 14: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/14.jpg)
Applications
● Scientific applications● Imaging (VUMC, AMOLF)● Bioinformatics (sequence analysis, cell
modeling)● Astronomy (data analysis challenge)
● Multimedia content analysis● Games and model checking● Semantic web (distributed reasoning)
![Page 15: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/15.jpg)
Multimedia content analysis
● Automatically extract information from images & video
● Extract feature vectors from images● Describe properties (color, shape)● Data-parallel task on a cluster
● Compute on consecutive images● Task-parallelism on a grid
![Page 16: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/16.jpg)
MMCA
‘Most Visionary Research’ award at AAAI 2007, (Frank Seinstra et al.)
![Page 17: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/17.jpg)
Games and Model Checking
● Can solve entire Awari game on wide-area DAS-3● Needs 10G private optical network (StarPlane)
● Distributed model checking has very similar communication pattern● Search huge state spaces, random work
distribution, bulk asynchronous transfers
● Can efficiently run DeVinE model checker on wide-area DAS-3, use up to 1 TB memory
![Page 18: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/18.jpg)
Awards
SCALE 2008
DACH 2008 - BS DACH 2008 - FT
AAAI-VC 2007
ISWC 2008 Multimedia
Computing
Astronomy
Semantic Web (van Harmelen et al.)
![Page 19: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/19.jpg)
Outline
● History of distributed systems
● Clusters, grids, clouds, networked world
● Programming distributed systems
● Driving applications
● The Ibis system
● Ibis on smartphones
![Page 20: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/20.jpg)
Ibis Philosophy● Real-world distributed applications should
be developed and compiled on a local workstation, and simply be launched from there
![Page 21: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/21.jpg)
Ibis Approach● Virtual Machines (Java) deal with
heterogeneity● Provide range of programming
abstractions● Designed for dynamic/faulty environments● Easy deployment through middleware-
independent programming interfaces● Modular and flexible: can replace Ibis
components by external ones
![Page 22: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/22.jpg)
Ibis Design
● Functionality from programming languages● High-Performance
Application Programming System
● Functionality fromoperating systems● Distributed Application
Deployment System
![Page 23: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/23.jpg)
Ibis System
![Page 24: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/24.jpg)
Programming system
● Programming models:● Message passing (RMI, MPJ)
● Divide-and-conquer (Satin)
● Master-worker (Maestro)
● Jorus: (multimedia applications)
● IPL (Ibis Portability Layer)● Java-centric “run-anywhere” library● Point-to-point, multicast, streaming, ….● Simple model (Join-Elect-Leave ) for tracking
resources, supports malleability & fault-tolerance
![Page 25: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/25.jpg)
SmartSockets library
● Detects connectivity problems● Tries to solve them automatically
● With as little help from the user as possible● Integrates existing and several new
solutions● Reverse connection setup, STUN, TCP splicing,
SSH tunneling, smart addressing, etc.● Uses network of hubs as a side channel
![Page 26: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/26.jpg)
Example
![Page 27: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/27.jpg)
Example
![Page 28: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/28.jpg)
Deployment system
● IbisDeploy GUI
● JavaGAT:● Java Grid Application Toolkit
● Make applications independent of underlying middleware
● Zorilla P2P system● Jobs management, gossiping,
clustering, flood scheduling
![Page 29: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/29.jpg)
● Runs simultaneously on clusters (DAS-3, Japan, Australia), Desktop Grid, Amazon EC2 Cloud
● Connectivity problems solved automatically by Ibis SmartSockets
Multimedia Content Analysis
Client
Broker
Servers IbisIbis
(Java)
![Page 30: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/30.jpg)
Connection management
With SmartSockets: run everywhere
Standard sockets: only local VU machines can be reached due to firewalls problems
![Page 31: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/31.jpg)
Ibis movie (part 1)
![Page 32: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/32.jpg)
Performance on 1 DAS-3 cluster
● Relative speedups of Java/Ibis and C++/MPI● Using TCP or Myricom’s MX protocol
● Sequential performance Java: 80% of C++
![Page 33: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/33.jpg)
Ibis Performance (wide area)
● Wide-area DAS:● Frame-rate increases linearly with
#clusters from 1 frame/sec to 4 frames/sec
● World-wide experiments: 22 frames/sec
![Page 34: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/34.jpg)
Smart Phones
● GSM + PC + GPS + camera + networks + ….
● Location-aware● What if everyone always carries a smart
phone (like a GSM now)?● Next wave in computing?
![Page 35: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/35.jpg)
Ibis on Smart Phones● Our focus: distributed smart phone
applications● Applications running on multiple phones● Integration with distributed computing
backbone
● Use Android for development● Google’s open-source platform● Java-based
![Page 36: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/36.jpg)
Distributed applications● Disaster management (Katrina)
● Use ad-hoc Wifi network when GSM network fails
● Finding nearby people with certain skills● Bus drivers, CPR
● Distributed decision support● Moving people to shelters (logistics)
● Social networks● Similar issues
● Find nearby friends, decide on restaurant
![Page 37: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/37.jpg)
Wild example
● Track position => automatic diary of your life
● Cross-comparisons between diariesHaven’t we met before?
Yes, on 23 Oct 2010, 3.48 pm atN 52°22.688´ E 004°53.990´
![Page 38: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/38.jpg)
eyeDentify● Object recognition on a G1 smartphone● Smartphone is a limited device:
● Can run only 64 x 48 pixels (memory bound)● 1024 x 768 pixels would take 5 minutes
● Distributed Ibis version:
+ =+1024 x 768 pixels
2.0 seconds
![Page 39: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/39.jpg)
Ibis movie (part 2)
![Page 40: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/40.jpg)
Interdroid
Distributed Communication
Data Management
Novel Mobile Distributed Applications
Context SensitiveProgrammingModels
![Page 41: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/41.jpg)
Current work● Raven: API for Viable Episodic Networking
● Decentralized synchronization API● Fine grained control over data sharing
● Bluetooth support for ad-hoc communication● Discovery of devices using multiple networks● Context Aware Programming Models
● Supporting distributed decision making● Representing and using context (location etc.)● Exploiting social relationships (Hyves, Facebook)
![Page 42: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/42.jpg)
Summary
● It’s a wild (distributed) world
![Page 43: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/43.jpg)
Acknowledgements
Niels DrostCeriel JacobsRoelof Kemp
Timo van KesselThilo KielmannJason Maassen
Rob van NieuwpoortNick Palmer
Kees van ReeuwijkFrank J. SeinstraKees Verstoep
Gosia Wrzesinska
![Page 44: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/44.jpg)
Big Acknowledgement
Andy
![Page 45: The Ibis model as a paradigm for programming distributed systems Henri Bal bal@cs.vu.nl Vrije Universiteit Amsterdam (from Grids and Clouds to Smartphones)](https://reader035.vdocument.in/reader035/viewer/2022062515/56649cba5503460f949818a2/html5/thumbnails/45.jpg)
Questions?