linked process
DESCRIPTION
A presentation on the XMPP-based Linked Process distributed computing framework.TRANSCRIPT
Linked ProcessAn Internet-scale distributed computing
framework
Center For Nonlinear Studies
Joshua Shinavier
August 5th, 2009
• Internet-scale distributed computing
• eXtensible Messaging and Presence Protocol (XMPP)
• Linked Process specification
• Current Linked Process implementation
• Demos
Overview
2
Abstract
3
The LANL-based Linked Process project takes a new approach to Internet-scale distributed computing. While existing large-scale
grid computing projects are typically very constrained in the kinds of computational tasks which can be performed, the kinds of
devices which can participate in computation, and in the overall architecture of the system, the Linked Process specification
provides the foundation for a much larger and more general-purpose distributed computing platform. Any device supporting the eXtensible Messaging and Presence Protocol (XMPP), be it a
supercomputer or a cellular phone, is a potential node in a global compute cloud, communicating with other nodes in a manner similar to human chat. The implementation currently under development provides a simple API and supports a number of
popular scripting languages, allowing software developers to write distributed applications with ease. This presentation will provide
an overview of the Linked Process specification and discuss a range of potential uses of the technology.
• Marko A. Rodriguez (LANL)
• http://markorodriguez.com/
• Joshua Shinavier (RPI / LANL)
• http://fortytwo.net/
• Peter Neubauer (Neo Technology)
• http://www.linkedin.com/neubauer/
• Max O. Bond (Santa Fe Complex)
• Mick Thompson (Santa Fe Complex)
• http://davidmichaelthompson.com
Contributors (to date)
4
• distributed computing
• combines computational power of multiple machines
• makes effective use of local resources
• Berkeley Open Infrastructure for Network Computing (BOINC)
• supports SETI@home, PrimeGrid, etc.
• cloud computing
• Amazon EC2
• Google App Engine
Internet-scale distributed computing
5
• eXtensible Messaging and Presence Protocol
• deals with “presence” and asynchronous message passing among clients and servers
• open standard
• based on machine-independent Jabber identifiers (JIDs)
6
XMPP is everywhere
7
• uses XMPP messaging for inter-machine communication
• any XMPP-enabled device may participate
• bring mobile devices into the cloud
• augment the compute power of a single device
• grid computing
• specification is called LoP, for “Linking Open Processors”
8
Linked Process
LoP is like chat...
9
... but among software agents
10
LoP architecture
11
• a job is a task to be performed by a virtual machine, e.g.
• computationally intensive operations
• manipulation of local resources
• LoP allows you to:
• submit a job -- <submit_job/>
• check on the status of a job -- <job_status/>
• abort a job -- <abort_job/>
Jobs: units of computation
12
Example: submitting a job
13
<iq from="[email protected]/LoPVillein/1234"
to="[email protected]/LoPVM/EFGH"
type="get" id="xxxx">
<submit_job xmlns="http://linkedprocess.org/2009/06/VirtualMachine#"
vm_password="abc123pass">
var temp=0;
for(i=0; i<10; i++) {
temp = temp + 1;
}
temp;
</submit_job>
</iq>
<iq from="[email protected]/LoPVM/EFGH"
to="[email protected]/LoPVillein/1234"
type="result" id="xxxx">
<submit_job xmlns="http://linkedprocess.org/2009/06/VirtualMachine#">
10
<submit_job/>
</iq>
request
response
• VM is maintained by an XMPP client
• manages jobs and data-typed “bindings” (variables) -- <manage_bindings/>
• provides a scripting environment using a particular language (e.g. JavaScript, Ruby, etc.)
• may be terminated at any time -- <terminate_vm/>
Virtual machines: addressable “computers” in the cloud
14
Example: spawning a VM
15
<iq from="[email protected]/LoPVillein/1234"
to="[email protected]/LoPFarm/ABCD"
type="get" id="xxxx">
<spawn_vm xmlns="http://linkedprocess.org/2009/06/Farm#"
vm_species="javascript" />
</iq>
<iq from="[email protected]/LoPFarm/ABCD"
to="[email protected]/LoPVillein/1234"
type="result" id="xxxx">
<spawn_vm xmlns="http://linkedprocess.org/2009/06/Farm#"
vm_jid="[email protected]/LoPVM/EFGH"
vm_password="abc123pass"
vm_species="javascript" />
</iq>
request
response
• farm is maintained by an XMPP client
• provides access to virtual machines
• farm allows you to:
• spawn a virtual machine -- <spawn_vm/>
• query for information about the environment (e.g. language support, security restrictions, etc.) -- disco#info
• multiple farms may share the same “bare” JID
Farms: LoP service providers
16
Example: service discovery
17
<iq id="zzzz" from="[email protected]/LoPFarm/ABCD" to="[email protected]/LoPVIllein/1234" type="result"> <query xmlns="http://jabber.org/protocol/disco#info"> <identity category="client" name="LoPVM" type="bot" /> <feature var="http://jabber.org/protocol/disco#info" /> <feature var="http://linkedprocess.org/2009/06/Farm#" /> <x xmlns="jabber:x:data" type="result"> <field label="supported virtual machine species" var="vm_species" type="list-single"> <option label="groovy 1.5.6 (groovy )"> <value>groovy</value> </option> <option label="javascript 1.6 (Mozilla Rhino 1.6R7)"> <value>javascript</value> </option> ... </field> <field label="exercise a permission" var="permission" type="boolean"> <value>1</value> </field> <field label="exit the farm process" var="exit" type="boolean"> <value>0</value> </field> <field label="execute a program" var="exec" type="list-multi" /> <field label="create a file system link" var="link" type="list-multi" /> ... </x> </query></iq>
• jobs operate within a VM sandbox
• subject to named permissions, e.g.
• file I/O
• network I/O
• introspection
• password-protection of VMs
• can specify limits on VMs per farm, number of jobs in a queue, VM and job timeouts, etc.
• XMPP supports SSL (Secure Socket Layer)
Security
18
• based on Java 1.6
• takes advantage of built-in Java security, scripting framework
• job scheduler serves as a miniature operating system
• supports a number of scripting languages
• JavaScript, Ruby, Python, Groovy
• support for additional languages is easy to add
• https://scripting.dev.java.net/
• deployed farms in New Mexico, New York, Sweden
Implementation (to date)
19
AWK
BeanShell
ejs
FreeMarker
Groovy
Jaskell
Java
JavaScript
JSR 223 supports...
20
Jelly
JEP
Jexl
jst
JudoScript
JUEL
OGNL
Pnuts
Python
Ruby
Scheme
Sleep
Tcl
Velocity
XPath
XSLT
• distributed primality testing
• Linked Data
• for the future:
• LoP API for Google’s MapReduce
• computational support for mobile devices
• distributed matrix operations, image processing, etc.
Demos
21
• http://linkedprocess.org/
• Rodriguez, M.A., “A Reflection on the Structure and Process of the Web of Data,” Bulletin of the American Society for Information Science and Technology, American Society for Information Science and Technology, volume 35, number 6, ISSN: 1550-8366, LA-UR-09-03724, pages 38-43, August 2009.
• XMPP Core spec: http://xmpp.org/rfcs/rfc3920.html
• XMPP Instant Messaging and Presence spec: http://xmpp.org/rfcs/rfc3921.html
• XMPP Extensions: http://xmpp.org/extensions/
See also
22