![Page 1: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/1.jpg)
Disorderly programming for a distributed world
Peter AlvaroUC Berkeley
![Page 2: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/2.jpg)
Joint work
• Peter Alvaro• Neil Conway• Joseph M. Hellerstein• William R. Marczak
• National Science Foundation• Air Force Office of Scientific Research• Gifts from IBM, Microsoft and Yahoo! Research
Thanks to
![Page 3: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/3.jpg)
The future is already here
• All systems are (or are becoming) distributed• Programming distributed systems is hard• Reasoning about them is harder
![Page 4: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/4.jpg)
Outline
1. Disorderly Programming2. The Bloom programming language3. CALM Analysis and visualization4. Challenge app: replicated shopping carts
![Page 5: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/5.jpg)
Programming distributed systems
![Page 6: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/6.jpg)
The state of the art
Order is pervasive in the Von Neumann model
• Program state is an ordered array• Program logic is a sequence of instructions
![Page 7: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/7.jpg)
The state of the art
Order is pervasive in the Von Neumann model
Parallelism and concurrency via retrofits:• Threads• Event-driven programming
![Page 8: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/8.jpg)
The state of the art
In distributed systems, order is
![Page 9: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/9.jpg)
The state of the art
In distributed systems, order is• expensive to enforce
![Page 10: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/10.jpg)
The state of the art
In distributed systems, order is• expensive to enforce• often unnecessary
![Page 11: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/11.jpg)
The state of the art
In distributed systems, order is• expensive to enforce• often unnecessary• easy to get wrong
![Page 12: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/12.jpg)
The art of the stateThe state of the art
![Page 13: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/13.jpg)
The art of the state
Disorderly programming
![Page 14: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/14.jpg)
The art of the state
Disorderly programming:
Computation as transformation
![Page 15: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/15.jpg)
The art of the state
Disorderly programming:• Program state is unordered collections• Program logic is an unordered ``bag’’ of rules
![Page 16: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/16.jpg)
The art of the state
Disorderly programming:• Independence and concurrency are assumed• Ordering is explicit
![Page 17: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/17.jpg)
The art of the state
Disorderly programming
![Page 18: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/18.jpg)
BUD: Bloom Under Development
• Ruby internal DSL• Set comprehension style of programming• Declarative semantics
![Page 19: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/19.jpg)
Operational model
![Page 20: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/20.jpg)
Bloom Rules do |mes, mem| [mem.address, mes.id, mes.payload]end
multicast <~ (message * members)
![Page 21: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/21.jpg)
Bloom Rules do |mes, mem| [mem.address, mes.id, mes.payload]end
multicast <~ (message * members)
![Page 22: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/22.jpg)
Bud language features
• Module system– Encapsulation and composition via mixins– Abstract interfaces, concrete implementations
• Metaprogramming and reflection– The program is data
• Pay-as-you-code schemas– Default is key => value
• CALM Analysis
![Page 23: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/23.jpg)
Writing distributed programs in Bloom
![Page 24: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/24.jpg)
Abstract Interfaces and Declarations
module DeliveryProtocol state do interface input, :pipe_in,
[:dst, :src, :ident] => [:payload] interface output, :pipe_sent, pipe_in.schema interface output, :pipe_out, pipe_in.schema endend
![Page 25: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/25.jpg)
Concrete Implementationsmodule BestEffortDelivery include DeliveryProtocol
state do channel :pipe_chan, pipe_in.schema end
bloom :snd do pipe_chan <~ pipe_in end
bloom :done dopipe_sent <= pipe_inpipe_out <= pipe_chan
endend
Asynchrony
![Page 26: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/26.jpg)
A simple key/value storemodule KVSProtocol state do interface input, :kvput, [:key] => [:reqid, :value] interface input, :kvdel, [:key] => [:reqid] interface input, :kvget, [:reqid] => [:key] interface output, :kvget_response, [:reqid] => [:key, :value] endend
![Page 27: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/27.jpg)
A simple key/value storemodule BasicKVS include KVSProtocol
state do table :kvstate, [:key] => [:value] end
bloom :mutate do kvstate <+ kvput {|s| [s.key, s.value]} kvstate <- (kvstate * kvput).lefts(:key => :key) end
bloom :get do temp :getj <= (kvget * kvstate).pairs(:key => :key) kvget_response <= getj do |g, t| [g.reqid, t.key, t.value] end end
bloom :delete do kvstate <- (kvstate * kvdel).lefts(:key => :key) endend
Nonmonotonicoperation
![Page 28: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/28.jpg)
CALM Analysis
![Page 29: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/29.jpg)
Asynchronous messaging
You never really know
![Page 30: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/30.jpg)
Asynchronous messaging
A B
C sendA B
C
![Page 31: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/31.jpg)
Monotonic Logic
The more you know, the more you know.
![Page 32: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/32.jpg)
Monotonic Logic
A B
C
E
D
A
C
E
select/filter
![Page 33: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/33.jpg)
Monotonic Logic
A B
C project /map
f(A) f(B)
f(C)
![Page 34: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/34.jpg)
Monotonic Logic
A BC D E
B
D
B
Djoin / compose
F
![Page 35: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/35.jpg)
Monotonic Logic is order-insensitive
A B
C
E
D
A
C
E
![Page 36: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/36.jpg)
Monotonic Logic is pipelineable
A B
C
E
D
A
C
E
![Page 37: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/37.jpg)
Nonmonotonic Logic
When do you know for sure?
![Page 38: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/38.jpg)
Nonmonotonic Logic
A BC D E
B
D
set minus A B
CD
E
Retraction!
Retraction!
X
Y
Z
![Page 39: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/39.jpg)
Nonmonotonic logic is order-sensitive
A BC D E
B
D
set minus A
C E
![Page 40: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/40.jpg)
Nonmonotonic logic is blocking
A
set minus A ?
A?
![Page 41: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/41.jpg)
Nonmonotonic logic is blocking
A
set minusA?
``Sealed’’
![Page 42: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/42.jpg)
CALM Analysis
• Asynchrony => loss of order• Nonmonotonicity => order-sensitivity• Asynchrony ; Nonmonotonicity =>
Inconsistency
[…]
![Page 43: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/43.jpg)
CALM Analysis
• Asynchrony => loss of order• Nonmonotonicity => order-sensitivity• Asynchrony ; Nonmonotonicity =>
Inconsistency
?
``Point of Order’’
[…]
![Page 44: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/44.jpg)
Resolving points of order
![Page 45: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/45.jpg)
Resolving points of order
1. Ask for permission
![Page 46: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/46.jpg)
Resolving points of order
1. Ask for permission
Coordination => strong consistency
![Page 47: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/47.jpg)
Resolving points of order
1. Ask for permission2. Ask for forgiveness
![Page 48: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/48.jpg)
Resolving points of order
1. Ask for permission2. Ask for forgiveness
Compensation, weak consistency
![Page 49: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/49.jpg)
Resolving points of order
1. Ask for permission2. Ask for forgiveness3. Ask differently?
Rewrite to reduce consistency cost…
![Page 50: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/50.jpg)
Shopping Carts
![Page 51: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/51.jpg)
Replicated Shopping Carts
Replicated for high availability and low latency
Challenge: Ensure that replicas are ``eventually consistent’’
![Page 52: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/52.jpg)
Replicated Shopping Carts
module CartClientProtocol state do
interface input, :client_action, [:server, :session, :reqid] => [:item, :action]
interface input, :client_checkout,[:server, :session, :reqid]
interface output, :client_response, [:client, :server, :session] => [:items]
endend
![Page 53: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/53.jpg)
Replicated Shopping Carts
module CartClientProtocol state do
interface input, :client_action, [:server, :session, :reqid] => [:item, :action]
interface input, :client_checkout,[:server, :session, :reqid]
interface output, :client_response, [:client, :server, :session] => [:items]
endend
![Page 54: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/54.jpg)
Carts done two ways
1. A “destructive” cart2. A “disorderly” cart
![Page 55: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/55.jpg)
``Destructive’’ Cartmodule DestructiveCart include CartProtocol include KVSProtocol
bloom :on_action do kvget <= action_msg {|a| [a.reqid, a.session] } kvput <= (action_msg * kvget_response).outer(:reqid => :reqid) do |a,r| val = (r.value || {}) [a.client, a.session, a.reqid, val.merge({a.item => a.action}) do
|k,old,new| old + new end]
end end
bloom :on_checkout do kvget <= checkout_msg {|c| [c.reqid, c.session] } response_msg <~ (kvget_response * checkout_msg).pairs
(:reqid => :reqid) do |r,c| [c.client, c.server, r.key, r.value.select {|k,v| v > 0}.to_a.sort] end endend
![Page 56: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/56.jpg)
``Destructive’’ Cartmodule DestructiveCart include CartProtocol include KVSProtocol
bloom :on_action do kvget <= action_msg {|a| [a.reqid, a.session] } kvput <= (action_msg * kvget_response).outer(:reqid => :reqid) do |a,r| val = (r.value || {}) [a.client, a.session, a.reqid, val.merge({a.item => a.action}) do
|k,old,new| old + new end]
end end
bloom :on_checkout do kvget <= checkout_msg {|c| [c.reqid, c.session] } response_msg <~ (kvget_response * checkout_msg).pairs
(:reqid => :reqid) do |r,c| [c.client, c.server, r.key, r.value.select {|k,v| v > 0}.to_a.sort] end endend
React to client updates
React to client checkout
![Page 57: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/57.jpg)
Destructive Cart Analysis
![Page 58: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/58.jpg)
Destructive Cart Analysis
Collapsed SCC
Asynchrony
Nonmonotonicity
Divergent results?
![Page 59: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/59.jpg)
Destructive Cart Analysis
Add coordination? E.g.,• Synchronous replication• Paxos
n = |client_action|m = |client_checkout| = 1
n rounds of coordination
![Page 60: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/60.jpg)
``Disorderly Cart’’module DisorderlyCart include CartProtocol
state do table :action_log, [:session, :reqid] => [:item, :action] scratch :item_sum, [:session, :item] => [:num] scratch :session_final, [:session] => [:items, :counts] end
bloom :on_action do action_log <= action_msg { |c| [c.session, c.reqid, c.item, c.action] } end
bloom :on_checkout do temp :checkout_log <= (checkout_msg * action_log).rights(:session => :session) item_sum <= checkout_log.group([action_log.session, action_log.item], sum(action_log.action)) do |s|
s if s.last > 0 end session_final <= item_sum.group([:session], accum(:item), accum(:num)) response_msg <~ (session_final * checkout_msg).pairs(:session => :session) do |c,m| [m.client, m.server, m.session, c.items.zip(c.counts).sort] end endend
![Page 61: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/61.jpg)
``Disorderly Cart’’module DisorderlyCart include CartProtocol
state do table :action_log, [:session, :reqid] => [:item, :action] scratch :item_sum, [:session, :item] => [:num] scratch :session_final, [:session] => [:items, :counts] end
bloom :on_action do action_log <= action_msg { |c| [c.session, c.reqid, c.item, c.action] } end
bloom :on_checkout do temp :checkout_log <= (checkout_msg * action_log).rights(:session => :session) item_sum <= checkout_log.group([action_log.session, action_log.item], sum(action_log.action)) do |s|
s if s.last > 0 end session_final <= item_sum.group([:session], accum(:item), accum(:num)) response_msg <~ (session_final * checkout_msg).pairs(:session => :session) do |c,m| [m.client, m.server, m.session, c.items.zip(c.counts).sort] end endend
Actions
Checkout
![Page 62: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/62.jpg)
Disorderly Cart Analysis
Asynchrony
Nonmontonicity
Divergent Results?
![Page 63: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/63.jpg)
Disorderly Cart Analysis
n = |client_action|m = |client_checkout| = 1
1 round of coordination
![Page 64: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/64.jpg)
Replicated Disorderly Cart
Asynchronous (uncoordinated)replication
Still just 1 roundof coordination
![Page 65: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/65.jpg)
Teaching
![Page 66: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/66.jpg)
Summary
• Why disorderly?– Order is a scarce (and distracting!) resource
• When is order really needed?– To resolve nonmonotonicity
• What is coordination for?– Re-establishing order, to guarantee consistency.
• CALM – A disorderly programming language– Tools to identify points of order
![Page 67: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/67.jpg)
More
Resources:http://boom.cs.berkeley.eduhttp://bloom-lang.org
Writeups:• Consistency Analysis in Bloom: A CALM and Collected Approach (CIDR’11)• Dedalus: Datalog in Time and Space (Datalog2.0)• The Declarative Imperative (PODS’10 Keynote address)• Model-theoretic Correctness Criteria for Distributed Systems (in
submission)
![Page 68: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/68.jpg)
Queries?
![Page 69: Disorderly programming for a distributed world](https://reader035.vdocument.in/reader035/viewer/2022062400/56816930550346895de07991/html5/thumbnails/69.jpg)
Languages regarding languages
Other Languages Bloom
Bloom
Other Languages