Download - Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India
![Page 1: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/1.jpg)
Idempotent Transactional Workflow
(POPL 2013)
G. RamalingamKapil Vaswani
Microsoft Research India
![Page 2: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/2.jpg)
Application
The Problem
Partitioned Data
scale-out
Can we simplify
writing suchapplications?
![Page 3: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/3.jpg)
Transfer (amt, acct1, acct2) {Debit amt from acct1;Credit amt to acct2;
}
![Page 4: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/4.jpg)
Transfer (amt, acct1, acct2) atomic {
Debit amt from acct1; Credit amt to acct2;
}
ACID Transaction+ Strong consistency− Distributed
transaction
![Page 5: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/5.jpg)
Transfer (amt, acct1, acct2) atomic {Debit …}; atomic {Credit …};
Workflow− Weaker consistency
− No isolation+ No distributed transaction
What about process failure? Claim: Workflows are common in applications over partitioned data
![Page 6: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/6.jpg)
Storage Layer
Application Logic
Stopping (non-byzantine) failure
The Problem
(failures handled by storage layer)
Goal• Fault-tolerance in application• A transactional workflow
engine• decentralized!
Modern Cloud Platforms
![Page 7: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/7.jpg)
request response
Making Workflows Fault-Tolerant
![Page 8: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/8.jpg)
Request or response
may be lost!
Taking a step back …
Resending messages
is a critical elementof fault-tolerance
Must be Idempotent!
(tolerate duplicatemessages)
Transfer (amt, acct1, acct2) {Debit amt from acct1;Credit amt to acct2;
}
![Page 9: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/9.jpg)
Goal:Idempotent Fault-Tolerance
• (Idempotent Workflow)• A program is said to be idempotent & fault-
tolerant iff– its behavior is unaffected by process failures– its behavior is unaffected by duplicate input
requests
• Behavioral equivalence:– duplicate output responses allowed– progress (liveness) conditions
• slightly weakened
![Page 10: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/10.jpg)
request response
Making WorkflowsIdempotent & Fault-Tolerant
![Page 11: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/11.jpg)
request response
Making Computations Idempotent
Make every effectful step idempotent:1. Associate unique id with every step2. Modify step to log execution of step3. Modify step to check if it has already
executedAll must be done atomically !
![Page 12: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/12.jpg)
AutomatedIdempotent Fault-Tolerance
• As a library– In C# & F#– Technically, a monad
• As a compiler
• As a programming-language construct
![Page 13: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/13.jpg)
Formal Results
Theorem. A well-typed monadic program isidempotent and fault-tolerant.
Theorem. compile[e] is an idempotent and fault-tolerant realization of e.
Any (well-typed) program e can be automatically translated (compiled) into a program compile[e]
![Page 14: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/14.jpg)
Idempotence: A Language Construct
• “idworkflow uid e’’
transfer (uid, amt, acct1, acct2) { idworkflow uid {
atomic T1 Debit amt from acct1 atomic T2 Credit amt to acct2}}
}
![Page 15: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/15.jpg)
Extensions
• Compensating actions– Undo earlier actions when later actions
encounter logical failure
• Automatic retry– Detect process failures & restart
• Checkpointing– Restart at most recent checkpoint
![Page 16: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/16.jpg)
Questions?
Fault-Tolerance & Idempotence: Simpler Together
![Page 17: Idempotent Transactional Workflow (POPL 2013) G. Ramalingam Kapil Vaswani Microsoft Research India](https://reader030.vdocument.in/reader030/viewer/2022032722/56649cf05503460f949c01c5/html5/thumbnails/17.jpg)
Storage Layer
Application Logic
client
service
partitioneddata
Problem Setting