the elements of decision alignment · the elements of decision alignment human to human human...
TRANSCRIPT
![Page 1: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/1.jpg)
The Elements of Decision Alignment
Mark S. Miller Bill Tulloh
UCI 2017
![Page 2: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/2.jpg)
The Elements of Decision Alignment
Mark S. Miller (cs) Bill Tulloh (econ)
UCI 2017
![Page 3: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/3.jpg)
When one object makes a request of another object, why do we expect the second object’s behavior to satisfy the first object’s wishes?
![Page 4: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/4.jpg)
Networks of entities making requests of other entities:
• Object-oriented programs • Human organizations • Human economies
![Page 5: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/5.jpg)
Borrowing Ideas from Economics
“Like an economist … we are interested in individual agents not so much for what they are internally as for what they have to offer each other
… much of object-oriented design is indeed Design by Contract.”
Object-Oriented Software ConstructionBertrand Meyer, 1997
![Page 6: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/6.jpg)
Overview
Making Requests
Aligning Decisions
Making Tradeoffs
Dividing and Composing Knowledge
![Page 7: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/7.jpg)
Making Requests
Parsing Gift for Dad
Stack Package delivery
Array+index Truck
Principal why
Agent how
Interface what
![Page 8: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/8.jpg)
Principal-Agent relationships
“Give this to my Dad?”Human to Human
(econ)
![Page 9: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/9.jpg)
Principal-Agent relationships
“Give this to my Dad?”Human to Human
(econ)
![Page 10: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/10.jpg)
Object to Object (software eng)
Human to Object (ui)
Object to Human (ui)
Human to Human (econ)
Principal-Agent relationships
![Page 11: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/11.jpg)
Principal-Agent networks
Object to Object (software eng)
Human to Object (ui)
Object to Human (ui)
Human to Human (econ)
![Page 12: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/12.jpg)
Definitions
From Econ: A Principal sends a request to an Agent. An Agent reacts to a request from a Principal.
![Page 13: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/13.jpg)
Definitions
From Econ: A Principal sends a request to an Agent. An Agent reacts to a request from a Principal.
Incentive Alignment is when a (human) Principal or Agent uses incentives to induce the other’s intentions to align with their own.
![Page 14: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/14.jpg)
Definitions
From Econ: A Principal sends a request to an Agent. An Agent reacts to a request from a Principal.
Incentive Alignment is when a (human) Principal or Agent uses incentives to induce the other’s intentions to align with their own.
align: Compose well without interference.
![Page 15: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/15.jpg)
Definitions
From Econ: A Principal sends a request to an Agent. An Agent reacts to a request from a Principal.
Incentive Alignment is when a (human) Principal or Agent uses incentives to induce the other’s intentions to align with their own.
align: Compose well without interference.
From us: Decision Alignment is when a Principal or Agent uses various tools to make it more likely for the other’s decisions and actions to align with their own.
![Page 16: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/16.jpg)
Definitions
From us: Decision Alignment is when a Principal or Agent uses various tools to make it more likely for the other’s decisions and actions to align with their own.
![Page 17: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/17.jpg)
Definitions
From us: Decision Alignment is when a Principal or Agent uses various tools to make it more likely for the other’s decisions and actions to align with their own.
This talk: Decision Alignment is when a Principal uses various tools to make it more likely for the Agent’s decisions and actions to align with its own.
![Page 18: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/18.jpg)
Information Hiding BenefitsCompose specialized knowledge
Parsing Gift for Dad
Stack Package delivery
Array+index Truck
Principal why
Agent how
Interface what
Only Principal knows whyOnly Agent knows how Shared knowledge burden: only what’s needed for request Minimize cascading changes
![Page 19: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/19.jpg)
Hidden Information HazardsThe principal-agent problem
Parsing Gift for Dad
Stack Package delivery
Array+index Truck
Principal why
Agent how
Interface what
Pre: Can the agent do what I want? Request: Will the agent try to do what I want? Post: Is the agent doing what I want?
![Page 20: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/20.jpg)
The Principal-Agent Loop
Request/Contract PostPre
Hidden characteristics
Adverse Selection
Execute the request
Incentive Alignment
Hidden actions
Moral Hazard
Econ: intentional misbehavior
![Page 21: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/21.jpg)
The Principal-Agent Loop
PostPre
Hidden characteristics Execute the request Hidden actions
Econ: intentional misbehavior CS: accidental misbehavior
Request/Contract
![Page 22: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/22.jpg)
The Principal-Agent Loop
PostPre
Hidden characteristics
Select agent. Screening. Agent signals.
Inspect internals. Abilities, limits.
Execute the request
Allow actions. Scope of authority.
Explain request. What the agent is supposed to do.
Reward cooperation. If agent does that.
Hidden actions
Monitor effects. What agent is doing, or did.
Feedback to guide future selection.
Request/Contract
![Page 23: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/23.jpg)
The Principal-Agent Loop
Request/ContractPre
Post
The Principal-Agent LoopOnly loosely ordered
![Page 24: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/24.jpg)
Ince
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
The Principal-Agent Loop
Pre
Post
The Principal-Agent LoopOnly loosely ordered
Request/Contract
![Page 25: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/25.jpg)
Ince
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
The Principal-Agent Loop
![Page 26: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/26.jpg)
Human toHuman
Human to Object
Object toObject
Object toHuman
Select agent
Inspect internals
Allow actions
Explain request
Reward cooperation
Monitor effects
From Incentive Alignment …
Incentive Alignment
![Page 27: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/27.jpg)
Human toHuman
Human to Object
Object toObject
Object toHuman
Select agent
Inspect internals
Allow actions
Explain request
Reward cooperation
Monitor effects
Incentive Alignment
… to Principal-Agent … Recognize synergies
Joint application
![Page 28: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/28.jpg)
Human toHuman
Human to Object
Object toObject
Object toHuman
Select agent
Inspect internals
Allow actions
Explain request
Reward cooperation
Monitor effects
… to Decision Alignment Recognize commonalities
Unify, GeneralizeIncentive Alignment
Joint application
![Page 29: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/29.jpg)
Human toHuman
Human to Object
Object toObject
Object toHuman
Select agent
Inspect internals
Allow actions
Explain request
Reward cooperation
Monitor effects
… to Decision Alignment Recognize commonalities
Unify, Generalize
Joint application
![Page 30: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/30.jpg)
![Page 31: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/31.jpg)
Ince
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
Package DeliveryFit, Reputation
![Page 32: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/32.jpg)
Ince
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
Package DeliveryHand over the package
![Page 33: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/33.jpg)
Ince
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
Package DeliveryDelivery address, instructions
![Page 34: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/34.jpg)
Ince
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
Package DeliveryPay
![Page 35: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/35.jpg)
Ince
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
Package DeliveryHope and pray?
![Page 36: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/36.jpg)
Ince
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
Package DeliveryTrack, Return receipt
![Page 37: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/37.jpg)
Ince
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
Package DeliveryReputation feedback, Rating
![Page 38: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/38.jpg)
Ince
Reward cooperation
Explanation&
agent
Monitoring&
Explanation&
agent
Monitoring&
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Internal Software Development Static vs. Dynamic
Developers & code Object to object
![Page 39: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/39.jpg)
Ince
Reward cooperation
Explanation&
agent
Monitoring&
Explanation&
agent
Monitoring&
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Internal Software Development De-emphasize Rewards
![Page 40: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/40.jpg)
Ince
Explanation&
agent
Monitoring&
Explanation&
agent
Monitoring&
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
Internal Software Development De-emphasize Rewards
![Page 41: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/41.jpg)
Ince
Explanation&
agent
Monitoring&
Explanation&
agent
Monitoring&
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
Internal Software Development Hire the best. Find libraries.
![Page 42: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/42.jpg)
Ince
Explanation&
agent
Monitoring&
Explanation&
agent
Monitoring&
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
Internal Software Development Code reviews. Some static checking.
![Page 43: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/43.jpg)
Ince
Explanation&
agent
Monitoring&
Explanation&
agent
Monitoring&
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
Internal Software Development All user’s authority.
![Page 44: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/44.jpg)
Ince
Explanation&
agent
Monitoring&
Explanation&
agent
Monitoring&
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
Internal Software Development All user’s authority.
![Page 45: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/45.jpg)
Explanation&
agent
Monitoring&
Explanation&
agent
Monitoring&
Inspect internals
Select agent
Agent reacts
Monitor effects
AllowAllowAllow actions
Explain request
Internal Software DevelopmentRights per request
Reward cooperation
![Page 46: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/46.jpg)
Explanation&
agent
Monitoring&
Explanation&
agent
Monitoring&
Inspect internals
Select agent
Agent reacts
Monitor effects
AllowAllowAllow actions
Explain request
Internal Software DevelopmentRights per request
Reward cooperation
![Page 47: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/47.jpg)
Explanation&
agent
Monitoring&
Explanation&
agent
Monitoring&
Inspect internals
Select agent
Agent reacts
Monitor effects
AllowAllowAllow actions
Explain request
Internal Software DevelopmentAPI Design
Reward cooperation
![Page 48: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/48.jpg)
Explanation&
agent
Monitoring&
Explanation&
agent
Monitoring&
Inspect internals
Select agent
Agent reacts
Monitor effects
AllowAllowAllow actions
Explain request
Internal Software DevelopmentTesting. Bug reports.
Reward cooperation
![Page 49: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/49.jpg)
The Elements of Decision Alignment
Human toHuman
Human to/fromObject
Object toObject
Select agent
Trademark Chain of custody
App stores White and black lists
Trusted developer Same origin
Inspect internals Accounting controls Trusted path
URL barTypes, Verification
Open source eyeballs
Allow actions Law, Contracts App permissions
PowerboxSecurity
Protection patterns
Explain request Language User interface Abstraction
Reward cooperation
Economics Incentive Alignment Objective functions Machine learning
Agorics
Monitor effects
Reviews, Complaints Word of mouth Bug reports Contracts, Testing
Backprop
![Page 50: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/50.jpg)
![Page 51: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/51.jpg)
Tuning Tradeoffs
Select agent
Inspect internals
Allow actions
Explain request
Reward cooperation
Monitor effects
![Page 52: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/52.jpg)
Select agent Open Entry Gated
Inspect internals Code Review Verify
Allow actions Broad Least Authority
Explain request Informal Specified
Reward cooperation Guide Induce
Monitor effects Prevent Repair Damage
s
a
e
m
r
Tuning Tradeoffs
i
![Page 53: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/53.jpg)
Package Delivery Business
Select agent Open Entry Gated
Inspect internals Code Review Verify
Allow actions Broad Least Authority
Explain request Informal Specified
Reward cooperation Guide Induce
Monitor effects Prevent Repair Damage
s
a
e
m
r
![Page 54: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/54.jpg)
Internal Software Development
Select agent Open Entry Gated
Inspect internals Code Review Verify
Allow actions Broad Least Authority
Explain request Informal Specified
Reward cooperation Guide Induce
Monitor effects Prevent Repair Damage
s
i
a
e
m
![Page 55: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/55.jpg)
Organizational Employment
Select agent Open Entry Gated
Inspect internals Interview Clearance
Allow actions Broad Least Authority
Explain request Informal Specified
Reward cooperation Guide Induce
Monitor effects Prevent Repair Damage
s
a
e
m
r
i
![Page 56: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/56.jpg)
Safe Plugin Boundary
Select agent Open Entry Gated
Inspect internals Code Review Verify
Allow actions Broad Least Authority
Explain request Informal Specified
Reward cooperation Guide Induce
Monitor effects Prevent Repair Damage
s
a
e
m
r
![Page 57: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/57.jpg)
Bitcoin, Etherium
Select agent Open Entry Gated
Inspect internals Code Review Verify
Allow actions Broad Least Authority
Explain request Informal Specified
Reward cooperation Guide Induce
Monitor effects Prevent Repair Damage
s
i
a
e
m
r
![Page 58: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/58.jpg)
Etherium Hard Fork
Select agent Open Entry Gated
Inspect internals Code Review Verify
Allow actions Broad Least Authority
Explain request Informal Specified
Reward cooperation Guide Induce
Monitor effects Prevent Repair Damage
s
i
a
e
m
r
![Page 59: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/59.jpg)
Etherium once repaired
Select agent Open Entry Gated
Inspect internals Code Review Verify
Allow actions Broad Least Authority
Explain request Informal Specified
Reward cooperation Guide Induce
Monitor effects Prevent Repair Damage
s
i
a
e
m
r
![Page 60: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/60.jpg)
Building Reliable Voting Machine SoftwareKa-Ping Yee, 2007 dissertation
Programmer may wish to bias the election.
Programmer and code as untrusted agent.
![Page 61: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/61.jpg)
Building Reliable Voting Machine SoftwareKa-Ping Yee, 2007 dissertation
Programmer may wish to bias the election.
Must write simple code that seems obviously correct.
400 lines of simple code in simple language. Extensive rationale justifying each line.
![Page 62: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/62.jpg)
Building Reliable Voting Machine SoftwareKa-Ping Yee, 2007 dissertation
Programmer may wish to bias the election.
Must write simple code that seems obviously correct.
Subject to extremely intense review.
Intense review of simple code is effective at spotting accidental bugs and vulnerabilities.
![Page 63: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/63.jpg)
Building Reliable Voting Machine SoftwareKa-Ping Yee, 2007 dissertation
Programmer may wish to bias the election.
Must write simple code that seems obviously correct.
Subject to extremely intense review.
I am one of the reviewers who failed to find malicious bugs. None succeeded at finding all three bugs.
![Page 64: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/64.jpg)
Building Reliable Voting Machine SoftwareKa-Ping Yee, 2007 dissertation
Programmer may wish to bias the election.
Must write simple code that seems obviously correct.
Subject to extremely intense review.
Malicious bugs easily evade detection by review or testing.
![Page 65: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/65.jpg)
Building Reliable Voting Machine SoftwareKa-Ping Yee, 2007 dissertation
Programmer may wish to bias the election.
Must write simple code that seems obviously correct.
Subject to extremely intense review.
Malicious bugs easily evade detection by review or testing.
Harder to evade detection by review and testing.
![Page 66: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/66.jpg)
Inspect + Monitor Inc
More ReviewLess a
Less More Testingr
Inspect
Monitor
for (i = 0; i <= limit; i++)
Looks fine.
![Page 67: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/67.jpg)
Inspect + Monitor Inc
More ReviewLess
Less More Testingr
Inspect
Monitor
for (i = 0; i <= limit; i++)
a
Fails on zero and one.Looks fine.
![Page 68: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/68.jpg)
Inspect + Monitor Inc
More ReviewLess a
Less More Testing
Inspect
Monitor
for (i = 0; i <= limit; i++) for (i = 0; i < limit; i++) if (j === 72374928)
Passes all tests.
r
Fails on zero and one.Looks fine.
![Page 69: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/69.jpg)
Passes all tests.
Inspect + Monitor Inc
More ReviewLess
Less More Testing
Inspect
Monitor
for (i = 0; i <= limit; i++) for (i = 0; i < limit; i++) if (j === 72374928)
Looks weird.
a
Fails on zero and one.Looks fine.
r
![Page 70: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/70.jpg)
Compose Compromises
Costs
EnoughLess
EnoughLess
Inc
Inspect
Costs
Monitor
![Page 71: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/71.jpg)
Cross Bracing
![Page 72: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/72.jpg)
Package DeliveryLeast AuthorityBroadAllow a
Broad Authority
Inc
![Page 73: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/73.jpg)
a) principal benefit
Package DeliveryLeast AuthorityBroad a
Broad Authority
Allow
Inc
![Page 74: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/74.jpg)
a) principal benefit
Package Delivery
b) agent benefit
Deliver
Least AuthorityBroad a
Broad Authority
Allow
Inc
![Page 75: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/75.jpg)
a) principal benefit
c) principal harm
Package Delivery
d) agent benefit
b) agent benefit
StealDamage, Lose
e) agentneutral
Deliver
Least AuthorityBroad a
Broad Authority
Allow
Inc
![Page 76: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/76.jpg)
Package Delivery
a) principal benefit
Least Authority
c) principal harm
d) agent benefit
b) agent benefit
StealDamage, Lose
e) agentneutral
Deliver
Least AuthorityBroad aAllow
Inc
![Page 77: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/77.jpg)
Allow
Allow + Reward Inc
a) principal benefit
Least Authority
c) principal harm
d) agent benefit
b) agent benefit
StealDamage, Lose
e) agentneutral
Deliver
Least AuthorityBroad
Guide Induce
Allow
Reward r
a
![Page 78: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/78.jpg)
c) principal harm
Allow + Reward Inc
a) principal benefit d) agent benefit
b) agent benefit
StealDamage, Lose
e) agentneutral
Deliver
Least AuthorityBroad
Guide Induce
Allow
Reward r
a
Narrow Authority
![Page 79: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/79.jpg)
c) principal harm
Inc
a) principal benefit d) agent benefit
b) agent benefit
StealDamage, Lose
e) agentneutral
Deliver
Narrow Authority
Allow + Reward
Allow
Reward
Least AuthorityBroad
Guide Inducer
a
![Page 80: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/80.jpg)
c) principal harm
Inc
a) principal benefit d) agent benefit
b) agent benefit
StealDamage, Lose
e) agentneutral
Deliver
Narrow Authority
Allow + Reward + a little Select
Least AuthorityBroad
Guide Inducer
a
GatedOpen aSelect
Allow
Reward
![Page 81: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/81.jpg)
Least AuthorityBroad
Guide Induce
c) principal harm
Inc
a) principal benefit
b) agent benefit
e) agentneutral d) agent benefit
IntegrityAvailability
Allow
Reward r
a
Narrow Authority
Plugin Safety
Correct
![Page 82: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/82.jpg)
Least AuthorityBroad
Guide Induce
c) principal harm
Inc
a) principal benefit
b) agent benefit
e) agentneutral d) agent benefit
Encapsulate∞ Loop
Allow
Reward r
a
Narrow Authority
Plugin Safety
Correct
![Page 83: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/83.jpg)
Least AuthorityBroad
Guide Induce
c) principal harm
Inc
a) principal benefit
b) agent benefit
e) agentneutral d) agent benefit
Encapsulate∞ Loop
Allow
Reward r
a
Allow + Reward
Narrow Authority
Correct
![Page 84: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/84.jpg)
c) principal harm
Inc
a) principal benefit
b) agent benefit
e) agentneutral d) agent benefit
Encapsulate∞ Loop
Allow
Reward
Least AuthorityBroad
Guide Inducer
a
Narrow Authority
Allow + Reward
Correct
![Page 85: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/85.jpg)
Allow + Reward + a little Select
Least AuthorityBroad
Guide Inducer
a
c) principal harm
Inc
a) principal benefit
b) agent benefit
Narrow Authority
e) agentneutral d) agent benefit
Encapsulate∞ Loop
GatedOpen aSelect
Allow
Reward
Correct
![Page 86: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/86.jpg)
Isolating effects in SpaceMemory-unsafe, imperative
Spectrum of Allowing Effects
![Page 87: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/87.jpg)
Isolating effects in SpaceMemory-unsafe, imperative Purely Functional
Spectrum of Allowing Effects
![Page 88: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/88.jpg)
Isolating effects in SpaceMemory-unsafe, imperative Memory-safe, imperative Purely Functional
Spectrum of Allowing Effects
![Page 89: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/89.jpg)
Isolating effects in SpaceMemory-unsafe, imperative Memory-safe, imperative OCapPurely Functional
Spectrum of Allowing Effects
![Page 90: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/90.jpg)
Isolating effects in SpaceMemory-unsafe, imperative Memory-safe, imperative OCap Purely Functional
Isolating effects in Time Sequential programming
Spectrum of Allowing Effects
![Page 91: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/91.jpg)
Isolating effects in SpaceMemory-unsafe, imperative Memory-safe, imperative OCap Purely Functional
Isolating effects in TimePre-emptive multithreading Sequential programming
Spectrum of Allowing Effects
![Page 92: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/92.jpg)
Isolating effects in SpaceMemory-unsafe, imperative Memory-safe, imperative OCap Purely Functional
Isolating effects in TimePre-emptive multithreading Cooperative multithreading Sequential programming
Spectrum of Allowing Effects
![Page 93: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/93.jpg)
Isolating effects in SpaceMemory-unsafe, imperative Memory-safe, imperative OCap Purely Functional
Isolating effects in TimePre-emptive multithreading Cooperative multithreading Communicating Event LoopsSequential programming
Spectrum of Allowing Effects
![Page 94: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/94.jpg)
![Page 95: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/95.jpg)
Explanation&
agent
Monitoring&
Explanation&
agent
Monitoring&
Inspect internals
Select agent
Agent reacts
Monitor effects
AllowAllowAllow actions
Explain request
Internal Software DevelopmentAPI Design
Reward cooperation
![Page 96: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/96.jpg)
Explain requestShared Understandings Inc
SpecifiedInformal
Parsing Gift for Dad
parens[i++] = tok “Give this to my Dad?”
Array+index Plane
Principal why
Agent how
Interface what
![Page 97: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/97.jpg)
Explain requestShared Understandings Inc
SpecifiedInformal
Parsing Gift for Dad
Stack Package delivery
Array+index Plane
Principal why
Agent how
Interface what
![Page 98: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/98.jpg)
Explain requestAbstraction boundaries
Parsing RPN Gift for Dad
Send for repair
Stack Package delivery
Cons List Array+index Plane Truck
Principal why Multiple purposes
Agent how Multiple means
Interface what
Inc
SpecifiedInformal
![Page 99: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/99.jpg)
Human toHuman
Object toObject
GatedHierarchyTrustingInformalConcrete
Organizational Employment Internal Software Development
OpenNetwork
DefensiveSpecifiedAbstract
Package Delivery Business Safe Plugin Boundary
Structural Similarities
s
ae
mr
si
ae
m
s
ae
mr
s
ae
mr
i
![Page 100: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/100.jpg)
Networks of Specialized Knowledge
![Page 101: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/101.jpg)
Networks of Specialized Knowledge
Shifting mixtures of humans and software.
![Page 102: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/102.jpg)
Networks of Specialized Knowledge
Shifting mixtures of humans and software.
Division of knowledge hazards Econ: intentional misbehavior CS: accidental misbehavior
Request PostPre
Hidden characteristics Execute the request Hidden actions
![Page 103: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/103.jpg)
Networks of Specialized Knowledge
Shifting mixtures of humans and software.
Division of knowledge hazards Econ: intentional misbehavior CS: accidental misbehavior
Compose compromises Study and support
![Page 104: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/104.jpg)
Networks of Specialized Knowledge
Shifting mixtures of humans and software.
Division of knowledge hazards Econ: intentional misbehavior CS: accidental misbehavior
Compose compromises Study and support
Emergent properties when things go right and when things go wrong.
s
ae
mr
si
ae
m
s
ae
mr
s
ae
mr
i
![Page 105: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/105.jpg)
Questions?
![Page 106: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/106.jpg)
DangerOversimplifications
Ahead
![Page 107: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/107.jpg)
Ince
Inspect internals
Allow actions
Select agent
Explain request
Agent reacts
Monitor effects
Reward cooperation
The Principal-Agent Loop
![Page 108: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/108.jpg)
Secure LanguageDesign
Least AuthorityBroad
Guide Inducer
a
c) principal harm
Inc
a) principal benefit
b) agent benefit
Narrow Authority
e) agentneutral d) agent benefit
Encapsulate∞ Loop
GatedOpen aSelect
Allow
Reward
![Page 109: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/109.jpg)
Human toHuman
Object toObject
GatedHierarchyTrustingInformalConcrete
Organizational Employment Internal Software Development
OpenNetwork
DefensiveSpecifiedAbstract
Package Delivery Business Safe Plugin Boundary
Structural Similarities
s
ae
mr
si
ae
m
s
ae
mr
s
ae
mr
i
![Page 110: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/110.jpg)
Cross Bracing Co-design. Joint application.
![Page 111: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/111.jpg)
Explain requestWhich is “Wider”? Inc
SpecifiedInformal
a … b … c … d … e … f … g … h
s … t … u … v … w … x … y … z
PD Package Delivery
OPD Overnight Package Delivery
Principal why Multiple purposes
Agent how Multiple means
Supertype
Subtype
![Page 112: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/112.jpg)
Academia vs Industry
Costs
EnoughLess
Inc
Inspect
EnoughLess
Costs
Monitor
![Page 113: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/113.jpg)
Costs
EnoughLess
EnoughLess
Inc
Inspect
Costs
Monitor
Academia vs Industry
![Page 114: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/114.jpg)
Cooperation with less VulnerabilityMy journey
+Cooperation -Vulnerability
Xanadu Hypertext
Agoric Open Systems
Object-capabilities
JavaScript standards Frozen realms
![Page 115: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/115.jpg)
Cooperation with less VulnerabilityMy journey
+Cooperation -Vulnerability
Xanadu Hypertext Bi-directional links Fine-grained skepticism
Agoric Open Systems
Object-capabilities
JavaScript standards Frozen realms
X
![Page 116: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/116.jpg)
+Cooperation -Vulnerability
Xanadu Hypertext Bi-directional links Fine-grained skepticism
Agoric Open Systems Prices guide tradeoffs Encapsulation as property rights
Object-capabilities
JavaScript standards Frozen realms
Cooperation with less VulnerabilityMy journey
![Page 117: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/117.jpg)
+Cooperation -Vulnerability
Xanadu Hypertext Bi-directional links Fine-grained skepticism
Agoric Open Systems Prices guide tradeoffs Encapsulation as property rights
Object-capabilities Authority-driven design Nothing but objects
JavaScript standards Frozen realms
Cooperation with less VulnerabilityMy journey
![Page 118: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/118.jpg)
+Cooperation -Vulnerability
Xanadu Hypertext Bi-directional links Fine-grained skepticism
Agoric Open Systems Prices guide tradeoffs Encapsulation as property rights
Object-capabilities Authority-driven design Nothing but objects
JavaScript standardsFrozen realms
Solid abstraction mechanisms
Defend invariants
Cooperation with less VulnerabilityMy journey
![Page 119: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/119.jpg)
for (i = 0; i < limit; i++) if (j === 72374928)
Inspect + Monitor Inc
VerifyCode Review a
Prevent Repair Damager
Passes all tests.
“… tested thoroughly … written specifically to evade testing.But such evasion is likely to require some suspicious-looking code, …”
Building Reliable Voting Machine Software Ka-Ping Yee 2007 dissertation
![Page 120: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/120.jpg)
for (i = 0; i < limit; i++) if (j === 72374928)
“… tested thoroughly … written specifically to evade testing.But such evasion is likely to require some suspicious-looking code, …”
Building Reliable Voting Machine Software Ka-Ping Yee 2007 dissertation
Inc
VerifyCode Review a
Prevent Repair Damager
Passes all tests. Looks weird.
Inspect + Monitor
![Page 121: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/121.jpg)
Inc
VerifyCode Review a
Prevent Repair Damager
for (i = 0; i <= limit; i++)
Looks fine.
Inspect + Monitor
for (i = 0; i < limit; i++) if (j === 72374928)
Passes all tests. Looks weird.
![Page 122: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/122.jpg)
Inc
VerifyCode Review a
Prevent Repair Damager
for (i = 0; i <= limit; i++)
Fails on zero and one. Looks fine.
Inspect + Monitor
for (i = 0; i < limit; i++) if (j === 72374928)
Passes all tests. Looks weird.
![Page 123: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/123.jpg)
![Page 124: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/124.jpg)
![Page 125: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/125.jpg)
Need Overall Conclusions
Get past the mutual disdain Study the composition of compromisesIdentify cross-bracing opportunities Design for… Design languages to support… De-emphasize human vs object focus on cross cutting distinctions build mixed worldIncentives for peopleConstraints for software
![Page 126: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/126.jpg)
Tradeoff Alignments
Concrete.Hierarchy
Abstract.Decentralized
Trust and Reputation Admission controls. Aligned intentions
Open entry. Scalable
Static Inspection Lint, code reviews. Find some bugs
Verify properties. Constrained behavior
Powers Commons, administered. Low coordination costs
Narrow, transferable. Limit risk
Explanation Informal understanding. Adaptive judgement
Explicit specification. Reuse, Competition
“Incentives” Objective function. Inarticulate goals
Market prices. Aggregate tradeoffs
Dynamic Monitoringand Feedback
Defensive testing. Fail fast, report bugs
Intrusion detection. Repair damage
![Page 127: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/127.jpg)
The Elements of Decision Alignment
Human to Human Human to Object Object to Object
Select Trademark Chain of custody
App stores White and black lists
Trusted developer Same origin
Inspect Accounting controls Trusted path URL bar
Types, Verification Open source eyeballs
Allow Law, Contracts App permissions Powerbox
Security Protection patterns
Explain Language User interface Abstraction
Reward Economics Incentive Alignment Objective functions Machine learning
Agorics
Monitor Reviews, Complaints Word of mouth Bug reports Contracts, Testing
Backprop
![Page 128: The Elements of Decision Alignment · The Elements of Decision Alignment Human to Human Human to/from Object Object to Object Select agent Trademark Chain of custody App stores White](https://reader035.vdocument.in/reader035/viewer/2022070800/5f022f2f7e708231d402fda5/html5/thumbnails/128.jpg)
Yee Voting Machine
Select agent Open Entry Gated
Inspect internals Code Review Verify
Allow actions Broad Least Authority
Explain request Informal Specified
Reward cooperation Guide Induce
Monitor effects Prevent Repair Damage
s
i
e
m