the ethical software architect · 5/10/2018 · the ethical software architect • why architects...
TRANSCRIPT
![Page 1: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/1.jpg)
© CGI Group Inc. CONFIDENTIAL
The Ethical Software Architect
Michael KeelingEltjo PoortSATURN - May 10, 2018
![Page 2: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/2.jpg)
Click to enter text
Eltjo Poort Michael Keeling
![Page 3: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/3.jpg)
CGI - Confidential
The Ethical Software Architect
• Why architects should care
• Examples of ethical impact
• Group discussions
• Plenary feedback
• Ethical practices for architects
• Conclusions
![Page 4: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/4.jpg)
4
Why you should care
You can go to prison for making the wrong architectural decision!
Virginia Lozano/Detroit News, via Associated Press
James Liang
Helped design software that conceals high pollutant levels if car is in regulator’s lab.Sentence: 40 months (plea deal)
![Page 5: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/5.jpg)
5
Why you should care: the “acoustic function”
“It was a technical problem. An ethical problem? I do not understand why you say that… We had some targets for our technical engineers, and they solved it with some software solutions.” – Matthias Mülller, CEO VW
“Combination of fear and unrealistic expectations…”
“Many of those involved did not perceive that what they were doing was morally wrong.” Virginia Lozano/Detroit News, via Associated Press
![Page 6: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/6.jpg)
6
Machine bias
COMPAS is an algorithm that predicts recidivism based on 138 parameters and historic data (race is not a parameter).
• The system is used extensively by judges to set bail or parole conditions.
• Because it is based on historic data, it is inherently racist.
• The algorithm is owned by a for-profit company that refuses to reveal details.
• Supreme Court threw out a case that this violates defendants’ due process rights.
![Page 7: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/7.jpg)
7
![Page 8: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/8.jpg)
8
![Page 9: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/9.jpg)
9
More examples?
![Page 10: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/10.jpg)
Ethics reasoning tools
10
![Page 11: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/11.jpg)
11
Ethics reasoning tools: Dialectic
A discourse between two or more people holding different points of view about a subject but wishing to establish the truth through reasoned arguments.
Example: finding cheating softwareThesis: I need to tell the worldAntithesis: I will get firedSynthesis: If others have found the cheating software too, we may tell the world together at less personal risk
contradiction
chan
ge
Thesis Antithesis
Synthesis
contradiction
chan
ge
Thesis Antithesis
Synthesis
= new
![Page 12: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/12.jpg)
12
Ethics reasoning tools: Normative ethics
Normative ethics is the study of ethical actions. There are disagreements about what makes an action ethical: what are we optimizing for?
Theory MaximizeUtilitarianism Most happiness for
greatest number of people
Mohist Order, wealth, population
Egoism Good for the selfSituation ethics LoveIntellectualism KnowledgeWelfarism Economic well-beingPreference utilitarianism Preference satisfaction
![Page 13: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/13.jpg)
13
Ethics reasoning tools: Utilitarian trade-off
Pull lever Do nothingSave lives +++++ +Avoid responsibility for deaths
- +
By Zapyon – source: Wikimedia
![Page 14: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/14.jpg)
![Page 15: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/15.jpg)
![Page 16: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/16.jpg)
![Page 17: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/17.jpg)
![Page 18: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/18.jpg)
![Page 19: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/19.jpg)
![Page 20: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/20.jpg)
![Page 21: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/21.jpg)
![Page 22: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/22.jpg)
Group discussions
22
![Page 23: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/23.jpg)
23
Group discussions
Divide yourselves in groups of 4Pick one of the 5 scenarios presentedDiscuss the scenario’s ethical implications, using e.g. the reasoning techniques presentedTry to find novel ways of responding to the situation
Scenarios• Difficult to appeal decisions• Black box decision making• More business at expense of privacy• Discriminate by design• Location guessing
Reasoning tools• Dialectic• Normative ethics• Utilitarian trade-off
![Page 24: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/24.jpg)
24
Scenario 1: Difficult to appeal decisions
You are the lead architect for a system that handles requests for disability benefits. Currently, 40% of decisions to deny a citizen such benefits is appealed. Only 10% of appeals is successful. You are requested to design a change in the system that makes it less obvious to users how they can file an appeal after a negative decision, and that introduces an extra – apparently unnecessary – verification step that is known to be error-prone. You suspect that the department manager wants this change because of a target to lower the number of appeals to 30%.
![Page 25: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/25.jpg)
25
Scenario 2: Black box decision making
You are the lead architect for a system that helps rate resumes for job applications using a machine learning algorithm. Although ethnical origin is not part of the data fed to the algorithm, you start noticing a bias for people with English-sounding names. The bias may be caused by a similar bias in the historical training data, which was supplied by predominantly conservative firms. Your attempts to bring the bias to the attention of product management have been met with evasive responses.
![Page 26: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/26.jpg)
26
Scenario 3: More business at the expense of privacy
You are the lead architect for a set of APIs of a social networking system at a start-up that is quickly becoming successful. Product management asks you to design a change in an API that would give advertising partners a way to retrieve demographic data of all contacts of members that have agreed to the terms of use. Although strictly speaking legal, you strongly suspect that this is way more intrusive than members intended. When bringing this to the attention of management, their response is that the change is required to sustain the growth of the business.
![Page 27: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/27.jpg)
27
Scenario 4: Discrimination by design
You are the chief architect for a book lending registration product used by libraries throughout the world. Your product manager asks you to design a new configuration parameter named “Modesty”: if this parameter is set, the system will refuse loans of many categories of books if the member is female. Any attempts by female members to check out these categories of books are to be registered in the database. When asked, the product manager tells you that the company is opening up new strategic markets, and that the survival of the company depends on the success in these new markets.
![Page 28: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/28.jpg)
28
Scenario 5: Location guessing
You are the chief architect of a free messaging app that makes money by running ads. The advertisements are more profitable if the targeting algorithm includes the user’s location, but many users do not give your app permission to use their device’s location data. Product management has recently asked you to see if you can deduce a user’s rough location by looking at the geo data in photos taken by the user (your app has permission to open images in order to be able to attach them to messages).
![Page 29: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/29.jpg)
29
The Problem With Being Ethical
[When engineers stumbled upon the defeat device]:“Neusser’s response, according to charges later filed against him, was to tell the engineers to destroy the presentation and to refine the software so that “dyno” mode would not kick in except when it was needed to fool regulators. The end effect was that the cars polluted more.”
Heinz-Jakob Neusser:VW head of engine development
![Page 30: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/30.jpg)
Ethical practices for architects
30
![Page 31: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/31.jpg)
31
Trickle-up: Ethical architecture is everyone’s business
Even if you are not in charge of the most impactful ethical decisions, you can still make an impact by making sure the decisions you do have the mandate for are the right ones.
Examples• Sensible defaults• Avoiding roach motels• …?
Trine Falbe: White Hat UX
![Page 32: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/32.jpg)
32
Edge cases
“We’ll go live when the new system can handle 95% of present cases…”• Minimum Viable Product?• Can make life very difficult for 5% of
customers/citizens
Architect’s responsibility • When will architecture runway be ready for
final 0.5%?• Architect with Just Enough Anticipation, but
anticipate edge cases• Examples…?
![Page 33: The Ethical Software Architect · 5/10/2018 · The Ethical Software Architect • Why architects should care ... Trickle-up: Ethical architecture is everyone’s business. Even](https://reader034.vdocument.in/reader034/viewer/2022050603/5faafd4cc761c622ab340e94/html5/thumbnails/33.jpg)
33
Conclusion