algorithmic fairness: measures, methods and …suresh/static/files/tutorial.pdfexample for hiring)...

ALGORITHMIC FAIRNESS: MEASURES, METHODS AND REPRESENTATIONS

SURESH VENKATASUBRAMANIAN UNIVERSITY OF UTAH

PODS 2019

RIPPED FROM THE HEADLINES…

FAT* RESEARCH AREAS.

Computer Science

Machine Learning

Algorithms

Databases

SociologyThe LawEconomicsPolitical SciencePhilosophyMedia Studies

GOALS FOR THIS TUTORIAL

• An overview of the state of play in (some) areas of research in fairness

• Some open questions coming out of these areas

• New directions and challenges: centering the affected and introducing context.

• Overarching concern: thinking about the larger context is crucial if we want to formalize and interpret fairness without making huge mistakes

DISCLAIMER

This is my idiosyncratic view of the

MEASURES

DEFINING (UN)FAIRNESS

(a1, …, ak)

p(a1,…,ak)(x)

Think of a binary classification task

DEFINING (UN)FAIRNESS

Fairness can be expressed as a function of the classifier , the protected attribute , the training labels

Given where and the goal is to find such that

is minimized.

(x1, y1), (x2, y2), …, (xn, yn) ∈ X × Y x = (z, p)f ∈ ℱ

∑ ℓ( f(xi), yi)

protected

unprotected

MEASURES OF FAIRNESS

Individual fairness:

Demographic parity (and friends):

Equalized error rates:

(group sensitive) Calibration:

D( f(x), f(x′�)) ≤ d(x, x′�)

Pr[ f(x) = 1 |p = 1] ≈ Pr[ f(x) = 1 |p = 0]

Pr[ f(x) ≠ y |p = 1] ≈ Pr[ f(x) ≠ y |p = 0]

∀g, r, Pr[y = 1 ∣ f(x) = r, p = g] = r

MEASURES OF FAIRNESS

Individual fairness:

Demographic parity (and friends):

Equalized error rates:

(group sensitive) Calibration:

D( f(x), f(x′�)) ≤ d(x, x′�)

Pr[ f(x) = 1 |p = 1] ≈ Pr[ f(x) = 1 |p = 0]

Pr[ f(x) ≠ y |p = 1] ≈ Pr[ f(x) ≠ y |p = 0]

∀g, r, Pr[y = 1 ∣ f(x) = r, p = g] = r

Fairness can be expressed as a function of the classifier , the protected attribute , the training labels

individual fairnessdemographic parity

equalized error rates

CONDITIONAL PARITY [RSZ17]

ℒ(x ∣ a = a, z = z) = ℒ(x ∣ a = a′�, z = z)∀a, a′�

Examples: • = predicted probability of positive outcome, = group, =

Demographic parity [RP07] • = decision, = group, = true outcome

equalized odds [HPS16] • x = predicted probability of positive outcome, a = group, z = actual

probability of outcome Group-sensitive calibration.

x a z ∅

NORMATIVE DIMENSIONS TO FAIRNESS

Normative: establishing, standardizing or pertaining to a norm.

All measures of fairness carry normative positioning within them

• Individual fairness: there exists an objective measure of ability and fairness is making sure people of similar ability are treated similarly

• Demographic parity: Group identity should have nothing to do with selection for a task.

• Equalized odds: Groups may have different innate skill levels, but we should make mistakes equally.

MORE HIDDEN ASSUMPTIONS

• All groups are equivalent and unfair treatment of one is the same as unfair treatment of another.

• All instances of unfairness boil down to individual decisions about people, and not structural factors that create the data used for learning bias decisions [FBG19]

• All instances of unfairness come from the process of making decisions. [OKBTG18]

Need a way to create “context knobs” for the design of fairness measures.

EXTENSIONS: REGRESSION

Given , find a mapping such that

is minimized.

• What is an appropriate form of fairness?

• What is the context for this? (credit score assignment?)

• What would the right normative concerns be? (POTS: [OKBTG18])

(xi, yi), xi ∈ ℝd, yi ∈ ℝ f : ℝd → ℝ

∥f(xi) − yi∥2

EXTENSIONS: RANKING

• Build a ranking scheme to rank individuals from top to bottom (for example for hiring)

• What are forms of fairness in rankings [YS17,ZBCHMB-Y17,CSV17,AJSD19,SJ]

• “demographic parity”: proportion of people from different groups in each top-k section should be roughly the same

• “quotas”: each group should have an upper/lower bound on number of members in top-k for different k

• generalize rank to “exposure” and equalize it [AS17,19,BGW18]

EXTENSIONS: CLUSTERING

• Given points labeled red and blue, find a good clustering so that each cluster has the same proportion of groups as in the overall population. [CKLV18]

• Good idea: each cluster should be representative

• Bad idea: if we consider this an example of redistricting, then the minority party loses all the seats!

METHODS

FAIRNESS IN CLASSIFICATION

Modify the training data prior to the

training process

Add fairness constraints

when learning the model

Modify the labels after

training

MODIFYING THE TRAINING DATA

• PRO: Might be able to account for data bias, deal with black box classifier

• CON: Might need to modify data extensively to remove skew - this is not well defined.

• CON: No transparency for what concerns are being addressed by skew elimination. Other biases might carry through.

TRAINING WITH A REGULARIZER

• PRO: Flexibility to work with any data. More control over learned models

• CON: All regularizers are proxies - unintended consequences.

• CON: Even harder to explain outcomes.

POST-PROCESSING THE PREDICTIONS

• PROS: Works with black box classifier and any training data. Has certain optimality properties.

• CON: Might very well be illegal (in the US, in certain sectors).

• CON: What is principled argument for post-processing labels?

MODIFYING THE TRAINING DATA

Given X, construct such that

• (we cannot predict p)

• A predictor learned on is similar to a predictor learned on .

Notes:

• Should try to change X’ minimally.

• Other biases might remain in X’.

X′� = g(X)

X′� ⊥ p(X)

f′� X′� fX

TRAINING WITH A REGULARIZER

Translate fairness conditions into constraints encoded into the optimization.

Demographic parity [ZVRG17a, ZVRG17b]:

In general constraints might not be convex, and so proxies are needed.

|∑p(x)=1 h(x)

∑p(x)=0 h(x)

N0| ≤ ϵ

POST-PROCESSING LABELS

• Run the training process without any intervention and construct a derived predictor based on the (joint distribution of) learned model, group attributes and ground truth outcome.

• At prediction time only use information from learned model and group outcome. [HPS16]

x = (z, p)

DESIGN DIMENSIONS

FEEDBACK FROM MODELUpdate mode

(a1, …, ak)

p(a1,…,ak)(x, t)

Batch learning with feedback

Model output contaminates training data Training data no longer drawn from “true” distribution.

[EFNSV18a,EFNSV18b]

DEALING WITH FEEDBACK

• What the system learns depends mostly on initial conditions, not the actual data.

• Small differences in input probabilities lead to huge differences in output predictions.

To Predict And Serve [LI16]

[EFNSV18a,EFNSV18b]

DEALING WITH FEEDBACK

• What the system learns depends mostly on initial conditions, not the actual data.

• Small differences in input probabilities lead to huge differences in output predictions.

To Predict And Serve [LI16]

Need to use reinforcement

learning rather than supervised

learning, but no one does that

FEEDBACK FROM BEHAVIOR

• Police presence might change the behavior of residents

• this is a good thing! BUT will render model inaccurate

• If a model is trained on one distribution, it will in general not work if underlying distribution changes

• Monitoring for changes in underlying distribution is hard.

MORE FEEDBACK PROBLEMS

• Strategic classification: What happens if players try to “game” the model?

• BAD: I realize that getting more credit cards will increase my credit score, so I go out and get some.

• GOOD: I realize that signing up for a regular health checkup will reduce my insurance costs, so I do it.

What's the difference?

PIPELINES

• Decisions are made in multiple stages.

• We want fairness guarantees to compose.

• BAD: In most settings, fairness guarantees do NOT compose

• INTERESTING: Under certain modeling assumptions, it’s better to intervene earlier in the pipeline rather than later.

Under what conditions can we compose fairness guarantees and how does this guide interventions?

College admission

JobGraduate school

PIPELINES

• Decisions are made in multiple stages.

• We want fairness guarantees to compose.

• BAD: In most settings, fairness guarantees do NOT compose

• INTERESTING: Under certain modeling assumptions, it’s better to intervene earlier in the pipeline rather than later.

Under what conditions can we compose fairness guarantees and how does this guide interventions?

College admission

JobGraduate school

STABILITY I: ALGORITHMS VARY

• Current approaches to achieving fairness have very different operating characteristics. [FSVCHR19]

STABILITY 2: WHAT CAN WE DO

• Stable and fair classification [HV19]

in a recent study, Friedler et al. observed that fair classification algorithms may not be stable with respect to variations in the training dataset -- a crucial consideration in several real-world applications. Motivated by their work, we

study the problem of designing classification algorithms that are both fair and stable. We propose an extended framework based on fair classification

algorithms that are formulated as optimization problems, by introducing a stability-focused regularization term.

Can we show that fairness guarantees generalize?

FAIRNESS AND PRIVACY

• VIEW: privacy helps with fairness because sensitive information about individuals will not be leaked

• VIEW: privacy hurts fairness because we can hide discrimination by not collecting sensitive information but inferring it.

Attempts to attack private data, or operate on private data, exhibit disparate effects on different groups.

• Disparate Vulnerability: on the unfairness of privacy attacks against machine learning. [YKT19]

• Fair Decision Making using privacy-protected data. [KMPHMM19]

• Differential Privacy has disparate impact on model privacy [BS19]

How do privacy and fairness really interact?

REPRESENTATIONS

NEW REPRESENTATIONS

• Core idea: a learned representation is “good” if it conceals information about protected attribute maximally, while affecting ability to classify minimally [ZWSPD13, MCPZ18].

x = (z, p)

x′� = (z′�, p)

Cannot predict from p z′� Can predict from y z′�

DISENTANGLED REPRESENTATIONS [H+18]

• Mask signal about protected attributes but without destroying realism of data.

• Formally,

(z, p) (w, p) ( z, p)Disentangled

representation

Input Reconstruction

and are independentw p

DISENTANGLED REPRESENTATIONS

• Disentangled representations can be used to preprocess training data (because is both realistic and independent of )

• Disentangled representations can be used to determine the influence of protected attributes on the classification [MPFSV19]

• But we still lack a principled explanation of bias in representations.

BIAS IN REPRESENTATIONS

• Can we identify bias in existing (learned) representations and correct it?

womandoctor

doctor nurse

THE GEOMETRY OF BIAS

• “bias” = distortion of representation along the "gender axis”

womandoctor nurse

• “mitigation” = reversal of this distortion.

womandoctor nurse

• “mitigation” = reversal of this distortion.

• Stay tuned for more….

womandoctor nurse

PRINCIPAL COMPONENT ANALYSIS

Given matrix M ∈ ℝm×n, find M ∈ ℝm×n with rank d such that

∥M − M∥F is minimized

PRINCIPAL COMPONENT ANALYSIS

Given matrix M ∈ ℝm×n, find M ∈ ℝm×n with rank d such that

∥M − M∥F is minimized

M = MWW⊤ where columns of WM⊤Mare the top eigenvectors of

[SAMADI ET AL, 2018]

FAIR PRINCIPAL COMPONENT ANALYSIS

L(X, Z ) = ∥X − Z∥F − ∥X − X∥F

max L(A, UA), L(B, UB) for rank-d U ∈ ℝm×n

L(X, Z ) = ∥X − Z∥F − ∥X − X∥F

max L(A, UA), L(B, UB) for rank-d U ∈ ℝm×n

Thm: Can obtain optimal U with rank at most d+1 (solve an SDP followed by LP)

WHAT NEXT?

CENTERING THE SOLUTION

CENTERING THE HARMED

CENTERING THE HARMS

…returning to the idea of unfairness suggests several new areas of inquiry, including quantifying different kinds of unfairness and bias….

Quantifying types of unfairness may not only add to the problems that machine learning can address, but also accords with realities of sentencing and policing behind much of the fairness research today: Individuals seeking justice do so when they believe that something has been unfair.

— 50 years of test (un)fairness: lessons for machine learning [HM19]

CENTERING THE HARMS

…returning to the idea of unfairness suggests several new areas of inquiry, including quantifying different kinds of unfairness and bias….

Quantifying types of unfairness may not only add to the problems that machine learning can address, but also accords with realities of sentencing and policing behind much of the fairness research today: Individuals seeking justice do so when they believe that something has been unfair.

— 50 years of test (un)fairness: lessons for machine learning [HM19]

We should think about

UNFAIRNESS rather than fairness

RECOURSE

Given positively and negatively labeled points, find a line separating them such that the margin is maximized.

RECOURSE FOR A (BAD) DECISION

Recourse: the ability of a person to change the decision of a model through actionable input variables [USL2019]

[DGNV19, ONGOING]

RECOURSE-EQUALIZED CLASSIFICATION

[DGNV19, ONGOING]

Given P = {(x1, y1), (x2, y2), …, (xn, yn)}, (xi, yi) ∈ ℝd × {+1, − 1}

min ∥w∥2

s.t. ∀i yi(w ⋅ xi + b) ≥ 1

min ∥w∥2

s.t. ∀i yi(w ⋅ xi + b) ≥ 1

min ∥w∥2

s.t. ∀i yi(w ⋅ xi + b) ≥ 1|rw,b(A) − rw,b(B) | ≤ ϵ

min ∥w∥2

s.t. ∀i yi(w ⋅ xi + b) ≥ 1

P = A ∪ B

rw,b(S) = ∑(x,y)∈S

w ⋅ x + b∥w∥|S− |

S− = {(x, y) ∈ S ∣ y = − 1}

min ∥w∥2

s.t. ∀i yi(w ⋅ xi + b) ≥ 1

P = A ∪ B

rw,b(S) = ∑(x,y)∈S

w ⋅ x + b∥w∥|S− |

S− = {(x, y) ∈ S ∣ y = − 1}

min ∥w∥2

GIVEN A COLLECTION OF LABELED POINTS, CAN WE COMPUTE A RECOURSE-EQUALIZED

MAXIMUM MARGIN LINEAR CLASSIFIER?

FAIRNESS IN SOCIAL NETWORKS

• Social standing [Coleman] within a network confers utility on an individual.

• Social “position” in a network is a class marker defined by the network, not the individual.

• Should we be considered about discrimination based on social position? [boyd, Marwick and Levy]

INFORMATION ACCESS

• Social networks grow through recommendations as well as organically

• Network position confers advantage ([Granovetter])

• Access to information that improves network position relies on …. network position

• “edges in social network” == “biased input data”

INFLUENCE MAXIMIZATION

Given a graph, a mechanism for spreading information and k seeds, how many nodes can be be

influenced?

GAPS IN INFORMATION ACCESS [FBBFSV19]

• New measure of access gap in a network

• Axiomatic considerations and a proposed cost function.

• Study of how to intervene in a network (by adding edges) to improve access gaps • Theoretical (negative) results • Empirical study of heuristics.

HARMS OF REPRESENTATION

Crawford, The Trouble with Bias, NeurIPS 2017 Keynote

STEREOTYPING

“associations and beliefs about the characteristics and attributes of a group and its members that shape how people think about and

respond to the group”

— SAGE handbook of prejudice, stereotyping and discrimination.

A specific mechanism for stereotyping:

…the tendency to assign characteristics to all members of a group based on stereotypical features shared by a few…

A MODEL FOR STEREOTYPING [AFSV19]

Points regress towards an exemplar

pα = (1 − α)p + αc

CONCLUSIONS

TAKEAWAYS

• We are the “end of the beginning” in algorithmic fairness.

TAKEAWAYS

• Our focus should always be on the broader ways in which algorithmic systems influence society.

TAKEAWAYS

• Our focus should always be on the broader ways in which algorithmic systems influence society.

• We must be bold in reimagining how we can use algorithms and tech ... for good?

ACM FAT* 2020

CRAFT: Critiquing and Rethinking Accountability, Fairness and Transparency

A number of prominent studies acknowledge that addressing the greater societal problems due to the introduction of automation, machine learning algorithms and optimization systems may require more holistic approaches.

In the spirit of reflection and response, we are planning a call for contributions that invites academics and different communities of practice (including journalism, advocacy, organizing, education, art, public authorities) to propose workshops, panels, debates and other formats that will be co-located with ACM FAT* 2020. The details of this call will be announced shortly.

Seda Gürses, Seeta Peña Gangadharan, Suresh Venkatasubramanian

ACKNOWLEDGEMENTS

• Sorelle Friedler and Carlos Scheidegger (11 papers and counting!)

• Solon Barocas, Andrew Selbst, Karen Levy, danah boyd and Seda Gürses for perspectives on the world outside CS

• Mohsen Abbasi, Ashkan Bashardoust, Danielle Ensign, Scott Neville, Pegah Nokhiz, Chitradeep Dutta Roy - my students past and present.

• The entire FAT* community.

POST-TALK UPDATES

• References have been added.

• Note that this talk skipped a number of key topics in the area:

• fairness and causality

• methods for determining the influence of variables on outcomes (the broader area of audit mechanisms)

• the entire area of explainability/interpretability

REFERENCES

• [1] J. S. Coleman, “Social capital in the creation of human capital,” American journal of sociology, vol. 94, pp. S95–S120, 1988.

• [2] M. S. Granovetter, “The strength of weak ties,” in Social networks, Elsevier, 1977, pp. 347–367.

• [3] D. Boyd, K. Levy, and A. Marwick, “The networked nature of algorithmic discrimination,” Data and Discrimination: Collected Essays. Open Technology Institute, 2014.

• [4] D. Ensign, S. A. Friedler, S. Neville, C. Scheidegger, and S. Venkatasubramanian, “Runaway Feedback Loops in Predictive Policing,” in Conference on Fairness, Accountability and Transparency, 2018, pp. 160–171.

• [5] D. Ensign, F. Sorelle, N. Scott, S. Carlos, and V. Suresh, “Decision making with limited feedback,” in Algorithmic Learning Theory, 2018, pp. 359–367.

• [6] M. Abbasi, S. Friedler, C. Scheidegger, and S. Venkatasubramanian, “Fairness in representation: quantifying stereotyping as a representational harm,” in Proceedings of the 2019 SIAM International Conference on Data Mining, 0 vols., Society for Industrial and Applied Mathematics, 2019, pp. 801–809.

• [7] B. Fish, A. Bashardoust, D. Boyd, S. Friedler, C. Scheidegger, and S. Venkatasubramanian, “Gaps in Information Access in Social Networks?,” in The World Wide Web Conference, New York, NY, USA, 2019, pp. 480–490.

• [8] B. Hutchinson and M. Mitchell, “50 Years of Test (Un)Fairness: Lessons for Machine Learning,” in Proceedings of the Conference on Fairness, Accountability, and Transparency, New York, NY, USA, 2019, pp. 49–58.

REFERENCES

• [9] I. Higgins et al., “Towards a Definition of Disentangled Representations,” arXiv:1812.02230 [cs, stat] , Dec. 2018.

• [10] E. Bagdasaryan and V. Shmatikov, “Differential Privacy Has Disparate Impact on Model Accuracy,” arXiv:1905.12101 [cs, stat] , May 2019.

• [11] S. Kuppam, R. Mckenna, D. Pujol, M. Hay, A. Machanavajjhala, and G. Miklau, “Fair Decision Making using Privacy-Protected Data,” arXiv:1905.12744 [cs] , May 2019.

• [12] M. Yaghini, B. Kulynych, and C. Troncoso, “Disparate Vulnerability: on the Unfairness of Privacy Attacks Against Machine Learning,” arXiv:1906.00389 [cs, stat] , Jun. 2019.

• [13] L. Huang and N. Vishnoi, “Stable and Fair Classification,” in International Conference on Machine Learning, 2019, pp. 2879–2890.

• [14] A. Bower, S. N. Kitchen, L. Niss, M. J. Strauss, A. Vargas, and S. Venkatasubramanian, “Fair Pipelines,” arXiv:1707.00391 [cs, stat] , Jul. 2017.

• [15] C. Dwork and C. Ilvento, “Fairness Under Composition,” arXiv:1806.06122 [cs, stat] , Jun. 2018.

• [16] K. Lum and W. Isaac, “To predict and serve?,” Significance, vol. 13, no. 5, pp. 14–19, 2016.

REFERENCES

• [17] B. Woodworth, S. Gunasekar, M. I. Ohannessian, and N. Srebro, “Learning Non-Discriminatory Predictors,” arXiv:1702.06081 [cs] , Feb. 2017.

• [18] A. J. Biega, K. P. Gummadi, and G. Weikum, “Equity of Attention: Amortizing Individual Fairness in Rankings,” in The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, New York, NY, USA, 2018, pp. 405–414.

• [19] G. Farnadi, B. Babaki, and L. Getoor, “Fairness in Relational Domains,” in Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA, 2018, pp. 108–114.

• [20] S. Mitchell, E. Potash, S. Barocas, A. D’Amour, and K. Lum, “Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions,” arXiv:1811.07867 [stat] , Nov. 2018.

• [21] C. T. Marx, R. L. Phillips, S. A. Friedler, C. Scheidegger, and S. Venkatasubramanian, “Disentangling Influence: Using Disentangled Representations to Audit Model Predictions,” arXiv:1906.08652 [cs, stat] , Jun. 2019.

• [22] B. Ustun, A. Spangher, and Y. Liu, “Actionable Recourse in Linear Classification,” in Proceedings of the Conference on Fairness, Accountability, and Transparency, New York, NY, USA, 2019, pp. 10–19.

• [23] S. Samadi, U. Tantipongpipat, J. H. Morgenstern, M. Singh, and S. Vempala, “The Price of Fair PCA: One Extra dimension,” in Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. Curran Associates, Inc., 2018, pp. 10999–11010.

• [24] D. Madras, E. Creager, T. Pitassi, and R. Zemel, “Learning Adversarially Fair and Transferable Representations,” arXiv:1802.06309 [cs, stat] , Feb. 2018.

REFERENCES

• [25] R. Overdorf, B. Kulynych, E. Balsa, C. Troncoso, and S. Gürses, “POTs: Protective Optimization Technologies,” Jun. 2018.

• [26] R. Overdorf, B. Kulynych, E. Balsa, C. Troncoso, and S. Gürses, “Questioning the assumptions behind fairness solutions,” Nov. 2018.

• [27] K. Yang, J. Stoyanovich, A. Asudeh, B. Howe, H. V. Jagadish, and G. Miklau, “A Nutritional Label for Rankings,” Proceedings of the 2018 International Conference on Management of Data - SIGMOD ’18, pp. 1773–1776, 2018.

• [28] Y. Ritov, Y. Sun, and R. Zhao, “On conditional parity as a notion of non-discrimination in machine learning,” arXiv:1706.08519 [cs, stat] , Jun. 2017.

• [29] F. Chierichetti, R. Kumar, S. Lattanzi, and S. Vassilvitskii, “Fair Clustering Through Fairlets,” in Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017, pp. 5029–5037.

• [30] A. Singh and T. Joachims, “Fairness of Exposure in Rankings,” arXiv:1802.07281 [cs] , Feb. 2018.

• [31] S. A. Friedler, C. Scheidegger, S. Venkatasubramanian, S. Choudhary, E. P. Hamilton, and D. Roth, “A comparative study of fairness-enhancing interventions in machine learning,” arXiv:1802.04422 [cs, stat] , Feb. 2018.

• [32] M. Zehlike, F. Bonchi, C. Castillo, S. Hajian, M. Megahed, and R. Baeza-Yates, “FA*IR: A Fair Top-k Ranking Algorithm,” arXiv:1706.06368 [cs] , pp. 1569–1578, 2017.

REFERENCES

• [33] K. Yang and J. Stoyanovich, “Measuring Fairness in Ranked Outputs,” arXiv:1610.08559 [cs] , Oct. 2016.

• [34] L. E. Celis, D. Straszak, and N. K. Vishnoi, “Ranking with Fairness Constraints,” arXiv:1704.06840 [cs] , Apr. 2017.

• [35] A. Asudehy, H. V. Jagadishy, J. Stoyanovichz, and G. Das, “Designing Fair Ranking Schemes,” arXiv:1712.09752 [cs] , Dec. 2017.

• [36] Moritz Hardt, Eric Price, and Nati Srebo, “Equality of Opportunity in Supervised Learning,” in Advances in Neural Information Processing Systems 29 (NIPS 2016), 2016.

• [37] M. B. Zafar, I. Valera, M. G. Rodriguez, and K. P. Gummadi, “Learning Fair Classifiers,” arXiv:1507.05259 [cs, stat] , Jul. 2015.

• [38] M. B. Zafar, I. Valera, M. G. Rodriguez, and K. P. Gummadi, “Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment,” arXiv:1610.08452 [cs, stat] , Oct. 2016.

• [39] R. S. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork, “Learning Fair Representations.,” ICML (3), vol. 28, pp. 325–333, 2013.

• [40] M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkatasubramanian, “Certifying and removing disparate impact,” in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015, pp. 259–268.

algorithmic fairness: measures, methods and …suresh/static/files/tutorial.pdfexample for hiring)...

Documents

suresh pro

krishnan suresh (“suresh”) suresh@engr.wisc.edu...

fairness in the customer relationship - amazon web...

subashini suresh, suresh renukappa, ibrahim alghanmi...

9709 y17-18 sy - xtremepapers

can i trust my fairness metric? assessing fairness with

fairness activities list - santee school district...fairness...

the relationship between perceived personal fairness, social...

fairness: perceptions of fairness in negotiation

suresh r – curriculum...

s¼y17 ® 5Æ µ:¦ b $'¨1jËn% û 55m

suresh materials

suresh srinivasan

-y17-rc] ecr cas purce air inlet a/c air side washinc

xml for science data access r. suresh (nasa/mtech)...

suresh narayanan

fairness cream

suresh final

fairness commission: headlines · 02 fairness commission...

y17-powder actuated tool -2