counterfactual reasoning in algorithmic fairnessfairware.cs.umass.edu/slides/silva.pdf ·...
TRANSCRIPT
CounterfactualReasoninginAlgorithmicFairness
RicardoSilvaUniversityCollegeLondonand
TheAlanTuringInstitute
JointworkwithMattKusner(Warwick/Turing),ChrisRussell(Sussex/Turing),andJoshuaLoftus(NYU)
FairnessandMachineLearning
• Thedream:ifweteachmachinestoperformsensitivedecisions,theywillnotsufferfromhumanbiases.
• Thereality:theGIGOprinciplestillholds,regardlessofwhetherwearetalkingofstatisticalmodelsorsoftware.
TheMessage
• Thereisonlysomuchdataalonecantellyouaboutfairness.
• I’mnottalkingabout“just”valuejudgments.
• Weshouldhighlighttherolethatthedata-generatingcausalprocesshasinshapingournotionsoffairness.
NobodyisSayingThisisEasy
• AtnopointIwillsuggestthatbuildingacausalmodeliseasy.
• Someuntestedanduntestableassumptionswillbeneeded.
• Theideaistomakeyourassumptionsasexplicitaspossible,hopefullybeing“lesswrong”intheend.
TheScopeofthisTalk
• Weconsiderpredictionandinterventionproblems(moreoftheformer).
• Inpredictionproblems,wewillhave:
X : features, or attributes of an individual
A : the protected attributes of an individual
Y : the target, what we would like to predict
Y : our prediction<latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="h0vKyMv1txiryC/6SUOT3b8sZzo=">AAACzXicdZFNb9NAEIbX5quEAoErlxUViAOKbC4gTlRcOBaJtIE4isbrcbzKfli746aRZYkDCPH3uPEr+Aus0wg1bZnTq5nZd2eeyWslPSXJ7yi+cfPW7Tt7dwf39u8/eDh8tH/sbeMEjoVV1k1y8KikwTFJUjipHYLOFZ7ky/d9/eQUnZfWfKJ1jTMNCyNLKYBCaj78M3nLn/OM8IycbnmGuq7aEoEah757ya3jQORk3hB6bksOhktTyFNZNKC6LBsc7hhQhVuT2llCQVhcMOiudfj8HwcCt0AKQ6wqIL5CvrKNKriSS+RkedizkIJ6h6xv2LUJgP4NsukL63bdYD48SEbJJvhVkW7FAdvG0Xz4KyusaDQaEgq8n6ZJTbMWHEmhsBtkjccaxBIWOA3SgEY/azeH6fizkCl4GSCW1hDfZC++aEF7v9Z56NRAlb9c65PX1aYNlW9mrTR1oGrE+Udlo3os/ZV5IV1gr9ZBgHAyzMpFBQ7CPZzvIaSXV74qjl+N0mSUfkzYHnvCnrIXLGWv2Tv2gR2xMRPROGqjb9H3+Ev8Nf5xjiuOttwes52If/4FarPjeg==</latexit><latexit sha1_base64="h0vKyMv1txiryC/6SUOT3b8sZzo=">AAACzXicdZFNb9NAEIbX5quEAoErlxUViAOKbC4gTlRcOBaJtIE4isbrcbzKfli746aRZYkDCPH3uPEr+Aus0wg1bZnTq5nZd2eeyWslPSXJ7yi+cfPW7Tt7dwf39u8/eDh8tH/sbeMEjoVV1k1y8KikwTFJUjipHYLOFZ7ky/d9/eQUnZfWfKJ1jTMNCyNLKYBCaj78M3nLn/OM8IycbnmGuq7aEoEah757ya3jQORk3hB6bksOhktTyFNZNKC6LBsc7hhQhVuT2llCQVhcMOiudfj8HwcCt0AKQ6wqIL5CvrKNKriSS+RkedizkIJ6h6xv2LUJgP4NsukL63bdYD48SEbJJvhVkW7FAdvG0Xz4KyusaDQaEgq8n6ZJTbMWHEmhsBtkjccaxBIWOA3SgEY/azeH6fizkCl4GSCW1hDfZC++aEF7v9Z56NRAlb9c65PX1aYNlW9mrTR1oGrE+Udlo3os/ZV5IV1gr9ZBgHAyzMpFBQ7CPZzvIaSXV74qjl+N0mSUfkzYHnvCnrIXLGWv2Tv2gR2xMRPROGqjb9H3+Ev8Nf5xjiuOttwes52If/4FarPjeg==</latexit><latexit sha1_base64="jgC96ZbtUjoycXYPDL+iaX6PzEU=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhK9gLitMCF4yLR3UJTVRNn0lh17MiebKmiSBxAiCs/jRu/gr+A043QdneZ09Ob5+eZN2mlpKMo+h2EV65eu35j7+bg1u07d+8N7z84cqa2AifCKGOnKThUUuOEJCmcVhahTBUep6u3Xf/4BK2TRn+gTYXzEpZa5lIAeWox/DN9xZ/yhPAz2bLhCZZV0eQIVFt07XNuLAciK9Oa0HGTc9Bc6kyeyKwG1SbJ4PWOARXYm1TWEArC7IxBe6nDx/84ENglkh9iXQDxNfK1qVXGlVwhJ8P9npkU1DkknWDXxgf0b5Ctzq/btoPFcBSNo23xiyDuwYj1dbgY/koyI+oSNQkFzs3iqKJ5A5akUNgOktphBWIFS5x5qKFEN2+2h2n5E89kPPch5kYT37JnXzRQOrcpU68sgQp3vteRl/VmNeUv543UlU9Vi9OP8lp1sXRX5pm0Pnu18QCElX5WLgqw4O9hXRdCfH7li+BofxxH4/h9NDp408exxx6xx+wZi9kLdsDesUM2YSKYBE3wNfgWfgq/hN/DH6fSMOjfPGQ7Ff78C0xY5Lo=</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="h0vKyMv1txiryC/6SUOT3b8sZzo=">AAACzXicdZFNb9NAEIbX5quEAoErlxUViAOKbC4gTlRcOBaJtIE4isbrcbzKfli746aRZYkDCPH3uPEr+Aus0wg1bZnTq5nZd2eeyWslPSXJ7yi+cfPW7Tt7dwf39u8/eDh8tH/sbeMEjoVV1k1y8KikwTFJUjipHYLOFZ7ky/d9/eQUnZfWfKJ1jTMNCyNLKYBCaj78M3nLn/OM8IycbnmGuq7aEoEah757ya3jQORk3hB6bksOhktTyFNZNKC6LBsc7hhQhVuT2llCQVhcMOiudfj8HwcCt0AKQ6wqIL5CvrKNKriSS+RkedizkIJ6h6xv2LUJgP4NsukL63bdYD48SEbJJvhVkW7FAdvG0Xz4KyusaDQaEgq8n6ZJTbMWHEmhsBtkjccaxBIWOA3SgEY/azeH6fizkCl4GSCW1hDfZC++aEF7v9Z56NRAlb9c65PX1aYNlW9mrTR1oGrE+Udlo3os/ZV5IV1gr9ZBgHAyzMpFBQ7CPZzvIaSXV74qjl+N0mSUfkzYHnvCnrIXLGWv2Tv2gR2xMRPROGqjb9H3+Ev8Nf5xjiuOttwes52If/4FarPjeg==</latexit><latexit sha1_base64="h0vKyMv1txiryC/6SUOT3b8sZzo=">AAACzXicdZFNb9NAEIbX5quEAoErlxUViAOKbC4gTlRcOBaJtIE4isbrcbzKfli746aRZYkDCPH3uPEr+Aus0wg1bZnTq5nZd2eeyWslPSXJ7yi+cfPW7Tt7dwf39u8/eDh8tH/sbeMEjoVV1k1y8KikwTFJUjipHYLOFZ7ky/d9/eQUnZfWfKJ1jTMNCyNLKYBCaj78M3nLn/OM8IycbnmGuq7aEoEah757ya3jQORk3hB6bksOhktTyFNZNKC6LBsc7hhQhVuT2llCQVhcMOiudfj8HwcCt0AKQ6wqIL5CvrKNKriSS+RkedizkIJ6h6xv2LUJgP4NsukL63bdYD48SEbJJvhVkW7FAdvG0Xz4KyusaDQaEgq8n6ZJTbMWHEmhsBtkjccaxBIWOA3SgEY/azeH6fizkCl4GSCW1hDfZC++aEF7v9Z56NRAlb9c65PX1aYNlW9mrTR1oGrE+Udlo3os/ZV5IV1gr9ZBgHAyzMpFBQ7CPZzvIaSXV74qjl+N0mSUfkzYHnvCnrIXLGWv2Tv2gR2xMRPROGqjb9H3+Ev8Nf5xjiuOttwes52If/4FarPjeg==</latexit><latexit sha1_base64="jgC96ZbtUjoycXYPDL+iaX6PzEU=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhK9gLitMCF4yLR3UJTVRNn0lh17MiebKmiSBxAiCs/jRu/gr+A043QdneZ09Ob5+eZN2mlpKMo+h2EV65eu35j7+bg1u07d+8N7z84cqa2AifCKGOnKThUUuOEJCmcVhahTBUep6u3Xf/4BK2TRn+gTYXzEpZa5lIAeWox/DN9xZ/yhPAz2bLhCZZV0eQIVFt07XNuLAciK9Oa0HGTc9Bc6kyeyKwG1SbJ4PWOARXYm1TWEArC7IxBe6nDx/84ENglkh9iXQDxNfK1qVXGlVwhJ8P9npkU1DkknWDXxgf0b5Ctzq/btoPFcBSNo23xiyDuwYj1dbgY/koyI+oSNQkFzs3iqKJ5A5akUNgOktphBWIFS5x5qKFEN2+2h2n5E89kPPch5kYT37JnXzRQOrcpU68sgQp3vteRl/VmNeUv543UlU9Vi9OP8lp1sXRX5pm0Pnu18QCElX5WLgqw4O9hXRdCfH7li+BofxxH4/h9NDp408exxx6xx+wZi9kLdsDesUM2YSKYBE3wNfgWfgq/hN/DH6fSMOjfPGQ7Ff78C0xY5Lo=</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="h0vKyMv1txiryC/6SUOT3b8sZzo=">AAACzXicdZFNb9NAEIbX5quEAoErlxUViAOKbC4gTlRcOBaJtIE4isbrcbzKfli746aRZYkDCPH3uPEr+Aus0wg1bZnTq5nZd2eeyWslPSXJ7yi+cfPW7Tt7dwf39u8/eDh8tH/sbeMEjoVV1k1y8KikwTFJUjipHYLOFZ7ky/d9/eQUnZfWfKJ1jTMNCyNLKYBCaj78M3nLn/OM8IycbnmGuq7aEoEah757ya3jQORk3hB6bksOhktTyFNZNKC6LBsc7hhQhVuT2llCQVhcMOiudfj8HwcCt0AKQ6wqIL5CvrKNKriSS+RkedizkIJ6h6xv2LUJgP4NsukL63bdYD48SEbJJvhVkW7FAdvG0Xz4KyusaDQaEgq8n6ZJTbMWHEmhsBtkjccaxBIWOA3SgEY/azeH6fizkCl4GSCW1hDfZC++aEF7v9Z56NRAlb9c65PX1aYNlW9mrTR1oGrE+Udlo3os/ZV5IV1gr9ZBgHAyzMpFBQ7CPZzvIaSXV74qjl+N0mSUfkzYHnvCnrIXLGWv2Tv2gR2xMRPROGqjb9H3+Ev8Nf5xjiuOttwes52If/4FarPjeg==</latexit><latexit sha1_base64="h0vKyMv1txiryC/6SUOT3b8sZzo=">AAACzXicdZFNb9NAEIbX5quEAoErlxUViAOKbC4gTlRcOBaJtIE4isbrcbzKfli746aRZYkDCPH3uPEr+Aus0wg1bZnTq5nZd2eeyWslPSXJ7yi+cfPW7Tt7dwf39u8/eDh8tH/sbeMEjoVV1k1y8KikwTFJUjipHYLOFZ7ky/d9/eQUnZfWfKJ1jTMNCyNLKYBCaj78M3nLn/OM8IycbnmGuq7aEoEah757ya3jQORk3hB6bksOhktTyFNZNKC6LBsc7hhQhVuT2llCQVhcMOiudfj8HwcCt0AKQ6wqIL5CvrKNKriSS+RkedizkIJ6h6xv2LUJgP4NsukL63bdYD48SEbJJvhVkW7FAdvG0Xz4KyusaDQaEgq8n6ZJTbMWHEmhsBtkjccaxBIWOA3SgEY/azeH6fizkCl4GSCW1hDfZC++aEF7v9Z56NRAlb9c65PX1aYNlW9mrTR1oGrE+Udlo3os/ZV5IV1gr9ZBgHAyzMpFBQ7CPZzvIaSXV74qjl+N0mSUfkzYHnvCnrIXLGWv2Tv2gR2xMRPROGqjb9H3+Ev8Nf5xjiuOttwes52If/4FarPjeg==</latexit><latexit sha1_base64="jgC96ZbtUjoycXYPDL+iaX6PzEU=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhK9gLitMCF4yLR3UJTVRNn0lh17MiebKmiSBxAiCs/jRu/gr+A043QdneZ09Ob5+eZN2mlpKMo+h2EV65eu35j7+bg1u07d+8N7z84cqa2AifCKGOnKThUUuOEJCmcVhahTBUep6u3Xf/4BK2TRn+gTYXzEpZa5lIAeWox/DN9xZ/yhPAz2bLhCZZV0eQIVFt07XNuLAciK9Oa0HGTc9Bc6kyeyKwG1SbJ4PWOARXYm1TWEArC7IxBe6nDx/84ENglkh9iXQDxNfK1qVXGlVwhJ8P9npkU1DkknWDXxgf0b5Ctzq/btoPFcBSNo23xiyDuwYj1dbgY/koyI+oSNQkFzs3iqKJ5A5akUNgOktphBWIFS5x5qKFEN2+2h2n5E89kPPch5kYT37JnXzRQOrcpU68sgQp3vteRl/VmNeUv543UlU9Vi9OP8lp1sXRX5pm0Pnu18QCElX5WLgqw4O9hXRdCfH7li+BofxxH4/h9NDp408exxx6xx+wZi9kLdsDesUM2YSKYBE3wNfgWfgq/hN/DH6fSMOjfPGQ7Ff78C0xY5Lo=</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="h0vKyMv1txiryC/6SUOT3b8sZzo=">AAACzXicdZFNb9NAEIbX5quEAoErlxUViAOKbC4gTlRcOBaJtIE4isbrcbzKfli746aRZYkDCPH3uPEr+Aus0wg1bZnTq5nZd2eeyWslPSXJ7yi+cfPW7Tt7dwf39u8/eDh8tH/sbeMEjoVV1k1y8KikwTFJUjipHYLOFZ7ky/d9/eQUnZfWfKJ1jTMNCyNLKYBCaj78M3nLn/OM8IycbnmGuq7aEoEah757ya3jQORk3hB6bksOhktTyFNZNKC6LBsc7hhQhVuT2llCQVhcMOiudfj8HwcCt0AKQ6wqIL5CvrKNKriSS+RkedizkIJ6h6xv2LUJgP4NsukL63bdYD48SEbJJvhVkW7FAdvG0Xz4KyusaDQaEgq8n6ZJTbMWHEmhsBtkjccaxBIWOA3SgEY/azeH6fizkCl4GSCW1hDfZC++aEF7v9Z56NRAlb9c65PX1aYNlW9mrTR1oGrE+Udlo3os/ZV5IV1gr9ZBgHAyzMpFBQ7CPZzvIaSXV74qjl+N0mSUfkzYHnvCnrIXLGWv2Tv2gR2xMRPROGqjb9H3+Ev8Nf5xjiuOttwes52If/4FarPjeg==</latexit><latexit sha1_base64="h0vKyMv1txiryC/6SUOT3b8sZzo=">AAACzXicdZFNb9NAEIbX5quEAoErlxUViAOKbC4gTlRcOBaJtIE4isbrcbzKfli746aRZYkDCPH3uPEr+Aus0wg1bZnTq5nZd2eeyWslPSXJ7yi+cfPW7Tt7dwf39u8/eDh8tH/sbeMEjoVV1k1y8KikwTFJUjipHYLOFZ7ky/d9/eQUnZfWfKJ1jTMNCyNLKYBCaj78M3nLn/OM8IycbnmGuq7aEoEah757ya3jQORk3hB6bksOhktTyFNZNKC6LBsc7hhQhVuT2llCQVhcMOiudfj8HwcCt0AKQ6wqIL5CvrKNKriSS+RkedizkIJ6h6xv2LUJgP4NsukL63bdYD48SEbJJvhVkW7FAdvG0Xz4KyusaDQaEgq8n6ZJTbMWHEmhsBtkjccaxBIWOA3SgEY/azeH6fizkCl4GSCW1hDfZC++aEF7v9Z56NRAlb9c65PX1aYNlW9mrTR1oGrE+Udlo3os/ZV5IV1gr9ZBgHAyzMpFBQ7CPZzvIaSXV74qjl+N0mSUfkzYHnvCnrIXLGWv2Tv2gR2xMRPROGqjb9H3+Ev8Nf5xjiuOttwes52If/4FarPjeg==</latexit><latexit sha1_base64="jgC96ZbtUjoycXYPDL+iaX6PzEU=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhK9gLitMCF4yLR3UJTVRNn0lh17MiebKmiSBxAiCs/jRu/gr+A043QdneZ09Ob5+eZN2mlpKMo+h2EV65eu35j7+bg1u07d+8N7z84cqa2AifCKGOnKThUUuOEJCmcVhahTBUep6u3Xf/4BK2TRn+gTYXzEpZa5lIAeWox/DN9xZ/yhPAz2bLhCZZV0eQIVFt07XNuLAciK9Oa0HGTc9Bc6kyeyKwG1SbJ4PWOARXYm1TWEArC7IxBe6nDx/84ENglkh9iXQDxNfK1qVXGlVwhJ8P9npkU1DkknWDXxgf0b5Ctzq/btoPFcBSNo23xiyDuwYj1dbgY/koyI+oSNQkFzs3iqKJ5A5akUNgOktphBWIFS5x5qKFEN2+2h2n5E89kPPch5kYT37JnXzRQOrcpU68sgQp3vteRl/VmNeUv543UlU9Vi9OP8lp1sXRX5pm0Pnu18QCElX5WLgqw4O9hXRdCfH7li+BofxxH4/h9NDp408exxx6xx+wZi9kLdsDesUM2YSKYBE3wNfgWfgq/hN/DH6fSMOjfPGQ7Ff78C0xY5Lo=</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="h0vKyMv1txiryC/6SUOT3b8sZzo=">AAACzXicdZFNb9NAEIbX5quEAoErlxUViAOKbC4gTlRcOBaJtIE4isbrcbzKfli746aRZYkDCPH3uPEr+Aus0wg1bZnTq5nZd2eeyWslPSXJ7yi+cfPW7Tt7dwf39u8/eDh8tH/sbeMEjoVV1k1y8KikwTFJUjipHYLOFZ7ky/d9/eQUnZfWfKJ1jTMNCyNLKYBCaj78M3nLn/OM8IycbnmGuq7aEoEah757ya3jQORk3hB6bksOhktTyFNZNKC6LBsc7hhQhVuT2llCQVhcMOiudfj8HwcCt0AKQ6wqIL5CvrKNKriSS+RkedizkIJ6h6xv2LUJgP4NsukL63bdYD48SEbJJvhVkW7FAdvG0Xz4KyusaDQaEgq8n6ZJTbMWHEmhsBtkjccaxBIWOA3SgEY/azeH6fizkCl4GSCW1hDfZC++aEF7v9Z56NRAlb9c65PX1aYNlW9mrTR1oGrE+Udlo3os/ZV5IV1gr9ZBgHAyzMpFBQ7CPZzvIaSXV74qjl+N0mSUfkzYHnvCnrIXLGWv2Tv2gR2xMRPROGqjb9H3+Ev8Nf5xjiuOttwes52If/4FarPjeg==</latexit><latexit sha1_base64="h0vKyMv1txiryC/6SUOT3b8sZzo=">AAACzXicdZFNb9NAEIbX5quEAoErlxUViAOKbC4gTlRcOBaJtIE4isbrcbzKfli746aRZYkDCPH3uPEr+Aus0wg1bZnTq5nZd2eeyWslPSXJ7yi+cfPW7Tt7dwf39u8/eDh8tH/sbeMEjoVV1k1y8KikwTFJUjipHYLOFZ7ky/d9/eQUnZfWfKJ1jTMNCyNLKYBCaj78M3nLn/OM8IycbnmGuq7aEoEah757ya3jQORk3hB6bksOhktTyFNZNKC6LBsc7hhQhVuT2llCQVhcMOiudfj8HwcCt0AKQ6wqIL5CvrKNKriSS+RkedizkIJ6h6xv2LUJgP4NsukL63bdYD48SEbJJvhVkW7FAdvG0Xz4KyusaDQaEgq8n6ZJTbMWHEmhsBtkjccaxBIWOA3SgEY/azeH6fizkCl4GSCW1hDfZC++aEF7v9Z56NRAlb9c65PX1aYNlW9mrTR1oGrE+Udlo3os/ZV5IV1gr9ZBgHAyzMpFBQ7CPZzvIaSXV74qjl+N0mSUfkzYHnvCnrIXLGWv2Tv2gR2xMRPROGqjb9H3+Ev8Nf5xjiuOttwes52If/4FarPjeg==</latexit><latexit sha1_base64="jgC96ZbtUjoycXYPDL+iaX6PzEU=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhK9gLitMCF4yLR3UJTVRNn0lh17MiebKmiSBxAiCs/jRu/gr+A043QdneZ09Ob5+eZN2mlpKMo+h2EV65eu35j7+bg1u07d+8N7z84cqa2AifCKGOnKThUUuOEJCmcVhahTBUep6u3Xf/4BK2TRn+gTYXzEpZa5lIAeWox/DN9xZ/yhPAz2bLhCZZV0eQIVFt07XNuLAciK9Oa0HGTc9Bc6kyeyKwG1SbJ4PWOARXYm1TWEArC7IxBe6nDx/84ENglkh9iXQDxNfK1qVXGlVwhJ8P9npkU1DkknWDXxgf0b5Ctzq/btoPFcBSNo23xiyDuwYj1dbgY/koyI+oSNQkFzs3iqKJ5A5akUNgOktphBWIFS5x5qKFEN2+2h2n5E89kPPch5kYT37JnXzRQOrcpU68sgQp3vteRl/VmNeUv543UlU9Vi9OP8lp1sXRX5pm0Pnu18QCElX5WLgqw4O9hXRdCfH7li+BofxxH4/h9NDp408exxx6xx+wZi9kLdsDesUM2YSKYBE3wNfgWfgq/hN/DH6fSMOjfPGQ7Ff78C0xY5Lo=</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit><latexit sha1_base64="Gz8RXW10rUfhoCUp5eXuuMqZf3s=">AAAC2HicdVFNj9MwEHXC11K+Chy5WFQgDqhKEBKI0wIXjotEdwtNVU2cSWPVsSN7sqWKInEAIa78NG78Cv4CTjdC291lTk9vnp9n3qSVko6i6HcQXrp85eq1veuDGzdv3b4zvHvv0JnaCpwIo4ydpuBQSY0TkqRwWlmEMlV4lK7edv2jY7ROGv2BNhXOS1hqmUsB5KnF8M/0FX/ME8LPZMuGJ1hWRZMjUG3RtU+5sRyIrExrQsdNzkFzqTN5LLMaVJskg9c7BlRgb1JZQygIs1MG7YUOH//jQGCXSH6IdQHE18jXplYZV3KFnAz3e2ZSUOeQdIJdGx/Qv0G2Or9u2w4Ww1E0jrbFz4O4ByPW18Fi+CvJjKhL1CQUODeLo4rmDViSQmE7SGqHFYgVLHHmoYYS3bzZHqbljzyT8dyHmBtNfMueftFA6dymTL2yBCrc2V5HXtSb1ZS/nDdSVz5VLU4+ymvVxdJdmWfS+uzVxgMQVvpZuSjAgr+HdV0I8dmVz4PDZ+M4Gsfvn4/23/Rx7LEH7CF7wmL2gu2zd+yATZgIJkETfA2+hZ/CL+H38MeJNAz6N/fZToU//wJNmOS+</latexit>
PredictionProblems• PredictionheremeansinferringapropertyYthatwillbeusedfordecisionmaking.
• Forexample:– Y=1means“thispersonwilldefaultonaloan”(forthedecision,“shouldIgivethispersonaloan”?)
– Y=1means“thispersonwillcommitacrimeintwoyears”(forthedecision,“shouldIreleasethisconvictnow?”)
• WewouldliketopredictYina“fair”way,meaningthatourpredictionsshouldnotbe“biased”againstparticularinstancesofA.
Primitives
• EvenifwetakethechoiceofwhatgoesinAasaprimitive,itisstillnotobviouswhatwemeanbybeingfair.
• Afirstidea:“ensurethatdoesnotuseA”.
• Thisisknowntobeunsatisfactory.
Y<latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit><latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit><latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit><latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit>
Examples• Equalizedodds:giventheoutcomeY,attributeAprovidesnofurtherinformationaboutmyprediction.
• Calibration:givenmyprediction,attributeAprovidesnofurtherinformationabouttheoutcomeY.
• IfAisonaverageinformativeofY,wecannotreconciletheabove.– Remember,herewedonotcontrolY(directly).Wedecideonpredictor.Y
<latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit><latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit><latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit><latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit>
Y<latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit><latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit><latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit><latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit>
Y<latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit><latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit><latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit><latexit sha1_base64="M+ZGxSf9Wvacj2hahaLwRB9eAwM=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUiyhNnJbDJmdmaZ6RVCyD948aCIV//Hm3/jJNmDJhY0FFXddHdFqRQWff/bW1ldW9/YLGwVt3d29/ZLB4cNqzPDeJ1pqU0ropZLoXgdBUreSg2nSSR5MxreTP3mEzdWaHWPo5SHCe0rEQtG0UmNzoAieeiWyn7Fn4EskyAnZchR65a+Oj3NsoQrZJJa2w78FMMxNSiY5JNiJ7M8pWxI+7ztqKIJt+F4du2EnDqlR2JtXCkkM/X3xJgm1o6SyHUmFAd20ZuK/3ntDOOrcCxUmiFXbL4oziRBTaavk54wnKEcOUKZEe5WwgbUUIYuoKILIVh8eZk0ziuBXwnuLsrV6zyOAhzDCZxBAJdQhVuoQR0YPMIzvMKbp70X7937mLeuePnMEfyB9/kDEfyOyA==</latexit>
PuttingItintheContextofaCausalModel
• Atoymodel:imagineAisrace,Xis“ownsaredcar”andYis“crashescarinoneyear”.
• Let’s(informally)drawacausaldiagramshowingcause-effectrelationshipsamongthose.Itwillincludea“unobservedtrait”Umeasuringaggressiveness.– Wewillgetintomoreformaldefinitionslater.
ACausalDiagram
SomeInitialConclusions
• AisnotacauseofY.
• IfwebuildapredictorbasedonX,ittellsussomethingbothaboutAandaboutU.
• Hence,ourpredictorwillbedifferentfordifferentvaluesofA,whichdoesnotseemappropriate.
ASecondCausalDiagram
WhichConclusions?
• AisnowacauseofY(indirectly).
• Itisnowimpossibletosatisfybothequalizedoddsandcalibrationsimultaneously.
• Judgmentcall:isthepathwayAèXèY“fair”?
ZoomingIn,withAnotherExample
• Aherestandsforrace,Yforloandefault.
• Sameidea,augmentedwithamediator:
ACausalPrimitive:CounterfactualFairness
• Ifwehavesomeprotectedattributelikerace,andadecisionsuchaslengthofsentence,thenourdecisionsatisfiescounterfactualfairnessif
• Acausalmodelisnecessarytoinfersuchclaimsfromdata.
“hadtheprotectedattributes(e.g.,race)oftheindividualbeendifferent,otherthingsbeingequal,thedecisionwouldhaveremainedthesame”
Workflow
• Regardlessofthemachinelearningalgorithmtobeused,workwithadomainexperttoestimateacausalmodelofyourdata.– It’samodeloftheworld,notofyoursoftware.
• Chooseanymachinelearningalgorithmofinterest,anyblack-boxthattakesasinputsobservedandunobservedvariablesinyourdomain.– Selectasetofvariablesbasedonwhichsetsrespectcounterfactualfairness.
– Ifnecessary,inferunobservedvariablesfromtheobservedones.
FormalizingtheIdea
• FormalnotionsofcounterfactualsdatebackatleasttoJerzyNeymaninthe1920s.
• IwillfollowmostlytheStructuralCausalModel(SCM)frameworkofJudeaPearl,whichhascloselinkstotheworkofJamesRobins,andthatofSpirtes,GlymourandScheines.
StructuralCausalModels
• Adirectedacyclicgraph(DAG)postulates“directcause-effect”pairs.– Eachvertexinthegraphisarandomvariableinadistribution.
• EachvariableVisgivenanequationthatdeterministicallydefinesthevalueofVasafunctionofits“parents”.– Suchequationsarepostulatedtobestructural,inthesensethatitfollowsthecause-effectdirection.
TheOperationalMeaning
• This“DAGwithequations”iscausalinthesensethatitmustencodetheeffectsofaperfectintervention.
Rain Barometer
URainUBarometer
Rain=fR(URain)Barometer=fB(Rain,UBarometer)
Interventions
• Anotherprimitive.Itisa“overriding”operator,setsavariabletoafixedvalueofinterest.Lowercasehererepresentsconstants.
r Barometer
URainUBarometer
Rain=rBarometer=fB(r,UBarometer)
Interventions
• Itisthenotionofinterventionthatleadstotheasymmetricnatureofcausality.
Rain b
URainUBarometer
Rain=fR(URain)Barometer=b
Interventions
• Itisthenotionofinterventionthatexplainswhy“correlationisnotcausation”.
Movingto
Florida
Dyingofoldage
Survivingoldage
yes Dyingofoldage
Survivingoldage
Notation:the“do”Operator
• Wemustexpresshow“Dyingofoldage”varieswith“MovingtoFlorida”inbothcases.
• Traditionally,conditionalprobabilitiescanbeusedforthat.Butnoticethat,inourexample,istrueintheobservationalcase(nointervention),butfalseintheinterventionalcase.
P(Dyingofoldage=True|MovingtoFlorida=True)≠P(Dyingofoldage=True|MovingtoFlorida=False)
Notation:the“do”Operator
• InPearl’scalculus,thisisdistinguishedbyusingthe“do”operatortoindicateaninterventionasopposedtoanobservation.
P(Dyingofoldage=True|MovingtoFlorida=True)≠P(Dyingofoldage=True|MovingtoFlorida=False)
P(Dyingofoldage=True|do(MovingtoFlorida=True))=P(Dyingofoldage=True|do(MovingtoFlorida=False))
AveragesVs.Individuals
• Thistypeofnotationcanbeusedtoexpresswhetheradrugiseffectiveornot,averagingoverapopulation,usingarandomizedcontrolledtrial:
• Itdoesnotmakeanyclaims,however,onwhetherthereisabalanceofpositive/negativecasesthatcancelout.
P(Healthy=True|do(Treatment=Drug))?=P(Healthy=True|do(Treatment=Placebo))
Notation:CounterfactualIndices
• Meanttocaptureindividual-levelvariability.
• ForVjavariableinthesystem,andViavariablebeingintervenedatvaluev,weuseVj(v)asthecounterfactualvalueofVj,hadVibeingsettov.
• Contextwilltelluswhichvariablethevalue“v”refersto.
Example
• Notice:itiscommontorepresentVj(v)asjustVjifViisnota(directorindirect)causeofVj.
r Barometer(r)
Urain(r) Ubarometer(r)
r Barometer(r)
Urain Ubarometer
“OtherThingsBeingEqual”
• Thatis,– Acounterfactualvaluereplacesthecauseofinterest
– Thecounterfactualvaluepropagates“downstream”thecausalgraphviathestructuralequations
– Everythingelseremainsthesame(“otherthingsbeingequal”),i.e.,thenon-descendantsofthemanipulatedvariable.
MultipleWorlds• Acounterfactualisjustadifferent“version”ofthesameindividual.All“versions”co-existinonebigjointdistribution.
Rain Barometer
URainUBarometer
r Barometer(r)
r’ Barometer(r’)
Workflow
• Regardlessofthemachinelearningalgorithmtobeused,workwithadomainexperttoestimateacausalmodelofyourdata.– It’samodeloftheworld,notofyoursoftware.
• Chooseanymachinelearningalgorithmofinterest,anyblack-boxthattakesasinputsobservedandunobservedvariablesinyourdomain.– Selectasetofvariablesbasedonwhichsetsrespectcounterfactualfairness.
– Ifnecessary,inferunobservedvariablesfromtheobservedones.
BacktoCounterfactualFairness
• Thelawofcounterfactualpropagationmeansthat,ifwewanttoensureitissufficient(andnecessary,ingeneral)toincludeonlythenon-descendantsofAinthedefinitionofthepredictor.
P (Y (a) = y | A = a,X = x) = P (Y (a0) = y | A = a,X = x)<latexit sha1_base64="ikA4UWRxXdG1jbKtP7ZSuE5mo/g=">AAACL3icdVDLSgMxFM34rPU16tJNsIgVpMyIoBuhKojLCvYhnaHcSdM2NPMgyYhD7R+58Ve6EVHErX9hpi2orR4IOZxzLsk9XsSZVJb1YszMzs0vLGaWsssrq2vr5sZmRYaxILRMQh6KmgeSchbQsmKK01okKPgep1Wve5H61TsqJAuDG5VE1PWhHbAWI6C01DAvS3mnAwrf5mEfn+LEwQ8OPtMMDnBNX/ep+p3Z+yfUMHNWwRoCTxN7THJojFLDHDjNkMQ+DRThIGXdtiLl9kAoRjjtZ51Y0ghIF9q0rmkAPpVub7hvH+9qpYlbodAnUHio/pzogS9l4ns66YPqyEkvFf/y6rFqnbg9FkSxogEZPdSKOVYhTsvDTSYoUTzRBIhg+q+YdEAAUbrirC7Bnlx5mlQOC7ZVsK+PcsXzcR0ZtI12UB7Z6BgV0RUqoTIi6BEN0Ct6M56MZ+Pd+BhFZ4zxzBb6BePzC/vYoRk=</latexit><latexit sha1_base64="ikA4UWRxXdG1jbKtP7ZSuE5mo/g=">AAACL3icdVDLSgMxFM34rPU16tJNsIgVpMyIoBuhKojLCvYhnaHcSdM2NPMgyYhD7R+58Ve6EVHErX9hpi2orR4IOZxzLsk9XsSZVJb1YszMzs0vLGaWsssrq2vr5sZmRYaxILRMQh6KmgeSchbQsmKK01okKPgep1Wve5H61TsqJAuDG5VE1PWhHbAWI6C01DAvS3mnAwrf5mEfn+LEwQ8OPtMMDnBNX/ep+p3Z+yfUMHNWwRoCTxN7THJojFLDHDjNkMQ+DRThIGXdtiLl9kAoRjjtZ51Y0ghIF9q0rmkAPpVub7hvH+9qpYlbodAnUHio/pzogS9l4ns66YPqyEkvFf/y6rFqnbg9FkSxogEZPdSKOVYhTsvDTSYoUTzRBIhg+q+YdEAAUbrirC7Bnlx5mlQOC7ZVsK+PcsXzcR0ZtI12UB7Z6BgV0RUqoTIi6BEN0Ct6M56MZ+Pd+BhFZ4zxzBb6BePzC/vYoRk=</latexit><latexit sha1_base64="ikA4UWRxXdG1jbKtP7ZSuE5mo/g=">AAACL3icdVDLSgMxFM34rPU16tJNsIgVpMyIoBuhKojLCvYhnaHcSdM2NPMgyYhD7R+58Ve6EVHErX9hpi2orR4IOZxzLsk9XsSZVJb1YszMzs0vLGaWsssrq2vr5sZmRYaxILRMQh6KmgeSchbQsmKK01okKPgep1Wve5H61TsqJAuDG5VE1PWhHbAWI6C01DAvS3mnAwrf5mEfn+LEwQ8OPtMMDnBNX/ep+p3Z+yfUMHNWwRoCTxN7THJojFLDHDjNkMQ+DRThIGXdtiLl9kAoRjjtZ51Y0ghIF9q0rmkAPpVub7hvH+9qpYlbodAnUHio/pzogS9l4ns66YPqyEkvFf/y6rFqnbg9FkSxogEZPdSKOVYhTsvDTSYoUTzRBIhg+q+YdEAAUbrirC7Bnlx5mlQOC7ZVsK+PcsXzcR0ZtI12UB7Z6BgV0RUqoTIi6BEN0Ct6M56MZ+Pd+BhFZ4zxzBb6BePzC/vYoRk=</latexit><latexit sha1_base64="ikA4UWRxXdG1jbKtP7ZSuE5mo/g=">AAACL3icdVDLSgMxFM34rPU16tJNsIgVpMyIoBuhKojLCvYhnaHcSdM2NPMgyYhD7R+58Ve6EVHErX9hpi2orR4IOZxzLsk9XsSZVJb1YszMzs0vLGaWsssrq2vr5sZmRYaxILRMQh6KmgeSchbQsmKK01okKPgep1Wve5H61TsqJAuDG5VE1PWhHbAWI6C01DAvS3mnAwrf5mEfn+LEwQ8OPtMMDnBNX/ep+p3Z+yfUMHNWwRoCTxN7THJojFLDHDjNkMQ+DRThIGXdtiLl9kAoRjjtZ51Y0ghIF9q0rmkAPpVub7hvH+9qpYlbodAnUHio/pzogS9l4ns66YPqyEkvFf/y6rFqnbg9FkSxogEZPdSKOVYhTsvDTSYoUTzRBIhg+q+YdEAAUbrirC7Bnlx5mlQOC7ZVsK+PcsXzcR0ZtI12UB7Z6BgV0RUqoTIi6BEN0Ct6M56MZ+Pd+BhFZ4zxzBb6BePzC/vYoRk=</latexit>
• A,Employedcannotbeused.PrejudicedandQualificationscan.
• IfitisjudgedthatPrejudicedcannotbeused,itshouldbelabelledasaprotectedattributed.
Examples
• A,Xcannotbeused.Ucan.
HowtoExtractUnobservedVariables?
• Usethe“factual”distributiontogetadistributionovertheunobservedvariablesbystandardprobabilisticconditioning.
• MonteCarlodataaugmentationapproach:replaceeachdatapointinyourtrainingsamplebyasetoftrainingpointswiththeunobservedvariablesbeingfilledbyaMonteCarlosample.
P(Unobserved|Observed)
Workflow
• Regardlessofthemachinelearningalgorithmtobeused,workwithadomainexperttoestimateacausalmodelofyourdata.– It’samodeloftheworld,notofyoursoftware.
• Chooseanymachinelearningalgorithmofinterest,anyblack-boxthattakesasinputsobservedandunobservedvariablesinyourdomain.– Selectasetofvariablesbasedonwhichsetsrespectcounterfactualfairness.
– Ifnecessary,inferunobservedvariablesfromtheobservedones.
Algorithm
Interpretation
• ExtractcausesofYwhicharenotmediatorsbetweenAandY.
• Findthe“bestapproximation”toYwithinthespaceoffunctionsthatexcludesuchmediators.
• EvenifYis“unfair”(Aisacauseofit),byconstructionthepredictorwillbecounterfactuallyfair.
Challenges
• Counterfactualfairnessclarifiesthatalgorithmicfairnessingeneralisnotexplicitlymodelinghowthewordbecomesfairerwithfairpredictions.– Evenifourdecisionofgivingaloanisfair,itdoesn’tmeanthatinaggregatetheprobabilityofapersonofaparticulardemographicgroupwon’thavedifficultiesinrepayingit(AstillcausesY).
• Thedelayedimpactoffairpredictionsisalsoaresearchtopic– seeLiuetal.,https://arxiv.org/pdf/1803.04383.pdf
Workflow
• Regardlessofthemachinelearningalgorithmtobeused,workwithadomainexperttoestimateacausalmodelofyourdata.– It’samodeloftheworld,notofyoursoftware.
• Chooseanymachinelearningalgorithmofinterest,anyblack-boxthattakesasinputsobservedandunobservedvariablesinyourdomain.– Selectasetofvariablesbasedonwhichsetsrespectcounterfactualfairness.
– Ifnecessary,inferunobservedvariablesfromtheobservedones.
SomeWordsofCaution• Structuralequationsuseunobservedvariables.
• Itiscommonthatsomeofthesevariablesare“default”choicesbasedonsomegenericmodelingassumptionsuchasadditiveerrors.
• Natureandsocietycouldn’tcarelesswhetheryourmathematicallyconvenientwayofseparatingsignaland“noise”iselegantornot.
Output=Signal+Noise
SomeWordsofCaution
• Thatis,thereareinfinitelymanystructuralequationsVj=fj(Vi,Uj)compatiblewithP(Vi|Vj)andP(Vi|do(Vj)).
• Signalvs.noisemustbedeterminedbyreal-worldassumptions(“simplicity”assumptions,oftheOckham’srazortype,canbesometimesadequateaslongascaveatsareadvertised).
SomeWordsofCaution
• Bynow,thereareseveralgoodpapersonhowtotacklefairnessbygeneratingunobservedvariableswhichareindependentofA,usingassortedmethods.
X
U
A“Thenoffyougotoplug-inUonamachinelearningalgorithm!”
However• Therearepapersnotcausallymotivated,whichIfindof
difficultinterpretation.– Remember:thereareinfinitelymanywaysofextractingU.
• Therearepaperscausallymotivated,butwhichcommitthemselvestoadomain-freefamilyofstructuralequations.OKenough,butwhywouldyoudothat?– Counterfactualfairnessemphasizesthatthecausalmodelingstepisseparatefromthepredictionlearningprocess.
• Finally,dobewareofanypaperthatclaimstodoassumption-freeextractionof“causallatentfactors”.Thosearesellingyousnakeoil!
InterpretationofCounterfactuals• Butwhatdoesitmeantosay“hadmyracebeendifferent”??– First,makesuretounderstandthedifferencebetween“A”and“PerceptionofA”:thesecanleadtoconceptuallydifferentinterpretations,evenifthemodelstaysthesame.
– Withoutgoingindetails,ifthosecounterfactualsmakeyoufeeluneasy,justinterpretthemascomparingtwodifferentpeoplewhohappentomatchonthe“otherthingsbeingequal”factors.• Thisisalsorelatedtofairnessthroughawareness(Dworketal.,2011,https://arxiv.org/abs/1104.3913)
Non-CounterfactualCausalModels
• Contrarytofolkknowledge,causalitydoesnotrequirecounterfactuals:the“do”operatorisanwayofcomparingtreatmentswithoutcomparingindividuals.
• However,iffeaturesXareaffectedbyA,theningeneralthereisnoindividualwhere
• IffeaturesXarenotaffectedbyA,thenwecanshowwedonotneedtoexplicitlymodelstructuralequationsanyway!
P(Y=y|X=x,do(A=a))=P(Y=y|X=x,do(A=a’))
TheUpside• Becausestructuralcausalmodelsrelyonunobservedvariables,atleasttheycanbepartiallyfalsifiedbyeventuallymeasuringsomeofthosevariables.
• Justkeepinmind:– whileitispreposteroustosayyouhave“the”causalmodelofasocialprocess,youshould(must?)beabletoexplainyourassumptionstoaregulatororacustomer.
– Havingpassedtestableimplications,theremainingcomponentsofacounterfactualmodelshouldbeunderstoodasconjecturesformulatedaccordingtothebestofourknowledge.Suchmodelsshouldalwaysbedeemedprovisionalandpronetomodifications.
Illustration
• TheLawSchoolAdmissionCouncilconductedasurveyacross163lawschoolsintheUnitedStates– Itcontainsinformationon21,790lawstudentssuchastheirentranceexamscores(LSAT),theirgrade-pointaverage(GPA)collectedpriortolawschool,andtheirfirstyearaveragegrade(FYA).
• Task:predictifanapplicantwillhaveahighFYA– Exampleofdecisionproblem:makeanoffer
Setup
• Iwillpresentsomesimplecausalmodelsforthisdomain,whichbynomeansIintendtosellaswell-thoughtmodels.Theirpurposeisforillustration.
• Wewillfitrealdatatoamodel,thengeneratesyntheticcounterfactualsoutofit.Thepointistoquantifytowhatextentacausally-obliviousmethodviolatescounterfactualfairness.
TwoModels
“FairK” “FairAdd”
PredictiveError(RealData)
• Comparisonagainst“Full”(linearmodelwithallvariables)and“Unaware”(linearmodelwithoutraceandgender,buttheothertwopredictors)– Evaluationbyrootmeansquarederror
FairnessViolations(SimulatedCounterfactuals)
Extension:UsingMultipleModels
• Wejustsawtwodifferentcounterfactualmodelsthatgivedifferentpredictionsdespitebeingundistinguishablegiventhesamedata.
• ThisisOKassuminglittledifferencebetweenmodels,butwemayhavecompetingtheorieswithsomesizeabledifference.Wewouldliketobe“approximatelycounterfactuallyfair”toallofthem.
(ε,δ)-CounterfactualFairness• Thefollowingconstraintprovidesarelaxationofcounterfactualfairness:
• Theideaistosimultaneouslysatisfysuchconstraintsaccordingtodifferentcounterfactualmodels.– Itisnothardtoshowthatthisproblemingeneralhasnosolutionifε=0,hencetheneedforanapproximateversion.
P (|Y (a)� Y (a0)| ✏ | A = a,X = a) � 1� �<latexit sha1_base64="6ftP5coO2gomwvZHxhd+kDhDw4c=">AAACM3icbVBdaxNBFJ1ta61R29Q++nIxiAnYsFsK+iL040X6FMG0kWwIdyc3ydDZ2XXmbiEk/U996R/xQSg+KMVX/4OzaQra9sAwh3PuYeaeJNfKcRheBUvLK49WH689qTx99nx9o7r54thlhZXUlpnObCdBR1oZarNiTZ3cEqaJppPk9LD0T87IOpWZzzzJqZfiyKihkshe6lePWvVZPEaGL3VswDbc8jeNGcSavkJMuVM6MzHMYtiHD4BvoVNeDYhH3o/K0IA0Y79aC5vhHHCfRAtSEwu0+tVv8SCTRUqGpUbnulGYc2+KlpXUdF6JC0c5ylMcUddTgym53nS+8zm89soAhpn1xzDM1X8TU0ydm6SJn0yRx+6uV4oPed2Ch+97U2XygsnIm4eGhQbOoCwQBsqSZD3xBKVV/q8gx2hRsq+54kuI7q58nxzvNKOwGX3are0dLOpYEy/FK1EXkXgn9sRH0RJtIcWF+C5+il/BZfAjuA5+34wuBYvMlvgPwZ+/koulag==</latexit><latexit sha1_base64="6ftP5coO2gomwvZHxhd+kDhDw4c=">AAACM3icbVBdaxNBFJ1ta61R29Q++nIxiAnYsFsK+iL040X6FMG0kWwIdyc3ydDZ2XXmbiEk/U996R/xQSg+KMVX/4OzaQra9sAwh3PuYeaeJNfKcRheBUvLK49WH689qTx99nx9o7r54thlhZXUlpnObCdBR1oZarNiTZ3cEqaJppPk9LD0T87IOpWZzzzJqZfiyKihkshe6lePWvVZPEaGL3VswDbc8jeNGcSavkJMuVM6MzHMYtiHD4BvoVNeDYhH3o/K0IA0Y79aC5vhHHCfRAtSEwu0+tVv8SCTRUqGpUbnulGYc2+KlpXUdF6JC0c5ylMcUddTgym53nS+8zm89soAhpn1xzDM1X8TU0ydm6SJn0yRx+6uV4oPed2Ch+97U2XygsnIm4eGhQbOoCwQBsqSZD3xBKVV/q8gx2hRsq+54kuI7q58nxzvNKOwGX3are0dLOpYEy/FK1EXkXgn9sRH0RJtIcWF+C5+il/BZfAjuA5+34wuBYvMlvgPwZ+/koulag==</latexit><latexit sha1_base64="6ftP5coO2gomwvZHxhd+kDhDw4c=">AAACM3icbVBdaxNBFJ1ta61R29Q++nIxiAnYsFsK+iL040X6FMG0kWwIdyc3ydDZ2XXmbiEk/U996R/xQSg+KMVX/4OzaQra9sAwh3PuYeaeJNfKcRheBUvLK49WH689qTx99nx9o7r54thlhZXUlpnObCdBR1oZarNiTZ3cEqaJppPk9LD0T87IOpWZzzzJqZfiyKihkshe6lePWvVZPEaGL3VswDbc8jeNGcSavkJMuVM6MzHMYtiHD4BvoVNeDYhH3o/K0IA0Y79aC5vhHHCfRAtSEwu0+tVv8SCTRUqGpUbnulGYc2+KlpXUdF6JC0c5ylMcUddTgym53nS+8zm89soAhpn1xzDM1X8TU0ydm6SJn0yRx+6uV4oPed2Ch+97U2XygsnIm4eGhQbOoCwQBsqSZD3xBKVV/q8gx2hRsq+54kuI7q58nxzvNKOwGX3are0dLOpYEy/FK1EXkXgn9sRH0RJtIcWF+C5+il/BZfAjuA5+34wuBYvMlvgPwZ+/koulag==</latexit><latexit sha1_base64="6ftP5coO2gomwvZHxhd+kDhDw4c=">AAACM3icbVBdaxNBFJ1ta61R29Q++nIxiAnYsFsK+iL040X6FMG0kWwIdyc3ydDZ2XXmbiEk/U996R/xQSg+KMVX/4OzaQra9sAwh3PuYeaeJNfKcRheBUvLK49WH689qTx99nx9o7r54thlhZXUlpnObCdBR1oZarNiTZ3cEqaJppPk9LD0T87IOpWZzzzJqZfiyKihkshe6lePWvVZPEaGL3VswDbc8jeNGcSavkJMuVM6MzHMYtiHD4BvoVNeDYhH3o/K0IA0Y79aC5vhHHCfRAtSEwu0+tVv8SCTRUqGpUbnulGYc2+KlpXUdF6JC0c5ylMcUddTgym53nS+8zm89soAhpn1xzDM1X8TU0ydm6SJn0yRx+6uV4oPed2Ch+97U2XygsnIm4eGhQbOoCwQBsqSZD3xBKVV/q8gx2hRsq+54kuI7q58nxzvNKOwGX3are0dLOpYEy/FK1EXkXgn9sRH0RJtIcWF+C5+il/BZfAjuA5+34wuBYvMlvgPwZ+/koulag==</latexit>
LawSchoolRevisited
COMPAS
ADifferentDirection:Interventions• Sofar,wehavesolelydiscussedthecreationofpredictors.
• Ideally,wewouldliketodestroythepathwaysbetweenAandY,theoutcomeofinterest.
• Thisisingeneralnotpossible.Butlet’sassumewehaveaninterventionwiththeabilityofchangingthecontributionofAtoY.Howisrelatedtocounterfactualfairness?
ImperfectInterventionsandInterference
• Wewillassumetwogeneralizationsoftheconceptofinterventionusedsofar.
• Aninterventionisrepresentedgenericallyasasetof(action)variables,whichhereIwilldenoteasZ.– WecandefineZ=0asthe“noaction”choice!– Z≠0justmeansthatoneormorestructuralequationswillchange,notnecessarilytoaconstant(“imperfect”,or“soft”intervention).
Ideally
• Havingavailablesome“Z=z”whichcompletelyoverridesthestructuralequationforYtonotdependonanythingthatstartsonA.
Y
A
Z
InReality
• Nosuchaninterventionistypicallyavailable.• Andthisisnotapredictionproblemanymore.WhathappenstoY?
• Setup:– AssumeYisencodedsothathighvaluesaregood.– Modelallowsforinterference:thatis,treatmentZigiventopersonimightaffectpersonj.
– Howisthisrelatedtocounterfactualfairness?
OptimizationProblemandConstraints
E[Yi(ai, z) | Ai = ai, Xi = xi]� E[Yi(a0, z) | Ai = ai, Xi = xi]| {z }
Gia0
< ⌧
<latexit sha1_base64="4c9mlclgUDbW9Ia5Ysx9IkEqukU=">AAACe3icbVFdaxNBFJ1dv2r8ivooyMUgidKGXVHsg0KriD5WMG0kuwx3J3fbobOzy8ysNK77J/xpffOf+CI42eRB0x4YOJxzD3fuvVmlpHVR9CsIr1y9dv3G1s3erdt37t7r339waMvaCJqIUpVmmqElJTVNnHSKppUhLDJFR9np+6V/9I2MlaX+4hYVpQUea5lLgc5LvP8zqfWcTGZQUNNLCnQnWQYfZl+5HCGX27CScvj+LIEfkMA+l/AWOmva0TMuU9iBzexwI3pJsou2vPnIG4nDtoU3kDiseX8QjaMOcJHEazJgaxzw/nkyL0VdkHZCobWzOKpc2qBxUihqe0ltqUJxisc081RjQTZtut218NQrc8hL45920Kn/JhosrF0Uma9cjmM3vaV4mTerXb6bNlJXtSMtVo3yWoErYXkImEtDwqmFJyiM9H8FcYL+Ds6fq+eXEG+OfJEcvhjH0Tj+/HKw9269ji32iD1hIxaz12yPfWIHbMIE+x08DobBKPgTDsLn4faqNAzWmYfsP4Sv/gIWTboB</latexit><latexit sha1_base64="4c9mlclgUDbW9Ia5Ysx9IkEqukU=">AAACe3icbVFdaxNBFJ1dv2r8ivooyMUgidKGXVHsg0KriD5WMG0kuwx3J3fbobOzy8ysNK77J/xpffOf+CI42eRB0x4YOJxzD3fuvVmlpHVR9CsIr1y9dv3G1s3erdt37t7r339waMvaCJqIUpVmmqElJTVNnHSKppUhLDJFR9np+6V/9I2MlaX+4hYVpQUea5lLgc5LvP8zqfWcTGZQUNNLCnQnWQYfZl+5HCGX27CScvj+LIEfkMA+l/AWOmva0TMuU9iBzexwI3pJsou2vPnIG4nDtoU3kDiseX8QjaMOcJHEazJgaxzw/nkyL0VdkHZCobWzOKpc2qBxUihqe0ltqUJxisc081RjQTZtut218NQrc8hL45920Kn/JhosrF0Uma9cjmM3vaV4mTerXb6bNlJXtSMtVo3yWoErYXkImEtDwqmFJyiM9H8FcYL+Ds6fq+eXEG+OfJEcvhjH0Tj+/HKw9269ji32iD1hIxaz12yPfWIHbMIE+x08DobBKPgTDsLn4faqNAzWmYfsP4Sv/gIWTboB</latexit><latexit sha1_base64="4c9mlclgUDbW9Ia5Ysx9IkEqukU=">AAACe3icbVFdaxNBFJ1dv2r8ivooyMUgidKGXVHsg0KriD5WMG0kuwx3J3fbobOzy8ysNK77J/xpffOf+CI42eRB0x4YOJxzD3fuvVmlpHVR9CsIr1y9dv3G1s3erdt37t7r339waMvaCJqIUpVmmqElJTVNnHSKppUhLDJFR9np+6V/9I2MlaX+4hYVpQUea5lLgc5LvP8zqfWcTGZQUNNLCnQnWQYfZl+5HCGX27CScvj+LIEfkMA+l/AWOmva0TMuU9iBzexwI3pJsou2vPnIG4nDtoU3kDiseX8QjaMOcJHEazJgaxzw/nkyL0VdkHZCobWzOKpc2qBxUihqe0ltqUJxisc081RjQTZtut218NQrc8hL45920Kn/JhosrF0Uma9cjmM3vaV4mTerXb6bNlJXtSMtVo3yWoErYXkImEtDwqmFJyiM9H8FcYL+Ds6fq+eXEG+OfJEcvhjH0Tj+/HKw9269ji32iD1hIxaz12yPfWIHbMIE+x08DobBKPgTDsLn4faqNAzWmYfsP4Sv/gIWTboB</latexit><latexit sha1_base64="4c9mlclgUDbW9Ia5Ysx9IkEqukU=">AAACe3icbVFdaxNBFJ1dv2r8ivooyMUgidKGXVHsg0KriD5WMG0kuwx3J3fbobOzy8ysNK77J/xpffOf+CI42eRB0x4YOJxzD3fuvVmlpHVR9CsIr1y9dv3G1s3erdt37t7r339waMvaCJqIUpVmmqElJTVNnHSKppUhLDJFR9np+6V/9I2MlaX+4hYVpQUea5lLgc5LvP8zqfWcTGZQUNNLCnQnWQYfZl+5HCGX27CScvj+LIEfkMA+l/AWOmva0TMuU9iBzexwI3pJsou2vPnIG4nDtoU3kDiseX8QjaMOcJHEazJgaxzw/nkyL0VdkHZCobWzOKpc2qBxUihqe0ltqUJxisc081RjQTZtut218NQrc8hL45920Kn/JhosrF0Uma9cjmM3vaV4mTerXb6bNlJXtSMtVo3yWoErYXkImEtDwqmFJyiM9H8FcYL+Ds6fq+eXEG+OfJEcvhjH0Tj+/HKw9269ji32iD1hIxaz12yPfWIHbMIE+x08DobBKPgTDsLn4faqNAzWmYfsP4Sv/gIWTboB</latexit>
maxz1,...,zn
nX
i=1
E[Yi(z) | Ai = ai, Xi = xi]
s.t.,nX
i=1
zi B
Gia0 ⌧ 8a0 2 A, i 2 1, . . . , n,<latexit sha1_base64="wE8PNAxqRV0xhraKiAlRilriD8Q=">AAAC23icbVJdb9MwFHXCx0b5WIFHXq6oYEOKqgQhDVFN2oYQPA6JbkV1sG5cZ7PmOFnsoLUhvPAAQrzyx3jjb/ALcNpKQMeVLB2fc+71vbaTQkljw/Cn51+6fOXq2vq1zvUbN29tdG/fOTR5VXIx5LnKy1GCRiipxdBKq8SoKAVmiRJHyenzVj96L0ojc/3GTgsRZ3isZSo5Wkex7i+a4TmrZywKgKpJbk0AM6YboKbKWC13ouadfkgH4Hz2JEngxfgtk1uLXQqzRxQ+UNhjEnYAmQxgNIfnTMauHiZC1eLsWV7YhlLaMX3bd+cMVovPXBJV4gz2gepcV1kiSnD+VnvpjLjZLHRqsWrzB21HaV6iUoCbQKVeNMhRwV4Acs5EwXIiHfwp2+mwbi/sh/OAiyBagh5ZxgHr/qCTnFeZ0JYrNGYchYWNayyt5Eo0HVoZUSA/xWMxdlBjJkxcz9+mgQeOmYBr1S1tYc7+nVFjZsw0S5yzHcCsai35P21c2fRpXEtdVFZovjgorRTYHNqHhoksBbdq6gDyUrpegZ9gidy679BeQrQ68kVw+Lgfhf3o9ZPe7v7yOtbJPXKfbJGIbJNd8ooc kCHh3sj76H32vvix/8n/6n9bWH1vmXOX/BP+998Ayt1Q</latexit><latexit sha1_base64="wE8PNAxqRV0xhraKiAlRilriD8Q=">AAAC23icbVJdb9MwFHXCx0b5WIFHXq6oYEOKqgQhDVFN2oYQPA6JbkV1sG5cZ7PmOFnsoLUhvPAAQrzyx3jjb/ALcNpKQMeVLB2fc+71vbaTQkljw/Cn51+6fOXq2vq1zvUbN29tdG/fOTR5VXIx5LnKy1GCRiipxdBKq8SoKAVmiRJHyenzVj96L0ojc/3GTgsRZ3isZSo5Wkex7i+a4TmrZywKgKpJbk0AM6YboKbKWC13ouadfkgH4Hz2JEngxfgtk1uLXQqzRxQ+UNhjEnYAmQxgNIfnTMauHiZC1eLsWV7YhlLaMX3bd+cMVovPXBJV4gz2gepcV1kiSnD+VnvpjLjZLHRqsWrzB21HaV6iUoCbQKVeNMhRwV4Acs5EwXIiHfwp2+mwbi/sh/OAiyBagh5ZxgHr/qCTnFeZ0JYrNGYchYWNayyt5Eo0HVoZUSA/xWMxdlBjJkxcz9+mgQeOmYBr1S1tYc7+nVFjZsw0S5yzHcCsai35P21c2fRpXEtdVFZovjgorRTYHNqHhoksBbdq6gDyUrpegZ9gidy679BeQrQ68kVw+Lgfhf3o9ZPe7v7yOtbJPXKfbJGIbJNd8ooc kCHh3sj76H32vvix/8n/6n9bWH1vmXOX/BP+998Ayt1Q</latexit><latexit sha1_base64="wE8PNAxqRV0xhraKiAlRilriD8Q=">AAAC23icbVJdb9MwFHXCx0b5WIFHXq6oYEOKqgQhDVFN2oYQPA6JbkV1sG5cZ7PmOFnsoLUhvPAAQrzyx3jjb/ALcNpKQMeVLB2fc+71vbaTQkljw/Cn51+6fOXq2vq1zvUbN29tdG/fOTR5VXIx5LnKy1GCRiipxdBKq8SoKAVmiRJHyenzVj96L0ojc/3GTgsRZ3isZSo5Wkex7i+a4TmrZywKgKpJbk0AM6YboKbKWC13ouadfkgH4Hz2JEngxfgtk1uLXQqzRxQ+UNhjEnYAmQxgNIfnTMauHiZC1eLsWV7YhlLaMX3bd+cMVovPXBJV4gz2gepcV1kiSnD+VnvpjLjZLHRqsWrzB21HaV6iUoCbQKVeNMhRwV4Acs5EwXIiHfwp2+mwbi/sh/OAiyBagh5ZxgHr/qCTnFeZ0JYrNGYchYWNayyt5Eo0HVoZUSA/xWMxdlBjJkxcz9+mgQeOmYBr1S1tYc7+nVFjZsw0S5yzHcCsai35P21c2fRpXEtdVFZovjgorRTYHNqHhoksBbdq6gDyUrpegZ9gidy679BeQrQ68kVw+Lgfhf3o9ZPe7v7yOtbJPXKfbJGIbJNd8ooc kCHh3sj76H32vvix/8n/6n9bWH1vmXOX/BP+998Ayt1Q</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="Z/0GUlhD2yyTXjoOAb1lGkJlzSA=">AAAC0HicbVLLbtQwFHXCq4QCA1s2V1TQIkWjpBsQo0oUhGBZJKYdNA7WjcdprTpOGjuoMyFsWIAQX8eO3+ALcCYjAVOuZOn4nPu201JJY6Pop+dfunzl6rWN68GNzZu3bg/ubB6aoq64GPNCFdUkRSOU1GJspVViUlYC81SJo/T0RacffRCVkYV+a+elSHI81jKTHK2j2OAXzfGcNQsWh0DVrLAmhAXTLVBT56yRe3H7Xj+kI3B+9iRN4eX0HZM7/S2DxSMKHynsMwl7gEyGMFnCcyYTlw9ToRpx9rQobUspDczQDl2d0XryhQuiSpzBc6C60HWeigqcf6e9co643fY6tVh38aOuo6yoUCnAbaBS9w1yVLAfglwycbiaSId/0gYBG2xFw2hpcBHEK7BFVnbABj/orOB1LrTlCo2ZxlFpkwYrK7kSbUBrI0rkp3gspg5qzIVJmuXbtPDAMTNwrbqjLSzZvyMazI2Z56nz7AYw61pH/k+b1jZ7kjRSl7UVmveFslqBLaB7aJjJSnCr5g4gr6TrFfgJVsit+w7dEuL1kS+Cw91hHA3jNxHZIPfIfbJDYvKYPCOvyQEZE+5NvE/eF++r n/if/W/9unxvtbe75B/zv/8GcCncCQ==</latexit><latexit sha1_base64="Z/0GUlhD2yyTXjoOAb1lGkJlzSA=">AAAC0HicbVLLbtQwFHXCq4QCA1s2V1TQIkWjpBsQo0oUhGBZJKYdNA7WjcdprTpOGjuoMyFsWIAQX8eO3+ALcCYjAVOuZOn4nPu201JJY6Pop+dfunzl6rWN68GNzZu3bg/ubB6aoq64GPNCFdUkRSOU1GJspVViUlYC81SJo/T0RacffRCVkYV+a+elSHI81jKTHK2j2OAXzfGcNQsWh0DVrLAmhAXTLVBT56yRe3H7Xj+kI3B+9iRN4eX0HZM7/S2DxSMKHynsMwl7gEyGMFnCcyYTlw9ToRpx9rQobUspDczQDl2d0XryhQuiSpzBc6C60HWeigqcf6e9co643fY6tVh38aOuo6yoUCnAbaBS9w1yVLAfglwycbiaSId/0gYBG2xFw2hpcBHEK7BFVnbABj/orOB1LrTlCo2ZxlFpkwYrK7kSbUBrI0rkp3gspg5qzIVJmuXbtPDAMTNwrbqjLSzZvyMazI2Z56nz7AYw61pH/k+b1jZ7kjRSl7UVmveFslqBLaB7aJjJSnCr5g4gr6TrFfgJVsit+w7dEuL1kS+Cw91hHA3jNxHZIPfIfbJDYvKYPCOvyQEZE+5NvE/eF++r n/if/W/9unxvtbe75B/zv/8GcCncCQ==</latexit><latexit sha1_base64="0TQT7/ceH+4OXOofio+LZ2mzcLc=">AAAC23icbVJNb9QwEHXCVwlfCxy5jFhBixStEi4gVpXaIgTHIrHtonWwJl5va9Vx0thB3Q3hwgGEuPLHuPE3+AU4m5WALSNZen7vzXjGdlooaWwU/fT8CxcvXb6ycTW4dv3GzVu923cOTF6VXIx4rvJynKIRSmoxstIqMS5KgVmqxGF68rzVD9+L0shcv7HzQiQZHmk5kxyto1jvF83wjNULFodA1TS3JoQF0w1QU2Wslttx804/pENwPnucpvBi8pbJrW43g8UjCh8o7DIJ24BMhjBewjMmE1cPU6FqcfosL2xDKQ3MwA7cOcP14guXRJU4hT2gOtdVlooSnL/VXjojbjadTi1Wbf6w7WiWl6gU4CZQqbsGOSrYDUEumThcTaTDP2WDgPX60SBaBpwH8Qr0ySr2We8Hnea8yoS2XKExkzgqbFJjaSVXogloZUSB/ASPxMRBjZkwSb18mwYeOGYKrlW3tIUl+3dGjZkx8yx1znYAs6615P+0SWVnT5Na6qKyQvPuoFmlwObQPjRMZSm4VXMHkJfS9Qr8GEvk1n2H9hLi9ZHPg4PHgzgaxK+j/s7e6jo2yD1yn2yRmDwhO+QV2Scj wr2x99H77H3xE/+T/9X/1ll9b5Vzl/wT/vff/3vdTA==</latexit><latexit sha1_base64="wE8PNAxqRV0xhraKiAlRilriD8Q=">AAAC23icbVJdb9MwFHXCx0b5WIFHXq6oYEOKqgQhDVFN2oYQPA6JbkV1sG5cZ7PmOFnsoLUhvPAAQrzyx3jjb/ALcNpKQMeVLB2fc+71vbaTQkljw/Cn51+6fOXq2vq1zvUbN29tdG/fOTR5VXIx5LnKy1GCRiipxdBKq8SoKAVmiRJHyenzVj96L0ojc/3GTgsRZ3isZSo5Wkex7i+a4TmrZywKgKpJbk0AM6YboKbKWC13ouadfkgH4Hz2JEngxfgtk1uLXQqzRxQ+UNhjEnYAmQxgNIfnTMauHiZC1eLsWV7YhlLaMX3bd+cMVovPXBJV4gz2gepcV1kiSnD+VnvpjLjZLHRqsWrzB21HaV6iUoCbQKVeNMhRwV4Acs5EwXIiHfwp2+mwbi/sh/OAiyBagh5ZxgHr/qCTnFeZ0JYrNGYchYWNayyt5Eo0HVoZUSA/xWMxdlBjJkxcz9+mgQeOmYBr1S1tYc7+nVFjZsw0S5yzHcCsai35P21c2fRpXEtdVFZovjgorRTYHNqHhoksBbdq6gDyUrpegZ9gidy679BeQrQ68kVw+Lgfhf3o9ZPe7v7yOtbJPXKfbJGIbJNd8ooc kCHh3sj76H32vvix/8n/6n9bWH1vmXOX/BP+998Ayt1Q</latexit><latexit sha1_base64="wE8PNAxqRV0xhraKiAlRilriD8Q=">AAAC23icbVJdb9MwFHXCx0b5WIFHXq6oYEOKqgQhDVFN2oYQPA6JbkV1sG5cZ7PmOFnsoLUhvPAAQrzyx3jjb/ALcNpKQMeVLB2fc+71vbaTQkljw/Cn51+6fOXq2vq1zvUbN29tdG/fOTR5VXIx5LnKy1GCRiipxdBKq8SoKAVmiRJHyenzVj96L0ojc/3GTgsRZ3isZSo5Wkex7i+a4TmrZywKgKpJbk0AM6YboKbKWC13ouadfkgH4Hz2JEngxfgtk1uLXQqzRxQ+UNhjEnYAmQxgNIfnTMauHiZC1eLsWV7YhlLaMX3bd+cMVovPXBJV4gz2gepcV1kiSnD+VnvpjLjZLHRqsWrzB21HaV6iUoCbQKVeNMhRwV4Acs5EwXIiHfwp2+mwbi/sh/OAiyBagh5ZxgHr/qCTnFeZ0JYrNGYchYWNayyt5Eo0HVoZUSA/xWMxdlBjJkxcz9+mgQeOmYBr1S1tYc7+nVFjZsw0S5yzHcCsai35P21c2fRpXEtdVFZovjgorRTYHNqHhoksBbdq6gDyUrpegZ9gidy679BeQrQ68kVw+Lgfhf3o9ZPe7v7yOtbJPXKfbJGIbJNd8ooc kCHh3sj76H32vvix/8n/6n9bWH1vmXOX/BP+998Ayt1Q</latexit><latexit sha1_base64="wE8PNAxqRV0xhraKiAlRilriD8Q=">AAAC23icbVJdb9MwFHXCx0b5WIFHXq6oYEOKqgQhDVFN2oYQPA6JbkV1sG5cZ7PmOFnsoLUhvPAAQrzyx3jjb/ALcNpKQMeVLB2fc+71vbaTQkljw/Cn51+6fOXq2vq1zvUbN29tdG/fOTR5VXIx5LnKy1GCRiipxdBKq8SoKAVmiRJHyenzVj96L0ojc/3GTgsRZ3isZSo5Wkex7i+a4TmrZywKgKpJbk0AM6YboKbKWC13ouadfkgH4Hz2JEngxfgtk1uLXQqzRxQ+UNhjEnYAmQxgNIfnTMauHiZC1eLsWV7YhlLaMX3bd+cMVovPXBJV4gz2gepcV1kiSnD+VnvpjLjZLHRqsWrzB21HaV6iUoCbQKVeNMhRwV4Acs5EwXIiHfwp2+mwbi/sh/OAiyBagh5ZxgHr/qCTnFeZ0JYrNGYchYWNayyt5Eo0HVoZUSA/xWMxdlBjJkxcz9+mgQeOmYBr1S1tYc7+nVFjZsw0S5yzHcCsai35P21c2fRpXEtdVFZovjgorRTYHNqHhoksBbdq6gDyUrpegZ9gidy679BeQrQ68kVw+Lgfhf3o9ZPe7v7yOtbJPXKfbJGIbJNd8ooc kCHh3sj76H32vvix/8n/6n9bWH1vmXOX/BP+998Ayt1Q</latexit><latexit sha1_base64="wE8PNAxqRV0xhraKiAlRilriD8Q=">AAAC23icbVJdb9MwFHXCx0b5WIFHXq6oYEOKqgQhDVFN2oYQPA6JbkV1sG5cZ7PmOFnsoLUhvPAAQrzyx3jjb/ALcNpKQMeVLB2fc+71vbaTQkljw/Cn51+6fOXq2vq1zvUbN29tdG/fOTR5VXIx5LnKy1GCRiipxdBKq8SoKAVmiRJHyenzVj96L0ojc/3GTgsRZ3isZSo5Wkex7i+a4TmrZywKgKpJbk0AM6YboKbKWC13ouadfkgH4Hz2JEngxfgtk1uLXQqzRxQ+UNhjEnYAmQxgNIfnTMauHiZC1eLsWV7YhlLaMX3bd+cMVovPXBJV4gz2gepcV1kiSnD+VnvpjLjZLHRqsWrzB21HaV6iUoCbQKVeNMhRwV4Acs5EwXIiHfwp2+mwbi/sh/OAiyBagh5ZxgHr/qCTnFeZ0JYrNGYchYWNayyt5Eo0HVoZUSA/xWMxdlBjJkxcz9+mgQeOmYBr1S1tYc7+nVFjZsw0S5yzHcCsai35P21c2fRpXEtdVFZovjgorRTYHNqHhoksBbdq6gDyUrpegZ9gidy679BeQrQ68kVw+Lgfhf3o9ZPe7v7yOtbJPXKfbJGIbJNd8ooc kCHh3sj76H32vvix/8n/6n9bWH1vmXOX/BP+998Ayt1Q</latexit><latexit sha1_base64="wE8PNAxqRV0xhraKiAlRilriD8Q=">AAAC23icbVJdb9MwFHXCx0b5WIFHXq6oYEOKqgQhDVFN2oYQPA6JbkV1sG5cZ7PmOFnsoLUhvPAAQrzyx3jjb/ALcNpKQMeVLB2fc+71vbaTQkljw/Cn51+6fOXq2vq1zvUbN29tdG/fOTR5VXIx5LnKy1GCRiipxdBKq8SoKAVmiRJHyenzVj96L0ojc/3GTgsRZ3isZSo5Wkex7i+a4TmrZywKgKpJbk0AM6YboKbKWC13ouadfkgH4Hz2JEngxfgtk1uLXQqzRxQ+UNhjEnYAmQxgNIfnTMauHiZC1eLsWV7YhlLaMX3bd+cMVovPXBJV4gz2gepcV1kiSnD+VnvpjLjZLHRqsWrzB21HaV6iUoCbQKVeNMhRwV4Acs5EwXIiHfwp2+mwbi/sh/OAiyBagh5ZxgHr/qCTnFeZ0JYrNGYchYWNayyt5Eo0HVoZUSA/xWMxdlBjJkxcz9+mgQeOmYBr1S1tYc7+nVFjZsw0S5yzHcCsai35P21c2fRpXEtdVFZovjgorRTYHNqHhoksBbdq6gDyUrpegZ9gidy679BeQrQ68kVw+Lgfhf3o9ZPe7v7yOtbJPXKfbJGIbJNd8ooc kCHh3sj76H32vvix/8n/6n9bWH1vmXOX/BP+998Ayt1Q</latexit><latexit sha1_base64="wE8PNAxqRV0xhraKiAlRilriD8Q=">AAAC23icbVJdb9MwFHXCx0b5WIFHXq6oYEOKqgQhDVFN2oYQPA6JbkV1sG5cZ7PmOFnsoLUhvPAAQrzyx3jjb/ALcNpKQMeVLB2fc+71vbaTQkljw/Cn51+6fOXq2vq1zvUbN29tdG/fOTR5VXIx5LnKy1GCRiipxdBKq8SoKAVmiRJHyenzVj96L0ojc/3GTgsRZ3isZSo5Wkex7i+a4TmrZywKgKpJbk0AM6YboKbKWC13ouadfkgH4Hz2JEngxfgtk1uLXQqzRxQ+UNhjEnYAmQxgNIfnTMauHiZC1eLsWV7YhlLaMX3bd+cMVovPXBJV4gz2gepcV1kiSnD+VnvpjLjZLHRqsWrzB21HaV6iUoCbQKVeNMhRwV4Acs5EwXIiHfwp2+mwbi/sh/OAiyBagh5ZxgHr/qCTnFeZ0JYrNGYchYWNayyt5Eo0HVoZUSA/xWMxdlBjJkxcz9+mgQeOmYBr1S1tYc7+nVFjZsw0S5yzHcCsai35P21c2fRpXEtdVFZovjgorRTYHNqHhoksBbdq6gDyUrpegZ9gidy679BeQrQ68kVw+Lgfhf3o9ZPe7v7yOtbJPXKfbJGIbJNd8ooc kCHh3sj76H32vvix/8n/6n9bWH1vmXOX/BP+998Ayt1Q</latexit>
Mainfamilyofconstraints:
(asopposedto|E[Yi(ai, z) | Ai = ai, Xi = xi]� E[Yi(a
0, z) | Ai = ai, Xi = xi]| < ⌧<latexit sha1_base64="xWNizRs8+QU/6klSOqWWnZ6CX5Y=">AAACZnicbVFLS8NAGNzEd3zFinjw8mERFbQkIuhBwQeCxwpWK00Im+2mXbp5sLsRa9s/6c2zF3+GSdqDpg4sDDPfsPvN+glnUlnWp6bPzM7NLywuGcsrq2vr5kblScapILRBYh6Lpo8l5SyiDcUUp81EUBz6nD77vdvcf36lQrI4elT9hLoh7kQsYASrTPLM0dAJser6Pty1Xjx2gD12BGMpgPdDB4bgwLXH4BIKq1nQN4+5cAxGKbtfiv6TLKJDuABH4dTwzKpVswrANLEnpIomqHvmh9OOSRrSSBGOpWzZVqLcARaKEU5HhpNKmmDSwx3aymiEQyrdQVHTCPYypQ1BLLITKSjU34kBDqXsh342mW8hy14u/ue1UhWcuwMWJamiERlfFKQcVAx559BmghLF+xnBRLDsrUC6WGCisp/JS7DLK0+Tp5OabdXsh9Pq1c2kjkW0g3bRAbLRGbpC96iOGoigL21Jq2ib2re+pm/p2+NRXZtkNtEf6PADY1+w0g==</latexit><latexit sha1_base64="xWNizRs8+QU/6klSOqWWnZ6CX5Y=">AAACZnicbVFLS8NAGNzEd3zFinjw8mERFbQkIuhBwQeCxwpWK00Im+2mXbp5sLsRa9s/6c2zF3+GSdqDpg4sDDPfsPvN+glnUlnWp6bPzM7NLywuGcsrq2vr5kblScapILRBYh6Lpo8l5SyiDcUUp81EUBz6nD77vdvcf36lQrI4elT9hLoh7kQsYASrTPLM0dAJser6Pty1Xjx2gD12BGMpgPdDB4bgwLXH4BIKq1nQN4+5cAxGKbtfiv6TLKJDuABH4dTwzKpVswrANLEnpIomqHvmh9OOSRrSSBGOpWzZVqLcARaKEU5HhpNKmmDSwx3aymiEQyrdQVHTCPYypQ1BLLITKSjU34kBDqXsh342mW8hy14u/ue1UhWcuwMWJamiERlfFKQcVAx559BmghLF+xnBRLDsrUC6WGCisp/JS7DLK0+Tp5OabdXsh9Pq1c2kjkW0g3bRAbLRGbpC96iOGoigL21Jq2ib2re+pm/p2+NRXZtkNtEf6PADY1+w0g==</latexit><latexit sha1_base64="xWNizRs8+QU/6klSOqWWnZ6CX5Y=">AAACZnicbVFLS8NAGNzEd3zFinjw8mERFbQkIuhBwQeCxwpWK00Im+2mXbp5sLsRa9s/6c2zF3+GSdqDpg4sDDPfsPvN+glnUlnWp6bPzM7NLywuGcsrq2vr5kblScapILRBYh6Lpo8l5SyiDcUUp81EUBz6nD77vdvcf36lQrI4elT9hLoh7kQsYASrTPLM0dAJser6Pty1Xjx2gD12BGMpgPdDB4bgwLXH4BIKq1nQN4+5cAxGKbtfiv6TLKJDuABH4dTwzKpVswrANLEnpIomqHvmh9OOSRrSSBGOpWzZVqLcARaKEU5HhpNKmmDSwx3aymiEQyrdQVHTCPYypQ1BLLITKSjU34kBDqXsh342mW8hy14u/ue1UhWcuwMWJamiERlfFKQcVAx559BmghLF+xnBRLDsrUC6WGCisp/JS7DLK0+Tp5OabdXsh9Pq1c2kjkW0g3bRAbLRGbpC96iOGoigL21Jq2ib2re+pm/p2+NRXZtkNtEf6PADY1+w0g==</latexit><latexit sha1_base64="xWNizRs8+QU/6klSOqWWnZ6CX5Y=">AAACZnicbVFLS8NAGNzEd3zFinjw8mERFbQkIuhBwQeCxwpWK00Im+2mXbp5sLsRa9s/6c2zF3+GSdqDpg4sDDPfsPvN+glnUlnWp6bPzM7NLywuGcsrq2vr5kblScapILRBYh6Lpo8l5SyiDcUUp81EUBz6nD77vdvcf36lQrI4elT9hLoh7kQsYASrTPLM0dAJser6Pty1Xjx2gD12BGMpgPdDB4bgwLXH4BIKq1nQN4+5cAxGKbtfiv6TLKJDuABH4dTwzKpVswrANLEnpIomqHvmh9OOSRrSSBGOpWzZVqLcARaKEU5HhpNKmmDSwx3aymiEQyrdQVHTCPYypQ1BLLITKSjU34kBDqXsh342mW8hy14u/ue1UhWcuwMWJamiERlfFKQcVAx559BmghLF+xnBRLDsrUC6WGCisp/JS7DLK0+Tp5OabdXsh9Pq1c2kjkW0g3bRAbLRGbpC96iOGoigL21Jq2ib2re+pm/p2+NRXZtkNtEf6PADY1+w0g==</latexit>
)
IntuitiveToyExample
• ProtectedattributeAissuchthatAin{b,w},Xissomequantitativemeasureofprofessionalcompetence,andYisameasureofwealthin5years’time.
• Zi=1meansindividualigetsasubsidytomovetoaneighborhoodwithbettertransportlinks.
IntuitiveToyExample
• Supposestructuralequationis
• Soiftherearetwoindividuals,oneoftypewandoneoftypeb,andZ1+Z2=1.
• Withoutthefairnessconstraint,typewgetsthesubsidyeveniftypebhasupto50moreunitsofprofessionalability!
Yi = Xi + 100Zi + 50Zi ⇥ I(Ai = w) + Ui<latexit sha1_base64="G6O0fZJi/VPIziL8S528M+CDGfg=">AAACJXicbVBNS8NAEN3Ur1q/oh69LBahIpREFD0oVL3orYJpq00Im+22Xbr5YHejlNA/48W/4sWDRQRP/hU3aQ7aOrA7jzdvmJnnRYwKaRhfWmFufmFxqbhcWlldW9/QN7caIow5JhYOWchbHhKE0YBYkkpGWhEnyPcYaXqDq7TefCRc0DC4k8OIOD7qBbRLMZKKcvWze5fCc9hS/wE0DeMhA8dZtiX1iYC2j2Tf85KbUeUiEz/tK4nl0pKrl42qkQWcBWYOyiCPuquP7U6IY58EEjMkRNs0IukkiEuKGRmV7FiQCOEB6pG2ggFS850ku3IE9xTTgd2QqxdImLG/OxLkCzH0PaVMNxbTtZT8r9aOZffUSWgQxZIEeDKoGzMoQ5haBjuUEyzZUAGEOVW7QtxHHGGpjE1NMKdPngWNw6ppVM3bo3LtMrejCHbALqgAE5yAGrgGdWABDJ7BK3gHY+1Fe9M+tM+JtKDlPdvgT2jfP7P3oHo=</latexit><latexit sha1_base64="G6O0fZJi/VPIziL8S528M+CDGfg=">AAACJXicbVBNS8NAEN3Ur1q/oh69LBahIpREFD0oVL3orYJpq00Im+22Xbr5YHejlNA/48W/4sWDRQRP/hU3aQ7aOrA7jzdvmJnnRYwKaRhfWmFufmFxqbhcWlldW9/QN7caIow5JhYOWchbHhKE0YBYkkpGWhEnyPcYaXqDq7TefCRc0DC4k8OIOD7qBbRLMZKKcvWze5fCc9hS/wE0DeMhA8dZtiX1iYC2j2Tf85KbUeUiEz/tK4nl0pKrl42qkQWcBWYOyiCPuquP7U6IY58EEjMkRNs0IukkiEuKGRmV7FiQCOEB6pG2ggFS850ku3IE9xTTgd2QqxdImLG/OxLkCzH0PaVMNxbTtZT8r9aOZffUSWgQxZIEeDKoGzMoQ5haBjuUEyzZUAGEOVW7QtxHHGGpjE1NMKdPngWNw6ppVM3bo3LtMrejCHbALqgAE5yAGrgGdWABDJ7BK3gHY+1Fe9M+tM+JtKDlPdvgT2jfP7P3oHo=</latexit><latexit sha1_base64="G6O0fZJi/VPIziL8S528M+CDGfg=">AAACJXicbVBNS8NAEN3Ur1q/oh69LBahIpREFD0oVL3orYJpq00Im+22Xbr5YHejlNA/48W/4sWDRQRP/hU3aQ7aOrA7jzdvmJnnRYwKaRhfWmFufmFxqbhcWlldW9/QN7caIow5JhYOWchbHhKE0YBYkkpGWhEnyPcYaXqDq7TefCRc0DC4k8OIOD7qBbRLMZKKcvWze5fCc9hS/wE0DeMhA8dZtiX1iYC2j2Tf85KbUeUiEz/tK4nl0pKrl42qkQWcBWYOyiCPuquP7U6IY58EEjMkRNs0IukkiEuKGRmV7FiQCOEB6pG2ggFS850ku3IE9xTTgd2QqxdImLG/OxLkCzH0PaVMNxbTtZT8r9aOZffUSWgQxZIEeDKoGzMoQ5haBjuUEyzZUAGEOVW7QtxHHGGpjE1NMKdPngWNw6ppVM3bo3LtMrejCHbALqgAE5yAGrgGdWABDJ7BK3gHY+1Fe9M+tM+JtKDlPdvgT2jfP7P3oHo=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="5cGLW4Oc6HfWRkuTbeluh14wMOc=">AAACGnicbVDLSgMxFL1TX7VWHd26CRahIpSMILpQUNzoroJ9aFuGTJq2oZkHSUYpQ3/Gjb/ixoVFBP/GzLQLbT2Q5HDuCffe40WCK43xt5VbWl5ZXcuvFzaKm1vb9k6xrsJYUlajoQhl0yOKCR6wmuZasGYkGfE9wRre8DqtN56YVDwM7vUoYh2f9APe45RoI7n2+YPL0QVqmvsIORg/ZuQke9ua+0yhtk/0wPOS23H5KjM/HxpLzeUF1y7hCs6AFokzIyWYoerak3Y3pLHPAk0FUarl4Eh3EiI1p4KNC+1YsYjQIemzlqEBMf07SbblGB0YpYt6oTQn0ChTf/9IiK/UyPeMM51YzddS8b9aK9a9s07CgyjWLKDTRr1YIB2iNDLU5ZJRLUaGECq5mRXRAZGEahNsGoIzv/IiqR9XHFxx7jDkYQ/2oQwOnMIl3EAVakDhBd7gAybWq/VufU7jylmz3HbhD6yvH9POnuo=</latexit><latexit sha1_base64="5cGLW4Oc6HfWRkuTbeluh14wMOc=">AAACGnicbVDLSgMxFL1TX7VWHd26CRahIpSMILpQUNzoroJ9aFuGTJq2oZkHSUYpQ3/Gjb/ixoVFBP/GzLQLbT2Q5HDuCffe40WCK43xt5VbWl5ZXcuvFzaKm1vb9k6xrsJYUlajoQhl0yOKCR6wmuZasGYkGfE9wRre8DqtN56YVDwM7vUoYh2f9APe45RoI7n2+YPL0QVqmvsIORg/ZuQke9ua+0yhtk/0wPOS23H5KjM/HxpLzeUF1y7hCs6AFokzIyWYoerak3Y3pLHPAk0FUarl4Eh3EiI1p4KNC+1YsYjQIemzlqEBMf07SbblGB0YpYt6oTQn0ChTf/9IiK/UyPeMM51YzddS8b9aK9a9s07CgyjWLKDTRr1YIB2iNDLU5ZJRLUaGECq5mRXRAZGEahNsGoIzv/IiqR9XHFxx7jDkYQ/2oQwOnMIl3EAVakDhBd7gAybWq/VufU7jylmz3HbhD6yvH9POnuo=</latexit><latexit sha1_base64="6VDkPHhJM2K4Gyz6mkhn0+38QmE=">AAACJXicbVDLSgMxFM34rPVVdekmWISKUDKC6EKh6kZ3FexDO8OQSTNtaOZBklHK0J9x46+4cWERwZW/YmY6C229kNzDuedy7z1uxJlUCH0Zc/MLi0vLhZXi6tr6xmZpa7spw1gQ2iAhD0XbxZJyFtCGYorTdiQo9l1OW+7gKq23HqmQLAzu1DCito97AfMYwUpTTuns3mHwHLb1fwhNhB4ycJxlSzGfSmj5WPVdN7kZVS4y8dOBljQcVnRKZVRFWcBZYOagDPKoO6Wx1Q1J7NNAEY6l7JgoUnaChWKE01HRiiWNMBngHu1oGGA9306yK0dwXzNd6IVCv0DBjP3dkWBfyqHvamW6sZyupeR/tU6svFM7YUEUKxqQySAv5lCFMLUMdpmgRPGhBpgIpneFpI8FJkobm5pgTp88C5pHVRNVzVtUrl3mdhTALtgDFWCCE1AD16AOGoCAZ/AK3sHYeDHejA/jcyKdM/KeHfAnjO8fsregdg==</latexit><latexit sha1_base64="G6O0fZJi/VPIziL8S528M+CDGfg=">AAACJXicbVBNS8NAEN3Ur1q/oh69LBahIpREFD0oVL3orYJpq00Im+22Xbr5YHejlNA/48W/4sWDRQRP/hU3aQ7aOrA7jzdvmJnnRYwKaRhfWmFufmFxqbhcWlldW9/QN7caIow5JhYOWchbHhKE0YBYkkpGWhEnyPcYaXqDq7TefCRc0DC4k8OIOD7qBbRLMZKKcvWze5fCc9hS/wE0DeMhA8dZtiX1iYC2j2Tf85KbUeUiEz/tK4nl0pKrl42qkQWcBWYOyiCPuquP7U6IY58EEjMkRNs0IukkiEuKGRmV7FiQCOEB6pG2ggFS850ku3IE9xTTgd2QqxdImLG/OxLkCzH0PaVMNxbTtZT8r9aOZffUSWgQxZIEeDKoGzMoQ5haBjuUEyzZUAGEOVW7QtxHHGGpjE1NMKdPngWNw6ppVM3bo3LtMrejCHbALqgAE5yAGrgGdWABDJ7BK3gHY+1Fe9M+tM+JtKDlPdvgT2jfP7P3oHo=</latexit><latexit sha1_base64="G6O0fZJi/VPIziL8S528M+CDGfg=">AAACJXicbVBNS8NAEN3Ur1q/oh69LBahIpREFD0oVL3orYJpq00Im+22Xbr5YHejlNA/48W/4sWDRQRP/hU3aQ7aOrA7jzdvmJnnRYwKaRhfWmFufmFxqbhcWlldW9/QN7caIow5JhYOWchbHhKE0YBYkkpGWhEnyPcYaXqDq7TefCRc0DC4k8OIOD7qBbRLMZKKcvWze5fCc9hS/wE0DeMhA8dZtiX1iYC2j2Tf85KbUeUiEz/tK4nl0pKrl42qkQWcBWYOyiCPuquP7U6IY58EEjMkRNs0IukkiEuKGRmV7FiQCOEB6pG2ggFS850ku3IE9xTTgd2QqxdImLG/OxLkCzH0PaVMNxbTtZT8r9aOZffUSWgQxZIEeDKoGzMoQ5haBjuUEyzZUAGEOVW7QtxHHGGpjE1NMKdPngWNw6ppVM3bo3LtMrejCHbALqgAE5yAGrgGdWABDJ7BK3gHY+1Fe9M+tM+JtKDlPdvgT2jfP7P3oHo=</latexit><latexit sha1_base64="G6O0fZJi/VPIziL8S528M+CDGfg=">AAACJXicbVBNS8NAEN3Ur1q/oh69LBahIpREFD0oVL3orYJpq00Im+22Xbr5YHejlNA/48W/4sWDRQRP/hU3aQ7aOrA7jzdvmJnnRYwKaRhfWmFufmFxqbhcWlldW9/QN7caIow5JhYOWchbHhKE0YBYkkpGWhEnyPcYaXqDq7TefCRc0DC4k8OIOD7qBbRLMZKKcvWze5fCc9hS/wE0DeMhA8dZtiX1iYC2j2Tf85KbUeUiEz/tK4nl0pKrl42qkQWcBWYOyiCPuquP7U6IY58EEjMkRNs0IukkiEuKGRmV7FiQCOEB6pG2ggFS850ku3IE9xTTgd2QqxdImLG/OxLkCzH0PaVMNxbTtZT8r9aOZffUSWgQxZIEeDKoGzMoQ5haBjuUEyzZUAGEOVW7QtxHHGGpjE1NMKdPngWNw6ppVM3bo3LtMrejCHbALqgAE5yAGrgGdWABDJ7BK3gHY+1Fe9M+tM+JtKDlPdvgT2jfP7P3oHo=</latexit><latexit sha1_base64="G6O0fZJi/VPIziL8S528M+CDGfg=">AAACJXicbVBNS8NAEN3Ur1q/oh69LBahIpREFD0oVL3orYJpq00Im+22Xbr5YHejlNA/48W/4sWDRQRP/hU3aQ7aOrA7jzdvmJnnRYwKaRhfWmFufmFxqbhcWlldW9/QN7caIow5JhYOWchbHhKE0YBYkkpGWhEnyPcYaXqDq7TefCRc0DC4k8OIOD7qBbRLMZKKcvWze5fCc9hS/wE0DeMhA8dZtiX1iYC2j2Tf85KbUeUiEz/tK4nl0pKrl42qkQWcBWYOyiCPuquP7U6IY58EEjMkRNs0IukkiEuKGRmV7FiQCOEB6pG2ggFS850ku3IE9xTTgd2QqxdImLG/OxLkCzH0PaVMNxbTtZT8r9aOZffUSWgQxZIEeDKoGzMoQ5haBjuUEyzZUAGEOVW7QtxHHGGpjE1NMKdPngWNw6ppVM3bo3LtMrejCHbALqgAE5yAGrgGdWABDJ7BK3gHY+1Fe9M+tM+JtKDlPdvgT2jfP7P3oHo=</latexit><latexit sha1_base64="G6O0fZJi/VPIziL8S528M+CDGfg=">AAACJXicbVBNS8NAEN3Ur1q/oh69LBahIpREFD0oVL3orYJpq00Im+22Xbr5YHejlNA/48W/4sWDRQRP/hU3aQ7aOrA7jzdvmJnnRYwKaRhfWmFufmFxqbhcWlldW9/QN7caIow5JhYOWchbHhKE0YBYkkpGWhEnyPcYaXqDq7TefCRc0DC4k8OIOD7qBbRLMZKKcvWze5fCc9hS/wE0DeMhA8dZtiX1iYC2j2Tf85KbUeUiEz/tK4nl0pKrl42qkQWcBWYOyiCPuquP7U6IY58EEjMkRNs0IukkiEuKGRmV7FiQCOEB6pG2ggFS850ku3IE9xTTgd2QqxdImLG/OxLkCzH0PaVMNxbTtZT8r9aOZffUSWgQxZIEeDKoGzMoQ5haBjuUEyzZUAGEOVW7QtxHHGGpjE1NMKdPngWNw6ppVM3bo3LtMrejCHbALqgAE5yAGrgGdWABDJ7BK3gHY+1Fe9M+tM+JtKDlPdvgT2jfP7P3oHo=</latexit><latexit sha1_base64="G6O0fZJi/VPIziL8S528M+CDGfg=">AAACJXicbVBNS8NAEN3Ur1q/oh69LBahIpREFD0oVL3orYJpq00Im+22Xbr5YHejlNA/48W/4sWDRQRP/hU3aQ7aOrA7jzdvmJnnRYwKaRhfWmFufmFxqbhcWlldW9/QN7caIow5JhYOWchbHhKE0YBYkkpGWhEnyPcYaXqDq7TefCRc0DC4k8OIOD7qBbRLMZKKcvWze5fCc9hS/wE0DeMhA8dZtiX1iYC2j2Tf85KbUeUiEz/tK4nl0pKrl42qkQWcBWYOyiCPuquP7U6IY58EEjMkRNs0IukkiEuKGRmV7FiQCOEB6pG2ggFS850ku3IE9xTTgd2QqxdImLG/OxLkCzH0PaVMNxbTtZT8r9aOZffUSWgQxZIEeDKoGzMoQ5haBjuUEyzZUAGEOVW7QtxHHGGpjE1NMKdPngWNw6ppVM3bo3LtMrejCHbALqgAE5yAGrgGdWABDJ7BK3gHY+1Fe9M+tM+JtKDlPdvgT2jfP7P3oHo=</latexit>
Considerations• Theremightbenofeasiblesolutionifτissmallenough.
• Itmightbethecasethatthe“counterfactualgap”ineachconstraintremainsconstantregardlessofZ.
• Thesearefeaturesoftheintervention,notofthefairnessframework.Again,agoodinterventionisamatterofrealworlddesign,notofalgorithmdesign!
Illustration(PartiallySyntheticData)
• NYCPublicSchools:interventionZistoprovidecalculusclassesinschools.
• AttributeAiswhetherschoolhasawhitemajority.
• OutcomeYisproportionofstudentstakingtheSAT/ACT.
• Geographicalinterferenceisassumed.
Results
Conclusion
• Iproposethatcausalmodelingshouldbeakeycomponentoffairnessconsiderations.
• Fairnesshasmultiplefacets.Hereweconsideredpredictionandpolicy-makingunderinterference.
• Muchmoreisrelevant:selectionbias,dynamicprediction/treatmentsetc.
• Goodsoftwaredesigncouldhelpbuildingmassiveexperimentsintheinternet,forinstance.
References• M.Kusner,C.Russell,J.LoftusandR.Silva(2017).“Counterfactual
Fairness”.NIPS2017.https://papers.nips.cc/paper/6995-counterfactual-fairness
• C.Russell,M.Kusner,J.LoftusandR.Silva(2017).“WhenWorldsCollide:IntegratingDifferentCounterfactualAssumptionsinFairness”.NIPS2017.https://papers.nips.cc/paper/7220-when-worlds-collide-integrating-different-counterfactual-assumptions-in-fairness
• J.Loftus,C.Russell,M.KusnerandR.Silva(2018).“CausalReasoningforAlgorithmicFairness”.https://arxiv.org/abs/1805.05859
• M.Kusner,C.Russell,J.LoftusandR.Silva(2018).“CausalInterventionsforFairness”.TobeinarXivatanymoment.
ThankYou