Cultural Computational Agents

Cultural Negotiating AgentsBy: Elnaz NouriISI NL Seminar, June 6th 2014

Hello1Goal: Building virtual agents that can bargain and negotiate with people using natural language dialogue.

Human Interacting with a Virtual HumanThe Big Picture

How to contribute to computer scienceAI people that do core reasoningComputational component to your work: what features and algorithms did you use

2The Need to Understand Human Negotiation

In addition to challenges from a natural language perspective the following aspects need to be considered as well:

Emotions, Ethics, Framing, Social relationships, Motivated illusions, Culture, etc

Negotiation is a decision-making process in which people mutually decide how to allocate resources [Pruitt, 1983]. The objective behavioral outcomes clearly represent an important aspect of negotiation performance but social and psychological measures are also important in negotiation [Curhan et al., 2006].Negotiations provide rich test beds for analyzing social aspects of decision making because they are in essence an interpersonal strategic decision-making process that involve give-and-take interactions between two or more parties with potentially conicting objectives [Thompson, 2004]

Building off of work in behavioral decision theory (Tversky and Kahneman 1974;Kahneman and Tversky, 1979), a number of deviations from rationality have been identified thatcan be expected in negotiations. Researchers found, for example, that negotiators tend to beinappropriately affected by the positive or negative frame in which they view risks (Neale andBazerman 1985; Bazerman, Magliozzi, and Neale 1985), to anchor their numeric estimates innegotiations -on irrelevant information such as arbitrary numbers or manipulated listing prices(Tversky and Kahneman 1974; Northcraft and Neale 1987)---addressed as above, to relydisproportionately on readily available information at the expense of critical but less salientinformation (Neale 1984), and to be overconfident about the likelihood of attaining outcomesthat favor themselves (Bazerman and Neale 1982; Neale and Bazerman 1985).

Offering an alternative to the game-theoretic study of negotiation, which takes place in a worldof impeccably rational, supersmart people, Raiffa developed a decision-analytic approach tonegotiations, one that described how erring folks like you and me actually behave, rather thanhow we should behave if we were smarter, thought harder, were more consistent, were allknowing(Raiffa 1982: 21).

Several promising areas of research haveemerged in recent years, drawing from other disciplines and informing the field of negotiations,including work on the influence of ethics, emotions, intuition, and training.

An individual's decision and their assessment of the outcome of the decision have been shown to be interdependent on their social relationship with other people involved [Bault et al., 2008, Maccheroni et al., 2012]. A decider's goals and calculation of utility can be aected by social concerns for others and this might result in consideration of factors other than self interest by the decider when evaluating a choice.3People's cultural background has been shown to affect the way they reach and fulfill agreements in negotiation. [Haim,2012]

The Importance of Considering CultureUS Vice President Joe Biden & Japanese Prime Minister Shinzo AbeDecember 2013

The dissemination of technology across geographical and ethnic borders is opening up opportunities for computer agents to negotiate with people of diverse cultures and backgrounds. For example, electronic commerce (e.g., ebay), crowd-sourcing (e.g., Amazon Turk) and deal-of-the-day applications (e.g., Groupon) already involve computer agentsthat make decisions together with people from different countries. People's cultural background has been shown to be akey determinant of the way they make and keep their agreements with others [7]. It is thus important for agent de-signers to model how people from various cultures respond to different kinds of decision-making behavior employed byothers. To succeed in such settings computer agents need to adapt to the culture and particular behavior of the individual they interact with.Prior work has addressed some of the computational challenges arising in repeated negotiation between people andcomputer agents [6, 11]. However, additional challenges arise when designing agents that adapt to different cultures.First, agents need to adopt a separate strategy in each culture, requiring large amounts of data to be collected of people's play in the dierent cultures. Second, people's individual behavior within a culture displays wide variance, as people's strategies are inconsistent and prone to noise [7].

[6] Y. Gal and A. Pfeer. Modeling reciprocity in human bilateral negotiation. In AAAI'07, 2007.[7] M. J. Gelfand and S. Christakopoulou. Culture and negotiator cognition: Judgment accuracy and negotiation processes in individualistic and collectivistic cultures. Organizational Behavior and Human Decision Processes, 79(3):248{269, 1999.[11] S. Kraus, P. Hoz-Weiss, J. Wilkenfeld, D. Andersen, and A. Pate. Resolving crises through automated bilateral negotiations. Articial Intelligence, 172(1):1{18, 2008.4Important Factor in Negotiation:

Interpersonal strategic decision-making processes [Thompson, 2004] [Curhan, 2006]Involve resource allocation and potentially conflicting objectives [Pruitt, 1983] Exchange of propositions and responses

NegotiationsOur focus is on Modeling the Interpersonal Decision Making in Negotiation:

Social GoalsAssessment of the relationship Offers ResponsesStrategiesOutcome

5InputDialogue ManagerDecision MakingComponentSpeech RecognizerText-to-Speech SystemResponse GeneratorLanguage UnderstandingComponents in a Spoken Dialogue Systemfor NegotiationOutputMy focus: Simulate Decision Making in Negotiation

OffersResponsesAssessment of RelationshipWe focus on the decision making component of a negotiating agent6Whats a suitable Decision Making Model for cultural negotiating agent?

Whats a suitable Social Decision Making Model for cultural negotiating agent?

Social Goals in NegotiationNegotiator's goals determines how the negotiation unfoldsPeople care about the outcome of others when making decisions. [Loewenstein, 1989]

Sacrificing ones own interest to help loved ones or harm adversaries.Participants withdrawal from experiments if they perceive inequity in remuneration. [Schmitt, 1972]Negotiations collapse when one party tries to maximize opponents displeasure rather than his own satisfaction. [Seigel, 1960]Disputants concern not only with their own outcome but also with the outcome of the opponents [Pruitt, 1986]Examples:Anomalies in behavioral games: Dictator game [Bolton, 1998]Prisoners Dilemma [Rapoport, 1965]

Example 1: The Negotiation Example[Nouri et al, Interspeech 2013]41 dyadic sessions (15 competitive, 13 individualistic and 13 cooperative)Different Goals:Individualistic: goal maximize points for themselves [Vself]Cooperative: maximize the joint gain with the other side [Vjoint]Competitive: try to maximize points for yourself and prevent the other side from getting points [Vcompete]

Prediction Models for Negotiation Outcome and Goals[Nouri et al, Interspeech 2013]

Prediction AccuracyPrediction AccuracyFeatures:# of words spoken per speaker per turn# of turns taken # of negotiation issue related wordsSentiment(positive, negative) and subjectivity scores.mean and standard deviation of the following acoustic features: peak slope, Normalized amplitude quotient (NAQ), f0, voiced/unvoiced, energy, energy slope, spectral stationary amount of silence and speaking time per speaker # of offers, acceptances or rejections10-fold cross validation (SVM) classifier with the RBF kernelPredict OutcomePredict StrategySignificant differences in offers, responses and distribution of offersShow the differences in behaviour and raw data11

Example 2: The Ultimatum Game [Gth, 1982] 2-turn game over a certain amount of money:Expected Results According to Game Theoretic Models:

Offer the minimum amount possibleAccept any offer greater than zero

Offer about half of the moneyReject a high number of low offersFrequency of offers made by proposersAcceptance ratio by respondersObserved Results:Dictator Games: similar but responders only acceptProposer makes offerResponder Accepts : split accordinglyRejects : both get zeroBargaining processes are often modeled as ultimatum bargaining games. [Stahl, 1972]Self Interest (the agent's own utility) [Scott, 1972]Other Interest (the utility of another agent) [MacCrimmon and Messik 1976]Total Utility (sum of individual utilities of all participating agents)Average Utility (may not be derivable from Total Utility)Relative Utilities (viewed in several ways, such as self/total, self-other, self/other, self/average)Self/Total [MacCrimmon and Messik 1976] [Loewenstein, 1989]Self/Other [MacCrimmon and Messik 1976] [Lurie, 1987]Self/AverageSelf-Other [Griesinger & Livingston, 1973] [Lurie, 1987]Minimum Utility (lower bound for any participant) [Rawls' Theory of Justice, 1984]Uncertainty (variation among possible outcomes)A suitable model: The MARV Decision Making Model[Nouri and Traum. CMVC Workshop, Reykjavik, Iceland, 2011][Nouri, Georgila and Traum, Journal of AI and Society, 2014]

Considers a combination of different valuation functions for evaluation of the utility:

**** Remove the ones not used

Average Utility (may not be derivable from Total Utility when the number of participants is variable)[Scott, 1972] distiguishes between three motives underlying concern for other peoples outcomes: avarice, altruism, egalitarianism[MacCrimmon and Messik 1976] concern for others: self-interest, self-sacrifice, altruism, aggression, cooperation and competition[Griesinger & Livingston, 1973] increase the difference between pay-offs[Lurie, 1987]13 Individual differences: Different weights to the same valuation functions= Value(Choicei)The MARV Decision Making Model

Assess the situation from multiple perspectiveCombine valuation functions by assigning proper weights to each function (e.g. linear combination)General Formula:{Vself , Vother, Vtotal, }Depends on the problemDepend on the decision maker[Nouri and Traum. CMVC Workshop, Reykjavik, Iceland, 2011]

Groups using the same weights (which differ from non-group members) Group (culture) differences

Add a numerical example: this is what you care about

14Going Back to the ExamplesConsiderable variation of offers and acceptance rates across 4 cultures [Roth 1993; Camerer 2003]Frequency of offers made by proposersAcceptance ratio by respondersObserved Results in 4 countries:Observed Results in 1 Country:(Roth, 1993)Example 1: Cultures are different in their objectives in the negotiation. Collectivistic cultures care about the gain of the other party more than individualistic cultures. [Carnevale, 1997] [Adair 2001]Example 2:

What each curve tells us?How are they different?

15Cultural differences are modeled by appropriate weight setup.= Value(Choicei)The MARV Decision Making Model

Assess the situation from multiple perspectiveCombine valuation functions by assigning proper weights to each function (e.g. linear combination)General Formula:{Vself , Vother, Vtotal, }Depends on the problemDepend on the decision maker[Nouri and Traum. CMVC Workshop, Reykjavik, Iceland, 2011]

Groups using the same weights (which differ from non-group members) Group (culture) differences

Add a numerical example: this is what you care about

16= Value(Choicei)Adapting MARV to model Culture

How to set the weights based on the Culture?Use of a Model of Culture

Groups using the same weights (which differ from non-group members) Group (culture) differences

17Using Culture Model to Set Up the WeightsHofstede dimensional model of culture:Power Distance (PDI)Individualism vs. Collectivism (IDV)Masculinity vs. Femininity (MAS) Uncertainty Avoidance (UAI)Long- vs. Short-Term Orientation (LTO)There are several existing models of culture: Hofstede, GLOBE, World Value Survey,Schwartz, [Nouri and D. Traum. CMVC Workshop, Reykjavik, Iceland, 2011]

IVR: Indulgence vs. RestraintMON: Monumentalism vs. Self-Effacement

18Are you the same person at work (or at school if youre a student) and at home? (LTO)Do other people or circumstances ever prevent you from doing what you really want to (IVR)how would you describe your state of health these days? (UAI)How important is religion in your life?(MON)How proud are you to be a citizen of your country? (MON)How often, in your experience, are subordinates afraid to contradict their boss (or students their teacher?) (PDI)One can be a good manager without having a precise answer to every question that a subordinate may raise about his or her work (UAI)Persistent efforts are the surest way to results (LTO)An organization structure in which certain subordinates have two bosses should be avoided at all cost (PDI)A company's or organization's rules should not be broken - not even when the employee thinks breaking the rule would be in the organization's best interest (UAI)To what extent We should honor our heroes from the past (LTO)have sufficient time for your personal or home life (IDV)Q2 have a boss (direct superior) you can respect (PDI)get recognition for good performance (MAS)have security of employment (IDV)have pleasant people to work with (MAS)do work that is interesting (IDV)be consulted by your boss in decisions involving your work (PDI)live in a desirable area (MAS)have a job respected by your family and friends (IDV)have chances for promotion (MAS)keeping time free for fun (IVR)moderation: having few desires (IVR)being generous to other people (MON)modesty: looking small, not big (MON)If there is something expensive you really want to buy but you do not have enough money, what do you do? (LTO)How often do you feel nervous or tense?(UAI)Are you a happy person? (IVR)

VS08 Hofstede Survey Questions

19Mapping the Culture Model to Weights on the ValuationsValue(Choicei)Hofstedes Dimensional Model of CultureCulture Model

Mapping for the Ultimatum Game ExampleFour valuations functions {Vself, Vother, Vself/Other, Vlower-bound}

The Individualism Dimension (IDV)

High individualism: has focus on self utilityLow individualism (high collectivism)Different valuations for in-group vs. out-group relationshipsFor in-group focuses on other utility and fairnessFactors to model:A society's position on this dimension is reflected in whether peoples self-image is defined in terms of I or we.

Dialogue SystemUses (TACQ ) architectureVirtual Humans playing Ultimatum Game with one another

Policy for proposals made and acceptance or rejections by the responder are based on the MARV model calculations (weights are set based on the culture-model)

Integration with Virtual Humans

Humans Playing Ultimatum Game through Natural Language Dialog with Virtual Humans (US Culture Model)

23

Integration with Virtual HumansVirtual Humans Playing Ultimatum Game with Virtual humans24Evaluation[Nouri and D. Traum. CMVC Workshop, Reykjavik, Iceland, 2011]Results of simulating the one shot Ultimatum GameCultureHofstede: PDI, IDV, MASVH mean offer ($)Human mean offer ($)VH rejection rate (%)Human rejection rate (%)ProposerResponderAustria11 , 55 , 7933.1339.219.116.1Chile63 , 23 , 2833.1334.001.06.7Ecuador78 , 8 , 6333.1334.501.07.5Germany31 , 64 , 6136.8836.709.19.5Israel13 , 54 , 4721.6641.7125.017.7Japan54 , 46 , 9532.5044.731.019.3Spain57 , 51 , 4228.7526.6625.029.2Sweden34 , 70 , 433.1335.3310.218.2US40 , 91 , 6241.8842.2512.017.2

The weights are set based on findings in the literature and Hofstedes model of culture

25LimitationsIs manualRelies on previous culture modelsMight not existThe interpretations of the culture model is done manually based on the literature and personal understandingMight not reflect reality of the culture

Can these shortcoming be addressed?Automatically Learning the Weights If cultural data of decision making is available then its possible to use Inverse Reinforcement Learning (IRL) for automatically learning the weights on the valuation functions.

IRL assumes that the expert is trying to optimize an unknown reward function that can be expressed as a linear combination of known features.

The goal is to find the reward function that makes agent's behavior similar to that of the goal(expert) data.

Learn reward functions for 4 different cultures (US, Japan, Israel, Yugoslavia) playing the Ultimatum Game. [Roth et al. 1991]First with MARV values: {Vself, Vother, Vratio, Vlower-bound}Cultural Data[E. Nouri, K. Georgila, and D. Traum. A Cultural Decision-Making Model for Negotiation based on Inverse Reinforcement Learning. CogSci 2012]The algorithms from [Abbeel and Ng, 2004] are not guaranteed to correctly recover the expert's true reward function, but they show that the algorithm will nonetheless find a policy that performs as well as the expert, where performance is measured with respect to the expert's unknown reward function.

28PolicyInverse Reinforcement LearningReinforcement Learning AgentIterates until simulated data is similar enough to cultural human dataEnvironmentinteractionReward FunctionSimulated BehaviorGeneratesCultural BehaviorCompareSimilar enough?YesNoUpdate the Reward functionReward FunctionInitial Random Reward FunctionEvaluation with Ultimatum Game[E. Nouri, K. Georgila, and D. Traum. A Cultural Decision-Making Model for Negotiation based on Inverse Reinforcement Learning. CogSci 2012]Experiment 1: compare learned policies to real cultural dataModels outperform two baselines First baseline: random rewardsStrong baseline: maximizing self interestrandomself interestIRLUS3.9519.822.84JP4.014.860.74IS3.6816.111.29YU9.283.491.73randomself interestIRLUS0.610.370.1JP0.640.250.16IS0.580.270.13YU0.570.260.11ProposerKL divergences for IRL and the two baselines for all cultures and roles

ResponderHumanIRLRandomSelf interestKalliori: question about IRL30Evaluation with Ultimatum GameExperiment 2: use reward function learned from each culture to train policies for each role (proposer, responder) for the agent against users of each culture playing the other role

Learned Reward Function for JapanJapanUSIsraelYugoslaviaTrain againstJapanUSIsraelYugoslaviaTest against

Compare with real dataShow the tables row and column informationMake 2 slides

31Evaluation with Ultimatum GamePolicy with reward function and users from same culture are more like observed data from that culture than from other combinations of reward function and users Models for culture outperform models for other cultures

USJPISYUUS2.843.114.612.71JP1.050.741.061.96IS1.822.041.294.27YU2.212.835.761.73USJPISYUUS0.10.130.080.06JP0.270.160.180.24IS0.250.140.130.2YU0.150.270.20.11ProposerResponderCross-culture results, comparison with human data from different cultures (KL divergences)Reward FunctionsReward FunctionsTrain and Test DataShow the tables row and column informationMake 2 slides

32LimitationsNeeds behavioral data to nd the values of peopleSummary and ConclusionAdapted a social utility decision making model to culture with two approachesShowed how the decision making model can be used to simulate cultures behavior in simple negotiation