dellarocas - reputation article

8/12/2019 Dellarocas - Reputation Article

1/8

Online Reputation Systems:

How to Design One That

Does What You Need

S P R I N G 2 0 1 0 V O L . 5 1 N O . 3

R E P R I N T N U M B E R 5 1 3 0 8

Chrysanthos Dellarocas


2/8

SLOANREVIEW.MIT.EDU SPRING 2010 MITSLOAN MANAGEMENT REVIEW 33

SOME OF THE MOST EXCITINGnew ideas in organizations are based on harnessing the collec-tive intelligence of crowds and communities. Think of user-generated content platforms, open source

software, crowdsourcing, and knowledge markets just a few examples of ways that large numbers of

loosely networked users are invited to create or evaluate products, content or ideas. This is the social web,

the interlinked virtual universe that to so many executives seems to offer the irresistible promise of provid-

ing something ideas, work, decisions for (almost) nothing, if only they could manage it right.

But the first step toward smart management of the social web is to understand something para-

doxical about it: The new platforms may be all about harnessing crowds and communities, but in

the end, those crowds and communities are nothing but a sum of individuals. And your companys

social web efforts will succeed only to the extent that you are able to attract good individuals, mo-

tivate them to perform good work, and empower them to get to know and trust one another

enough to collaborate toward the end

goals of the community.

The question is, How do you do

that?

The answer : By capitalizing on the

motivational power of reputation.

The best websites know that. (Even

poor ones know it they just dont

manage it as well.) Accordingly almost

all social web platforms have some formof member profile feature, a set of pages

where one can find out about individual

community members. Member profile

pages provide a space where commu-

nity members can write about them-

selves, and their interests. In addition,

profile pages often include statistics

about a members activity, testimonials

from other members and sometimes a

Social web platforms dont thrive by magic. They can succeedonly if they attract the right individuals, motivate them to act inthe right ways and empower them to know and trust others inthe network. Thats where online reputation systems come in.BY CHRYSANTHOS DELLAROCAS

Online Reputation Systems:

How to Design One ThatDoes What You NeedTHE LEADING

QUESTION

How can awebsiteattract the

contributorsit needs?

FINDINGS

Designers have tobe driven first by thebusiness objectivesof the website.

Four main aims:build trust; promotequality; facilitatemember matching;and sustain loyalty.

Design choicescan profoundlyaffect a communitysculture, easilyturning a goodspace into anineffective one.

M A N A G I N G T H E S O C I A L W E B

HOW YELP.COM MAKES REPUTATION A TOOL

Yelp.com, a site aggregating reviews that contain highly subjective judgments, has built an

online reputation system that enables similarly-minded users to spot each other.

Social Network

Feedback/Testimonials

Activity Statistics
http://www.sloanreview.mit.edu/http://www.sloanreview.mit.edu/


3/8

34 MITSLOAN MANAGEMENT REVIEW SPRING 2010 SLOANREVIEW.MIT.EDU


score or rating that reflects a members accomplish-

ments and status within that particular community.

Other members can use this information to decide

how much to trust a member, how much credibility to

assign to his or her postings or whether to engage in atransaction or collaboration with him or her.

These features are parts of online reputation sys-

tems. Reputation systems are ubiquitous but often

undernourished and badly designed compo-

nents of social web platforms. Yet they play a crucial

role in building trust, promoting quality, improving

collaboration and instilling loyalty. More than al-

most any other aspect of a social web platform,

reputation systems are the puzzle piece that can

make the difference between success and failure.

Theyre the unsung heroes of the social web.

To name a few examples: eBay, a trading com-

munity of virtual strangers, would not have been

possible without the reputation system that pro-

motes honesty and builds trust by enabling buyers

and sellers to rate each other. At the same time,

eBays reputation systems poorly designed has done

an inadequate job of deterring fraud and, over time,

has required eBay to put in place several additional

monitoring mechanisms. Amazons product reviews

feature is rendered both more useful and more cred-

ible by the presence of a scoring system that allowsreaders to vote on a reviews usefulness and uses

such votes to point customers to the most useful re-

views and thereby reward the most successful

reviewers. Yelp, a youthful urban reviews commu-

nity, similarly uses what is essentially a reputation

system to help users better interpret opinions that

are inherently subjective. On the other hand, Digg.

com, a social news filtering site, instilled unneces-

sary competition among its members through a

poorly designed reputation system. This ended up

promoting collusive behavior to the detriment of

the sites quality and eventual impact.

Over the past 10 years researchers and practitio-

ners in diverse fields have studied reputationsystems, arriving at important results.

Key Decision #1: What arethe key business objectivesof your reputation system?Before you even approach the website drawing

board it is important to understand and prioritize

the goals that a reputation system will serve in your

social web platform, both from the perspective of

your organization and from that of your users.

Reputation is a summary of ones past actionswithin the context of a specific community, pre-

sented in a manner that can help other community

members make decisions with respect to whether

and how to relate to that individual. The concept of

reputation is as old as society itself. Reputation ex-

ists and is propagated within human communities

whether a central system mediates and facilitates

the process or not. Small, tightly knit communities

arguably do not need central reputation systems,

since frequent interactions and gossip ensure that

relevant information is known to all.

The need for a central system increases with the

size of the community and the lack of frequent inter-

action among members. In web-based communities

with hundreds or thousands of members, where

most members typically know each other only virtu-

ally, some form of a reputation system is almost

always essential.

Reputation is most often associated with the no-

tion of trust. Although trust-building has histori-

cally been an important role of reputation, todays

web-based reputation systems serve a surprisingly

broad set of objectives that includes the following:Build Trust. Encouraging good behaviors and

discouraging bad ones within the context of a site

is perhaps the most obvious and widely discussed

role of reputation systems, best exemplified by eBay.

Promote Quality. In the majority of social web

platforms the quality of user contributions tends to

be very uneven: a small number of users typically

supplies the bulk of quality contributions. Reputa-

tion systems can enhance such systems by helping

BUILDTRUST

PROMOTEQUALITY

FACILITATEMEMBERMATCHING

SUSTAINLOYALTY

eBay

Amazon

Yelp

Xbox Live

CUSTOMIZED GOALS HOW COMPANIES PRIORITIZE

There are four main objectives that web-based reputation systems can be designed

to serve, and companies prioritize them differently, as shown here.


4/8


HOW eBAY AND EPINIONS SHOW REPUTATION

Good reputation systems are designed to promote just the behaviors that

advance their websites aims. Here, examples from Epinions and eBay.

to recognize and feature high-quality contributors.

This provides an incentive to contributors to try

harder (so that they can be recognized) while help-

ing users easily identify quality content.

Facilitate Member Matching.This objective is im-portant in settings where members vary widely in

interests and tastes and where the quality of contribu-

tions is characterized by a high degree of subjectivity.

In such settings reputation systems can help users as-

sess how similar or compatible they are with each

other so that they can decide how much to trust some-

ones posting or whether to initiate collaboration.

Sustain Loyalt y. Since reputation is usually

based on activity performed within a single com-

munity or web system, it constitutes a powerful

form of lock-in and can be used strategically to in-crease user loyalty and decrease attrition. Once

users have built a reputation on a site, they will be

reluctant to defect to a competitor since they would

then have to build their reputation from scratch. In

an increasingly competitive environment this di-

mension of reputation mechanisms must not

be overlooked.

All four objectives are relevant to the design of

most reputation systems. Different social web plat-

forms, however, assign different priorities to each

of these objectives (see Customized Goals How

Companies Prioritize). For example, building suf-

ficient trust so that buyers can feel comfortable

enough to send their money to sellers they have

never met and, very likely, will never buy from

again, is arguably the primary objective of eBays

reputation mechanism. On the other hand, the

most important objective of Amazons reviewer

reputation mechanism is to induce members to

contribute well thought out, high-quality reviews

and to identify (or, filter) the highest quality re-

views. Inducing top-quality contributions is high

on Yelps agenda as well. However, because themajority of Yelp reviews are subjective, it is usually

difficult to reliably assess a reviews quality. What

is more important is to provide tools that can help

readers make their personal assessment of a re-

views credibility and how well it fits their view

of the world. Finally, Xbox Live is an example of

a system where user loyalty and retention is

the overarching (business) objective of its reputa-

tion feature.

Key decision #2: What informationshould be included in your usersreputation profile?A fair reputation system should be based on a users ac-

tivity within a community. It should focus on reporting

summaries of a users actions or ratings of a users com-

munity contributions. It is usually OK to allow users

to post a few things about themselves. However, in al-

most all cases, reputation systems should not allowother users to post direct feedback on an individual,

unless it is tied to a specific action or community con-

tribution; allowing direct rating of individuals opens

the door to gaming, blackmail and more.

Users typically engage in numerous and varied

behaviors within the context of a site. Effective rep-

utation systems must carefully choose which aspects

of user behavior to track and report on. (See What

Information Should Your Reputation System

Social Network ActivityStatistics

NamedTiers

Star Ratings

AchievementBadges

NumericalScore


5/8



WHAT INFORMATION SHOULD YOUR REPUTATION SYSTEM FOCUS ON?

Effective reputation systems must carefully choose which aspects of user behavior to track and report on. This table summarizes how the various

common ways of aggregating and displaying reputation information relate to a reputation system's business goals.

PRIMARY OBJECTIVE OFREPUTATION SYSTEM

WHAT REPUTATION AGGREGATIONAND DISPLAY APPROACHES WORK BEST WHAT TO BE CAREFUL ABOUT

Build Trust Raw activity statistics (e.g., YahooAnswers activity summary).

Tiered membership levels (e.g., eBayspower sellers, Slashdots moderators).

Avoid the use of scores and rankings that instill unnecessarycompetition among users if your objective is to help determineusers who are just honest or reliable enough.

Reliance on cumulative metrics may tempt veteran users tooccasionally cheat.

Promote Quality Scoring mechanisms that can help userseasily separate the wheat from the chaff (e.g.,Topcoders user rating, Xbox Lives star rating).

Judicious use of user ranking methods(e.g., Amazons Top Reviewers).

Reliance on cumulative metrics may tempt seasoned users tooccasionally slack.

If you must use them, user rankings and leaderboards must notbe very conspicuous.

Facilitate MemberMatching

Raw activity statistics allow users to assesstheir similarity and compatibility with one other(e.g., Yelps user profile and friends network).

Avoid the use of scores; if you must use scores, dont use a singlescore; instead evaluate users across a variety of dimensions.

Avoid ranking users; if you must do so, maintain several differentrankings along different dimensions of contribution.

Sustain Loyalty Cumulative metrics that keep increasing thelonger a member stays on the site (e.g., eBaysfeedback score, Yahoo Answers points).

Newcomers must be given room to shine; use separate metrics fornew members, or use newness-based metrics alongsidecumulative metrics.

Rankings may increase the loyalty of users who end up at the topbut might alienate the rest.

Focus On?) Keeping in mind the business objectives

that the reputation system is intended to serve (Key

Decision #1), there are three aspects to this decision:

Which actions are most relevant to the reputa-

tion systems users?For example, if the key objectiveof the system is to help users decide whether a seller

is honest, then keeping track of the sellers percent-

age of completed transactions is a very relevant

indicator, whereas keeping track of a sellers own

purchase history is less relevant. On the other hand,

if a systems key objective is to help users determine

whether a reviewer has similar tastes to their own,

keeping track of a reviewers purchase history might

be a very relevant indicator.

Which user behaviors are desirable? The mere

act of publicly keeping track of someones actionscan encourage or discourage their incidence (de-

pending on whether the action in question has a

good or bad connotation). Therefore, if a site wants

to encourage the volume of contributions, it might

consider keeping track on reputation profiles of the

number of reviews posted or the number of com-

ments posted. On the other hand, if a site wants to

encourage the quality of contributions it might

want to hide information about contribution

volume and keep track of how other people rated a

particular contribution instead.

For which behaviors is it possible to obtain re-

liable information?The moment a reputation

system decides to highlight some aspect of behav-ior, individuals are bound to try to game the system

to their advantage. It is, therefore, important to

choose metrics that are reliable and difficult to ma-

nipulate. The main choice here is between internally

generated (first-hand) information and (second-

hand) feedback provided by others. Yelps reporting

of the volume of a users posted reviews or the size

of ones friends network are examples of first-hand

information. In contrast, eBays reliance on asking

the buyer and the seller what they thought about a

transaction is an example of second-hand feedback.Relying on internally-generated information about

user actions is generally preferable as it tends to be

more reliable. Unfortunately, such information is

not always available, or its collection might require

the development of costly additional infrastruc-

ture. For this reason reputation systems must often

rely on second-hand feedback provided by others.

When doing so, one must put in place mechanisms

to limit manipulation and gaming.


6/8


Key Decision #3: How shouldreputation information beaggregated and displayed?Reputation mechanisms employ a variety of meth-

ods for displaying outputs (see How eBay and

Epinions Show Reputation, p. 35). These fall into

three large categories:

Raw activity statistics. Examples: number of reviews

posted, number of transactions completed.

Scores and distinctions. Examples: star ratings

(such as Amazon rev iews), numerical scores

(eBays feedback score, TopCoders user rating),numbered levels or named member tiers (World

of Warcrafts player levels, Slashdots moderator

and meta-moderator tiers) or achievement

badges (eBay power seller, Amazon Top

Reviewer).

Leaderboards and other methods of displaying rela-

tive user rankings. Examples: the list of top Amazon

reviewers; Epinions author popularity ranking.

The choice of an aggregation and display method

is very important because it can determine, a) the

extent to which the reputation mechanism makes

a judgment versus allowing users to make their own

judgments, and b) the extent to which the presence

of the reputation system can create competition

among users.

Displaying raw statistics of a persons activity

within a community is perhaps the most neutral

method of summarizing someones reputation. Social

network statistics, in particular, have been rapidly

rising in popularity in many contexts. The advantage

of using raw statistics is that the reputation systemthus makes minimal judgments and allows users to

draw their own conclusions. The disadvantage is that

the burden of interpreting these quantities falls on

the shoulders of the user, who must be familiar

enough with the environment to draw the proper

conclusions.

The use of some form of explicit score, such as

star ratings, solves this problem by immediately

communicating to users whether somebodys perfor-

YELP: A CASE STUDY IN ONLINE REPUTATION SYSTEM

MANAGEMENT

Yelp.com is an online review community that specializes on reviews of restaurants and entertain-

ment establishments in major urban centers. At the same time, Yelp is a community of mostly young

users that relate to each other through their similar taste and interests. Yelp maintains a profile page

for its members (See How Yelp Makes Reputation a Tool). The profile page is essentially a reputa-tion mechanism.

An important objective of Yelps profile feature is to help other users decide how to interpret a re-

view by finding out more about a reviewers personality and tastes. At the same time the profile

allows a member to showcase her status and contributions to the system, building site loyalty and

providing incentives for continued contributions. Because Yelps domain (restaurants, entertainment

and other cultural establishments) is highly personality and taste-dependent, Yelps profile feature is

wisely avoiding the use of numerical scores and star ratings. Instead, a reviewers profile primarily

consists of a series of simple statistics that summarize salient aspects of the users activity. In addi-

tion to simple statistics, Yelp supports second-hand feedback in the form of compliments received

from other users. Note, however that: a) these compliments are displayed below the neutral statis-

tics and are thus (intentionally?) not highlighted as much; and b) there are several different types of

compliments (good writer, funny, youre cool), underlining the plurality of ways in which a user

can make valuable contributions and also making it difficult to compare users against each other.

Although Yelp supports some form of relative user ranking, or leaderboard, it is interesting to

note that the ranking feature is not easily accessible from the home page, and that Yelp offers multi-

ple dimensions across which users can be rank-ordered, once again emphasizing that one can stand

out in this community for multiple reasons and making it difficult to compare users against each

other. As a consequence, Yelps leaderboard feature doesnt add much value. Yelps reputation mech-

anism would be just as effective without it.

Overall, however, Yelps reputation system design is well-suited to its business goals of inducing

participation and valuable contributions and facilitating the matching of like users in a domain where

the notion of value has a highly subjective definition.


7/8



mance along some dimension is good or bad. This

method helps users more easily digest information but

also makes an explicit judgment about quality. It is,

therefore, best in settings where there is a commonly

agreed upon notion of what quality means.Leaderboards and rankings go a step further as they

not only imply judgment but also indicate a users stand-

ing relative to everyone else. They introduce a pecking

order among users. This can have a very strong impact

on a communitys culture and behavior. Supporting di-

rect comparison of users against one another sometimes

increases incentives to contribute. On the other hand,

such direct comparison instills a culture of competition

which might end up being disruptive in a number of

ways. First, obsession with rankings might lead some

users to manipulative and counter-productive behavior.Second, users who dont make it to the top ranks might

feel resentful and induced to exit the community. Expe-

rience has shown that in almost all cases, prominently

displayed rankings or leaderboards lead to more trouble

than benefits. If you insist on using them, make them a

relatively inconspicuous feature of your reputation

mechanism.

The story of Digg.com provides an example of the

dangers of introducing too much competition in a

community whose culture is otherwise meant to be

cooperative. Digg.com is a system that allows users to

recommend news articles that piqued their interest.

Other users vote recommended articles up or down.

An articles number of votes determines its relative

visibility on the site. The site was meant to be a grass-

roots method of determining what is interesting to

read about today. Digg.com implemented a reputa-

tion mechanism that originally included a leaderboard

of most influential Diggers, that is, of Digg mem-

bers whose recommended articles received the most

votes. Some of the leading Digg members became so

obsessed with their rankings that they formed a collu-

sion ring for the purpose of ensuring that theirrecommendations always made it to the top. This

ended up eroding the culture and dynamics of the

site. Amid controversy, in January 2007 Digg decided

to abolish the Digg leaderboard. Today, Diggs user

profile consists of various simple statistics only.

Another important design dimension is whether a

system bases its reporting on a users entire history on

the site or whether statistics, scores or rankings are based

on recent behavior only. Cumulative metrics keep in-

creasing the longer a user stays on a site. Thats good for

building loyalty but not so good for incentivizing con-

tinued contributions of seasoned users. It might also

discourage new members from joining since they may

judge it difficult or impossible to catch up with veteranmembers. Recent behavior metrics have the opposite

properties: they keep members on their toes, they give

newcomers equal opportunities but they reduce the cost

of moving to a competitors site. A judicious combina-

tion of both types of metrics usually works best.

After several iterations, eBay and Amazon have

both converged on such combinations. eBays reputa-

tion profile now includes both a feedback score, which

is a cumulative metric, as well as a breakdown of a

members recent feedback. Amazon now publishes

two different rankings of reviewers: one primarilybased on cumulative contributions and another based

on recent contributions.

OF COURSE,reputation systems in the social web are

more complex than a short article can explain, and

their design principles surprisingly subtle. Maybe its

unsurprising that so many web platforms have gotten

their systems wrong, with severe consequences.

But by starting with the fundamentals, reputation

system designers can go far toward creating exactly the

system that will advance their companys aims.

FURTHER READING

C. Dellarocas. The Digitization of Word-of-Mouth: Prom-

ise and Challenges of Online Reputation Systems.

Management Science 49 (10), October 2003, 1407-1424.

C. Dellarocas, F. Dini and G. Spagnolo. Designing Repu-

tation Mechanisms. Chapter 18 in Handbook of

Procurement, N. Dimitri, G. Piga, G. Spagnolo (eds.), Cam-

bridge University Press, 2007, 446-482.

A. Jsang, R. Ismail and C. Boyd. A survey of trust and

reputation systems for online service provision. Decision

Support Systems 43 (2), March 2007, 618-644.

P. Resnick. K. Kuwabara, R. Zeckhauser and E. Friedman

Reputation Systems. Communications of the ACM 43 (12),December 2000, 45-48.

Chrysanthos Dellarocasis an associate professor of

management at Boston University School of Manage-

ment and an affiliated researcher at MIT's Center for

Collective Intelligence.

Reprint 51308.

Copyright Massachusetts Institute of Technology, 2010.

All rights reserved.


8/8

PDFs Permission to Copy Back Issues Reprints

Articles published in MIT Sloan Management Review arecopyrighted by the Massachusetts Institute of Technology

unless otherwise specified at the end of an article.

MIT Sloan Management Review articles, permissions,

and back issues can be purchased on our Web site:www.pubservice.com/msstoreoryou may order through

our Business Service Center (9 a.m.-7 p.m. ET) at thephone numbers listed below. Paper reprints are available

in quantities of 250 or more.

To reproduce or transmit one or more MIT SloanManagement Review articles by electronic ormechanical means(including photocopying or archivingin any information storage or retrieval system) requires

written permission.To request permission, use our Web site(www.pubservice.com/msstore), call or e-mail:

Toll-free:800-876-5764 (US and Canada)International:818-487-2064

Fax:818-487-4550E-mail:[email protected]

Posting of full-text SMR articles on publicly accessibleInternet sites is prohibited.To obtain permission to post

articles on secure and/or password-protected intranet sites,e-mail your request to [email protected]

Customer ServiceMIT Sloan Management ReviewPO Box 15955

North Hollywood, CA 91615
http://www.pubservice.com/msstorehttp://www.pubservice.com/msstorehttp://www.pubservice.com/msstorehttp://www.pubservice.com/msstoremailto:[email protected]:[email protected]:[email protected]:[email protected]://www.pubservice.com/msstorehttp://www.pubservice.com/msstore

dellarocas - reputation article

Documents