Regret Minimizing Audits:A Learning-theoretic Basis for Privacy Protection

Jeremiah BlockiCarnegie Mellon University

Nicolas ChristinCarnegie Mellon University

Anupam DattaCarnegie Mellon University

Arunesh SinhaCarnegie Mellon University

Abstract—Audit mechanisms are essential for privacy pro-tection in permissive access control regimes, such as in hospi-tals where denying legitimate access requests can adverselyaffect patient care. Recognizing this need, we develop thefirst principled learning-theoretic foundation for audits. Ourfirst contribution is a game-theoretic model that captures theinteraction between the defender (e.g., hospital auditors) andthe adversary (e.g., hospital employees). The model takes prag-matic considerations into account, in particular, the periodicnature of audits, a budget that constrains the number of actionsthat the defender can inspect, and a loss function that capturesthe economic impact of detected and missed violations on theorganization. We assume that the adversary is worst-case as isstandard in other areas of computer security. We also formulatea desirable property of the audit mechanism in this modelbased on the concept of regret in learning theory. Our secondcontribution is an efficient audit mechanism that provablyminimizes regret for the defender. This mechanism learns fromexperience to guide the defender’s auditing efforts. The regretbound is significantly better than prior results in the learningliterature. The stronger bound is important from a practicalstandpoint because it implies that the recommendations fromthe mechanism will converge faster to the best fixed auditingstrategy for the defender.

I. INTRODUCTION

Audits complement access control and are essential forenforcing privacy and security policies in many situations.Specifically, audits serve an important function for pro-tecting privacy in organizations that collect, share and usepersonal information. Health care providers, in particular,use permissive access control policies to grant access topatient records since wrongly denying or delaying accessto a patient’s record could have adverse consequences onthe quality of patient care. Unfortunately, a permissiveaccess control regime opens up the possibility of recordsbeing inappropriately accessed. Recent studies have revealed

This work was partially supported by the U.S. Army Research Office con-tract “Perpetually Available and Secure Information Systems” (DAAD19-02-1-0389) to Carnegie Mellon CyLab, the NSF Science and TechnologyCenter TRUST, the NSF CyberTrust grant “Privacy, Compliance andInformation Risk in Complex Organizational Processes,” the AFOSR MURI“Collaborative Policies and Assured Information Sharing,” and HHS Grantno. HHS 90TR0003/01. Jeremiah Blocki was also partially supported bya NSF Graduate Fellowship. The views and conclusions contained inthis document are those of the authors and should not be interpretedas representing the official policies, either expressed or implied, of anysponsoring institution, the U.S. government or any other entity.

that numerous policy violations occur in the real world asemployees access records of celebrities, family members,and neighbors motivated by general curiosity, financial gain,child custody lawsuits and other considerations [1], [2].

Audit mechanisms help detect such violations of policy. Inpractice, organizations like hospitals conduct ad hoc auditsin which the audit log, which records accesses and dis-closures of personal information, is examined to determinewhether personal information was appropriately handled. Incontrast to access control, the audit process cannot be com-pletely automated for relevant privacy policies. For example,a recent formal study of privacy regulations [3] shows that alarge fraction of clauses in the HIPAA Privacy Rule [4] re-quires some input from human auditors to enforce. We seekto develop an appropriate mathematical model for studyingaudit mechanisms involving human auditors. Specifically,the model should capture important characteristics of prac-tical audit mechanisms (e.g., the periodic nature of audits),and economic considerations (e.g., cost of employing humanauditors, brand name erosion and other losses from policyviolations) that influence the coverage and frequency ofaudits.

This paper presents the first principled learning-theoreticfoundation for audits of this form. Our first contribution is arepeated game model that captures the interaction betweenthe defender (e.g., hospital auditors) and the adversary(e.g., hospital employees). The model includes a budgetthat constrains the number of actions that the defendercan inspect thus reflecting the imperfect nature of audit-based enforcement, and a loss function that captures theeconomic impact of detected and missed violations on theorganization. We assume that the adversary is worst-caseas is standard in other areas of computer security. We alsoformulate a desirable property of the audit mechanism in thismodel based on the concept of regret in learning theory [5].Our second contribution is a novel audit mechanism thatprovably minimizes regret for the defender. The mechanismlearns from experience and provides operational guidanceto the human auditor about which accesses to inspect andhow many of the accesses to inspect. The regret boundis significantly better than prior results in the learningliterature.

The importance of audits has been recognized in com-puter security and information privacy (see, for example,Lampson [6], Weitzner et al. [7]). However, unlike accesscontrol, which has been the subject of a significant body offoundational work, there is comparatively little work on thefoundations of audit. Our work aims to fill this gap.

A. Overview of Results

Mirroring the periodic nature of audits in practice, we usea repeated game model [8] that proceeds in rounds. A roundrepresents an audit cycle and, depending on the applicationscenario, could be a day, a week or even a quarter.Adversary model: In each round, the adversary performsa set of actions (e.g., accesses patient records) of which asubset violates policy. Actions are classified into types. Forexample, accessing celebrity records could be a differenttype of action from accessing non-celebrity records. Theadversary capabilities are defined by parameters that imposeupper bounds on the number of actions of each type thatshe can perform in any round. We place no additionalrestrictions on the adversary’s behavior. In particular, wedo not assume that the adversary violates policy following afixed probability distribution; nor do we assume that she isrational. Furthermore, we assume that the adversary knowsthe defender’s strategy (audit mechanism) and can adapt herstrategy accordingly.Defender model: In each round, the defender inspects asubset of actions of each type performed by the adversary.The defender has to take two competing factors into account.First, inspections incur cost. The defender has an auditbudget that imposes upper bounds on how many actionsof each type she can inspect. We assume that the cost ofinspection increases linearly with the number of inspections.So, if the defender inspects fewer actions, she incurs lowercost. Note that, because the defender cannot know withcertainty whether the actions not inspected were maliciousor benign, this is a game of imperfect information [9].Second, the defender suffers a loss in reputation for detectedviolations. The loss is higher for violations that are detectedexternally (e.g., by an Health and Human Services audit, orbecause information leaked as a result of the violation ispublicized by the media) than those that are caught by thedefender’s audit mechanism, thus incentivizing the defenderto inspect more actions.

In addition, the loss incurred from a detected violationdepends on the type of violation. For example, inappropriateaccess of celebrities’ patient records might cause higher lossto a hospital than inappropriate access of other patients’records. Also, to account for the evolution of public memory,we assume that violations detected in recent rounds causegreater loss than those detected in rounds farther in thepast. The defender’s audit mechanism has to take all theseconsiderations into account in prescribing the number ofactions of each type that should be inspected in a given

round, keeping in mind that the defender is playing againstthe powerful strategic adversary described earlier.

Note that for adequate privacy protection, the economicand legal structure has to ensure that it is in the best interestsof the organization to invest significant effort into auditing.Our abstraction of the reputation loss from policy violationsthat incentivizes organizations to audit can, in practice, beachieved through penalties imposed by government audits aswell as through market forces, such as brand name erosionand lawsuits.Regret property: We formulate a desirable property forthe audit mechanism by adopting the concept of regretfrom online learning theory. The idea is to compare theloss incurred when the real defender plays according tothe strategy prescribed by the audit mechanism to the lossincurred by a hypothetical defender with perfect knowledgeof the number of violations of each type in each round. Thehypothetical defender is allowed to pick a fixed strategy toplay in each round that prescribes how many actions of eachtype to inspect. The regret of the real defender in hindsight isthe difference between the loss of the hypothetical defenderand the actual loss of the real defender averaged over allrounds of game play. We require that the regret of theaudit mechanism quickly converge to a small value and, inparticular, that it tends to zero as the number of rounds tendsto infinity.

Intuitively, this definition captures the idea that althoughthe defender does not know in advance how to allocateher audit budget to inspect different types of accesses (e.g.,celebrity record accesses vs. non-celebrity record accesses),the recommendations from the audit mechanism should havethe desirable property that over time the budget allocationcomes close to the optimal fixed allocation. For example, ifthe best strategy is to allocate 40% of the budget to inspectcelebrity accesses and 60% to non-celebrity accesses, thenthe algorithm should quickly converge towards these values.Audit mechanism: We develop a new audit mechanism thatprovably minimizes regret for the defender. The algorithm,which we name RMA (for Regret Minimizing Audit) isefficient and can be used in practice. In each round of thegame, the algorithm prescribes how many actions of eachtype the defender should inspect. It does so by maintainingweights for each possible defender action and picking anaction with probability proportional to the weight of thataction. The weights are updated based on a loss estimationfunction, which is computed from the observed loss ineach round. Intuitively, the algorithm learns the optimaldistribution over actions by increasing the weights of actionsthat yielded better payoff than the expected payoff of thecurrent distribution and decreasing the weight of actions thatyielded worse payoff.

Our main technical result (Theorem 1) is that the exactbound on regret for RMA is approximately 2

√2 lnN

T where

N is the number of possible defender actions and T isthe number of rounds (audit cycles). This bound improvesthe best known bounds of O

(N1/3 logN

3√T

)for regret min-

imization over games of imperfect information. The mainnovelty is in the way we use a loss estimation function andcharacterize its properties to achieve the significantly betterbounds. Specifically, RMA follows the structure of a regretminimization algorithm for perfect information games, butuses the estimated loss instead of the true loss to updatethe weights in each round. We define two properties of theloss estimation function—accuracy (capturing the idea thatthe expected error in loss estimation in each round is zero)and independence (capturing the idea that errors in lossestimation in each round are independent of the errors inother rounds)—and prove that any loss estimation functionthat satisfies these properties results in regret that is closeto the regret from using an actual loss function. Thus, ourbounds are of the same order as regret bounds for perfectinformation games. The better bounds are important from apractical standpoint because they imply that the algorithmconverges to the optimal fixed strategy much faster.

The rest of the paper is organized as follows. Section IIpresents the game model formally. Section III presents theaudit mechanism and the theorem showing that the auditmechanism provably minimizes regret. Section IV discussesthe implications and limitations of these results. Section Vdescribes in detail the loss estimation function, a core pieceof the audit mechanism. Section VI presents the outline ofthe proof of the main theorem of the paper (Theorem 1)while the complete proofs are presented in the appendices.Section VII provides a detailed comparison with relatedwork, in particular, focusing on technical results on auditingin the computer security literature and regret minimization inthe learning theory literature. Finally, Section VIII presentsour conclusions and identifies directions for future work.

II. MODEL

We model the internal audit process as a repeated gameplayed between a defender (organization) and an adversary(employees). In the presentation of the model we will usethe following notations:• Vectors are represented with an arrow on top, e.g., ~v is a

vector. The ith component of a vector is given by ~v [i].~v ≤ ~a means that both vectors have the same numberof components and for any component i, ~v [i] ≤ ~a [i].

• Random variables are represented in boldface, e.g., xand X are random variables.

The repeated game we consider is fully defined by theplayers, the time granularity at which the game is played,the actions the players can take, and the utility the playersobtain as a result of the actions they take. We next discussthese different concepts in turn and illustrate them using arunning example from an hospital.

Players: The game is played between the organization andits employees. We refer to the organization as D (defender).We subsume all employees into a single player A (adver-sary). In this paper, we are indeed considering a worst-caseadversary, who would be able to control all employees andcoerce them into adopting the strategy most damaging tothe organization. In our running example, the players arethe hospital and all the employees.Round of play: In practice, audits are usually performedperiodically. Thus, we adopt a discrete-time model for thisgame, where time points are associated with rounds. Eachround of play corresponds to an audit cycle. We grouptogether all of the adversary’s actions in a given round.Action space: A executes tasks, i.e., actions that are per-mitted as part of their job. We only consider tasks thatcan later be audited, e.g., through inspection of logs. Wecan distinguish A’s tasks between legitimate tasks andviolations of a specific privacy policy that the organizationD must follow. Different types of violations may have adifferent impact on the organization. We assume that thereare K different types of violations that A can commit (e.g.,unauthorized access to a celebrity’s records, unauthorizedaccess to a family member’s records, ...). We further assumethat the severity of violations, in terms of economic impacton the organization, varies with the types.

In each audit cycle, the adversary A chooses two quanti-ties for each type k: the number of tasks she performs, andthe number of such tasks that are violations. If we denoteby Uk the maximum number of type k tasks that A canperform, then A’s entire action space is given by A×A withA =

∏Ki=1{1, . . . , Ui}. In a given audit cycle, an action by

A in the game is given by 〈~a,~v〉, where the componentsof ~a are the number of tasks of each type A performs,and the components of ~v are the number of violations ofeach type. Since violations are a subset of all tasks, wealways have ~v ≤ ~a. In our hospital example, we considertwo types of patient medical records: access to celebrityrecords and access to regular person’s record. A typicalaction may be 250 accesses to celebrity records with 10of them being violations and 500 accesses to non-celebrityrecords with 50 of them being violations. Then A’s actionis 〈〈250, 500〉, 〈10, 50〉〉.

We assume that the defender D can classify each adver-sary’s task by types. However, D cannot determine whethera particular task is legitimate or a violation without inves-tigating. D can choose to inspect or ignore each of A’stasks. We assume that inspection is perfect, i.e., if a violationis inspected then it is detected. The number of inspectionsthat D can conduct is bounded by the number of tasks thatA perform, and thus, D’s actions are defined by a vector~s ∈ A, with ~s ≤ ~a. That is, D chooses a certain number oftasks of each type to be inspected. Further, in each round t,D has a fixed budget Bt to allot to all inspections. Werepresent the (assumed fixed) cost of inspection for each

0 100 200 300 4000

100

200

300

400

500

600

Budget Line

number of type 2 actions

number oftype 1 actions

Actions of type 1

Actio

ns o

f typ

e 2

Figure 1. Feasible audit space, represented by the shaded area.

type of violation by ~C. The budgetary constraints on D arethus given by ~C · ~s ≤ Bt for all t. Continuing our hospitalexample, the maximum number of tasks of each type thatD can inspect is 〈250, 500〉. Assuming a budget of 1500and cost of inspection vector 〈4, 5〉, D’s inspection spaceis further constrained, and then the feasible inspections are{〈x, y〉 | 4x+ 5y ≤ 1500, 0 ≤ x ≤ 250, 0 ≤ y ≤ 500}. Thediscrete feasible audit points are indicated (not all points areshown) with the asterisks in Figure 1.Violation detection: Given the budgetary constraints Dfaces, D cannot, in general, inspect all of A’s tasks (i.e.,~C · ~a > Bt). Hence, some violations may go undetectedinternally, but could be detected externally. Governmentalaudits, whistle-blowing, information leaks are all but exam-ples of situations that could lead to external detection ofotherwise unnoticed violations. We assume that there is afixed exogenous probability p (0 < p < 1) of an internallyundetected violation getting caught externally.

Formally, we define the outcome of a single audit cycleas the outcome of the internal audit and the number ofviolations detected externally. Due to the probabilistic natureof all quantities, this outcome is a random variable. Let~Ot be the outcome for the tth round. Then ~Ot is a tuple〈~Ot

int,~Otext〉 of violations caught internally and externally.

By our definitions, the probability mass function for ~Otint

is parameterized by 〈~a,~v〉 and ~s, and the probability massfunction for ~Ot

ext conditioned on ~Otint is parameterized by

p. We make no assumptions about this probability massfunction. Observe that, because not all tasks can be in-spected, the organization does not get to know the exactnumber of violations committed by the employees, whichmakes this a game of imperfect information. In our hospitalexample, given that A’s action is 〈〈250, 500〉, 〈10, 50〉〉.In one possible scenario the hospital performs 〈125, 200〉inspections. These inspections result in 〈7, 30〉 violationsdetected internally and 〈2, 10〉 violations detected externally.

Utility function: Since we consider a worst-case adversary,A’s payoff function is irrelevant to our model. On the otherhand, the utility function of D influences the organization’sstrategy. We define D’s utility as the sum of D’s reputationand the cost of inspecting A’s actions. In essence, D hasto find the right trade-off between inspecting frequently(which incurs high costs) and letting violations occur (whichdegrades its reputation, and thus translates to lost revenue).

We assume that the cost of inspection is linear in thenumber of inspections for each type of action. Hence, if D’saction is ~s then inspection costs are ~C ·~s. In our running ex-ample of the hospital, this cost is 〈4, 5〉.〈125, 200〉 = 1500,which is the also the full budget in our example.

We assume that any violation caught (internally, or ex-ternally) in a round affects D’s reputation not only in thatround, but also in future rounds and that the exact effectin any future round is known. We capture this by defininga function rtk : {1, . . . , Uk} × {1, . . . , Uk} × N → R foreach type k of violation. In round t, rtk takes as inputsthe number of violations of type k detected internally, thenumber of violations of type k caught externally, and aninteger argument τ . rtk outputs the effect of the violations(measured as the loss in reputation) occurring in round t onD’s reputation in round t+ τ . We assume that violations ofa given type always have the same effect on reputation, thatis, rtk is actually independent of t, which allows us to usethe shorthand notation rk from here on.

Violations caught far in the past should have a lesserimpact on reputation than recently caught violations, thus, rkshould be monotonically decreasing in the argument τ . Wefurther assume violations are forgotten after a finite amountof rounds m, and hence do not affect reputation further.In other words, if τ ≥ m then for any type k, any roundt, and any tuple of violations caught 〈 ~Otint[k], ~Otext[k]〉,rk( ~O

tint[k],

~Otext[k], τ) = 0.Moreover, externally caught violations should have a

worse impact on reputation than internally detected vio-lations, otherwise the organization has a trivial incentivenever to inspect. Formally, rk has the following property.If for any two realized outcomes ~Ol and ~Oj at rounds land j, we have ~Olint[k] +

~Olext[k] = ~Ojint[k] +~Ojext[k]

(i.e., same number of total violations of type k in roundsj and l) and ~Olext[k] >

~Ojext[k] (i.e., for type k, the numberof violations caught externally is more than the numbercaught internally) then for any τ such that 0 ≤ τ < m,rk( ~O

l[k], τ) > rk( ~Oj [k], τ).

We use rk to define a measure of reputation. Because,by construction, violations only affect at most m rounds ofplay, we can write the reputation R0 of the organization atround t as a random variable function of the probabilisticoutcomes ~Ot, ..., ~Ot−m+1:

R0(~Ot, ..., ~Ot−m+1) = R−

K∑k=1

t∑j=t−m+1

rk(~Oj [k], t− j) ,

where R is the maximum possible reputation. We assumethat at the start of the game the reputation is R, and thatrk’s are so that R0 is always non-negative.

We cannot, however, directly use R0 in our utility func-tion. Indeed, R0 is history-dependent, and the repeated gameformalism requires that the utility function be independentof past history. Fortunately, a simple construction allowsto closely approximate the actual reputation, while at thesame time removing dependency on past events. Considerthe following function R:

R(~Ot) = R−K∑k=1

m−1∑j=0

rk(~Ot[k], j) .

Rather than viewing reputation as a function of violationsthat occurred in the past, in round t, the reputation functionR instead immediately accounts for reputation losses thatwill be incurred in the future (in rounds t+ τ , 0 ≤ τ < m)due to violations occurring in round t.

While R and R0 are different reputation functions, whenwe compute the difference of their averages over T rounds,denoting by ~vmax the maximum possible number of viola-tions, we obtain:

1

T

t+T∑τ=t

|R(~Oτ )−R0(~Oτ , . . . , ~Oτ−m)| ≤

1

T

K∑k=1

m−1∑j=1

j · rk(~vmax(k), j) .

The right-hand side of the above inequality goes to zero asT grows large. Hence, using R to model reputation insteadof R0 does not significantly impact the utility function ofthe defender. We define the utility function at round t in therepeated game by the random variable

Lt(〈~at, ~vt〉, ~st) = R(~Ot)− ~C · ~st .

Since utility gains are only realized through loss reduction,we will equivalently refer to L as a loss function from hereon.

An example of the loss of reputation function rk isrk(O, t) = ck(Oint + 2 × Oext)δ

t for 0 ≤ t < m andrk(O, t) = 0 for t ≥ m, where δ ∈ (0, 1). Observe that rkdecreases with t and puts more weight on external violations.Also for the same number of violations and same valuefor argument t rk has different values for different typesof violations due to ck that varies with the types. Then,considering only one type of violation, the loss function canbe written as

Lt(〈~at, ~vt〉, ~st) = R− c1m−1∑j=0

(Otint+2×Ot

ext)δj − ~C ·~st .

Observe that we can expand the summation in the aboveequation to get c1(1 + δ... + δm−1)Ot

int + 2c1(1 + δ... +

δm−1)Otext. Then let Rint = c1(1 + δ... + δm−1) and

similarly let Rext = 2c1(1 + δ... + δm−1). We can rewritethe loss equation above as

Lt(〈~at, ~vt〉, ~st) = R−Rint ·Otint −Rext ·Ot

ext − ~C · ~st .

III. AUDIT MECHANISM AND PROPERTY

In this section, we present our audit mechanism RMAand the main theorem that characterizes its property. RMAprescribes the number of tasks of each type that the defendershould inspect in each round of the repeated game. Theproperty compares the loss incurred by the defender whenshe follows RMA to the loss of a hypothetical defender whohas perfect knowledge of how many violations of each typeoccurred in each round, but must select one fixed action ~s toplay in every round. In particular, we obtain exact boundson the defender’s regret and demonstrate that the averageregret across all rounds converges to a small value relativelyquickly.

A. Audit Mechanism

Our Regret Minimizing Audit (RMA) mechanism ispresented as Algorithm 1. In each round of the game,RMA prescribes how many tasks of each type the defendershould inspect. It does so by maintaining weights for eachpossible defender action (referred to as “experts” followingstandard terminology in the learning literature) and pickingan action with probability proportional to the weight of thataction. For example, in a hospital audit, with two types oftasks—celebrity record access and regular record access—the possible defender actions ~s are of the form 〈k1, k2〉meaning that k1 celebrity record accesses and k2 regularrecord accesses are inspected. The weights are updated basedon an estimated loss function, which is computed from theobserved loss in each round. γ is a learning parameter forRMA. Its value is less than but close to 1. We show howto choose γ in sub-section III-B.

RMA is fast and could be run in practice. Specifically,the running time of RMA is O (N) per round where N isthe number of experts.

In more detail, RMA maintains weights wt~s for all ex-perts [10]. wt~s is the weight of the expert before round thas been played. Initially, all experts are equally weighted.In each round, an action is probabilistically selected for thedefender. As discussed in the Section II there are two factorsthat constrain the set of actions available to the defender:the number of tasks performed by the adversary and thebudget available for audits. In our hospital example we hadthe feasible audit space as {〈x, y〉 | 4x + 5y ≤ 1500, 0 ≤x ≤ 250, 0 ≤ y ≤ 500}. These considerations motivate thedefinition of the set AWAKEt of experts that are awake inround t (Step 1). Next, from this set of awake experts, one ischosen with probability pt~s proportional to the weight of thatexpert (Steps 2, 3, 4). Continuing our hospital example, 250celebrity record accesses and 500 regular record accesses

Algorithm 1 RMA• Initialize: Set w0

~s = 1 for each expert.• Select Move: On round t let 〈~at, ~vt〉 denote the action

of the adversary.1) Set

AWAKEt = {~s : ~s ≤ ~at ∧ ~C · ~s ≤ Bt } .

2) SetW t =

∑~s∈AWAKEt

wt~s .

3) Set

pt~s =wt~sW t

,

for ~s ∈ AWAKE. Otherwise set pt~s = 0.4) Play ~s with probability p~s (randomly select one

expert to follow).

• Estimate loss function: Set L = est(~Ot, ~st

).

• Update Weights: For each ~s ∈ AWAKEt update

wt+1~s = wt~sγ

Lt(~s)−γLt(RMA) ,

where Lt(RMA) =∑~s p

t~sL

t(~s), is the expected (esti-mated) loss of the algorithm.

will be inspected with probability 0.3 in a round if the expert〈250, 500〉 is awake in that round and its weight divided bythe total weight of all the experts who are awake is 0.3.Technically, this setting is close to the setting of sleepingexperts in the regret minimization literature [5], [11].

However, we also have to deal with imperfect information.Since only one action (say ~st) is actually played by thedefender in a round, she observes an outcome ~Ot for thataction. For example, the 〈250, 500〉 inspection might haveidentified 5 celebrity record access violations and 3 regularrecord access violations internally; the same number ofviolations may have been detected externally. Based on thisobservation, RMA uses an algorithm est to compute anestimated loss function L for all experts (not just the oneshe played). We describe properties of the loss functionfor which this estimation is accurate in subsection V. Wealso provide an example of a natural loss function thatsatisfies these properties. Finally, the estimated loss functionis used to update the weights for all the experts who areawake. Intuitively, the multiplicative weight update ensuresthat the weights of experts who performed better thantheir current distribution increase and the weights for thosewho performed worse decrease. In RMA the weight forexpert ~s increases when Lt(~s) − γLt(RMA) is negative,i.e., Lt(~s) < γLt(RMA), and since γ is close to 1, the lossof expert ~s is less than the loss of RMA, i.e., the expert ~sperformed better than RMA.

B. Property

The RMA mechanism provides the guarantee that the de-fender’s regret is minimal. Regret is a standard notion fromthe online learning literature. Intuitively, regret quantifies thedifference between the loss incurred by the defender whenshe follows RMA and the loss of a hypothetical defenderwho has perfect knowledge of how many violations of eachtype occurred in each round, but must select one fixedaction (or expert) to play in every round. Our main theoremestablishes exact bounds on the defender’s regret.

Let T denote the total number of rounds played, I(t)be a time selection function whose output is either 0 or 1,Lt(~s) = Lt (〈~at, ~vt〉, ~s) be the loss function at time t afterthe adversary has played 〈~at, ~vt〉, and pt~s be the probabilityof choosing the action ~s in round t while following RMA.We define the total loss of RMA as follows:

Loss (RMA, I) =T∑t=1

∑~s

I(t)pt~sLt(~s) .

Similarly for each fixed expert ~s we set the following lossfunction

Loss (~s, I) =

T∑t=1

I(t)Lt(~s) .

We use Regret (RMA, ~s) to denote our regret in hindsightof not playing the fixed action ~s when it was available.Formally,

Regret (RMA, ~s) = Loss (RMA, I~s)− Loss (~s, I~s) .

Here, I~s(t) is the time selection function that selects onlythe times t that action ~s is available.

As before, we use N to denote the total number of fixedactions available to the defender. If T is known in advancewe can obtain the bound in Theorem 1 below by setting γto be 1−

√2 lnNT . Otherwise, if T is not known in advance

we can dynamically tune γ to obtain similar bounds. SeeRemark 7 for more details on dynamic tuning.

Theorem 1. For all ε > 0,

Pr

∃~s, Regret(RMA,~s)T ≥ 2

√2 lnN

T +

2

√2 ln( 4N

ε )T + 2

T lnN

≤ ε .Remark 1. To prove this theorem we need to make severalreasonable assumptions about the accuracy of our lossfunction estimator est. We discuss these assumptions insection V. We also assume that losses have been scaled sothat Lt(x) ∈ [0, 1].

Remark 2. This bound is a worst case regret bound. Theguarantee holds against any attacker. Regret may typicallybe lower than this, e.g. when hospital employees do notbehave adversarially.

0 200 400 600 800 1000 1200 1400 1600 1800 20000

0.10.20.30.40.50.60.70.80.9

11.11.21.31.41.51.61.7

Time

Aver

age

Reg

ret

N=5 ε=0.01N=100 ε=0.01N=5 ε=0.10N=100 ε=0.10

Figure 2. Worst case Average Regret vs Time for different values of Nand ε

In order to understand what this bound means, considerthe following example scenario. Suppose that an employeeat a hospital can access two types of medical records—celebrity or regular. The defender can choose to inspectaccesses of a certain type lightly, moderately, or heavily. Inthis case, the defender has N = 9 possible pairs of actionsin each round. If the hospital performs daily audits (whichsome hospitals currently do for celebrity record accesses)over a 5 year period, then T = 365 × 5 = 1825. Forsimplicity, assume that each action ~s is available everyday. In this case, the theorem guarantees that except withprobability ε = 1

100 , the average regret of RMA does notexceed 29%:

Regret (RMA, ~s)

T< 0.29 .

Note that there are several existing algorithms for regretminimization in games with imperfect information [9], [12]–[14]. These algorithms do guarantee that as T → ∞ theaverage regret will tend to 0, but the convergence rate isunacceptably slow for our audit model (see Section VII-Bfor a more detailed comparison). The convergence rate ofRMA is significantly faster. Also, in contrast to prior work,we focus on exact (not asymptotic) regret bounds for ouralgorithm. This is important because in practice we careabout the value of the bound for a fixed value of T (as inthe example above), not merely that it tends to 0 as T →∞.

IV. DISCUSSION

A few characteristics of the model and algorithms de-scribed above may not necessarily be evident from thetechnical presentation given above, and warrant further dis-cussion.

Figure 2 shows the variation of average regret with timefor different values of N and ε. As can be seen, the RMAalgorithm produces smaller average regret bounds for highervalues of time T and lower values of N . In other words,and quite intuitively, a high audit frequency ensures lowregret. Some medical centers carry out audits every week;

RMA is particularly appropriate for such high frequencyaudits. Lower values of N means that RMA’s performanceis compared to fewer fixed strategies and hence yields lowerregret. One situation in which N could be low is whenthe fixed strategies correspond to discrete levels of auditscoverage used by the organization. Also, higher values of εyield smaller average regret bounds. Indeed, ε is a measureof uncertainty on the stated bound. Thus, when higher valuesof ε are tolerable, we obtain tighter regret bounds, but at theexpense of greater uncertainty on whether those bounds aremet.

We have already noted that all actions may not be avail-able at all times. The result in Theorem 1 bounds the averageregret taken over all rounds of the game. It is easy to modifythe proof of Theorem 1 to obtain average regret bounds foreach expert such that the average is taken over the timefor which that expert is awake. The bound thus obtainedis of the same order as the bound in Theorem 1, but allinstance of T in Theorem 1 are replaced by T~s, where T~s isthe time for which expert ~s is awake. Similar result for thetraditional sleeping experts setting can be found in Blumet al. [11]. The modified bound equation exhibits the factthat the average regret bound for a given inspection vector(average taken over the time for which that inspection wasavailable) depends on how often this inspection vector isavailable to the defender. If a given inspection vector isonly available for a few audit cycles, the average regretbound may be relatively high. The situation is analogous towhitewashing attacks [15], where the adversary behaves ina compliant manner for many rounds to build up reputation,attacks only once, and immediately leaves the game afterthe attack. For instance, a spy infiltrates an organization,becomes a trusted member by behaving as expected, andthen suddenly steals sensitive data. However, we argue that,rather than being an auditing issue, whitewashing attackscan be handled by a different class of mechanisms, e.g., thatprevent the adversary from vanishing once she has attacked.

Furthermore, RMA guarantees low average regret com-pared to playing a fixed action (i.e., inspection vector) inthe audit cycle in which that action was available; it doesnot guarantee violations will not happen. In particular, if acertain type k of violation results in catastrophic losses forthe organization (e.g., losses that threaten the viability of theorganization itself), tasks of type k should always be fullyinspected.

While the discussion surrounding the RMA algorithmfocused only on one kind of experts (which recommendhow many tasks of each type to inspect in any round), RMAapplies more generally to any set of experts. For example, wecould include an expert who recommends a low inspectionprobability when observed violations are below a certainthreshold and a higher inspection probability when observedviolations are above that threshold. Over time, the RMAalgorithm will perform as well as any such expert. Note.

however, that if we make the size (N ) of the set of expertstoo large, the regret bounds from Theorem 1 will be worse.Thus, in any particular application, the RMA algorithm willbe effective if appropriate experts are chosen without makingthe set of experts too large.

Finally, we note that non-compliance with external privacyregulations may not only cause a loss of reputation forthe organization, but can also result in fines being leviedagainst the organization. For instance, a hospital found inviolation of HIPAA provisions [4] in the United States willlikely face financial penalties in addition to damaging itsreputation. We can readily extend our model to accountfor such cases, by forcing the defender (organization) toperform some minimum level of audit (inspections) to meetthe requirements stipulated in the external regulations. Forexample, we can constrain the action space available to thedefender by removing strategies such as “never inspect.” Aslong as the budgetary constraints allow the organization toperform inspections in addition to the minimal level of auditrequired by law, the guarantees provided by RMA still hold.Indeed, Theorem 1 holds as long as there is at least oneawake expert in each round.

V. ESTIMATING LOSSES

RMA uses a function est(~Ot, ~st

)to estimate the loss

function Lt. In this section, we formally define twoproperties—accuracy and independence; the regret boundin Theorem 1 holds for any estimator function that satisfiesthese two properties. We also provide an example of a lossfunction estimator algorithm that provably satisfies theseproperties, thus demonstrating that such estimator functionscan in fact be implemented. The use of an estimator functionand the characterization of its properties is a novel contri-bution of this paper that allows us to achieve significantlybetter bounds than prior work in the regret minimizationliterature for repeated games of imperfect information (seeSection VII for a detailed comparison).

Estimator Properties: The function Lt = est(~Ot, ~st

)should be efficiently computable for practical applications.Note that the loss estimation at time t depends on theoutcome ~Ot (violations of each type detected internally andexternally) and the defender’s action ~st at time t. Intuitively,the function outputs an estimate of the loss function byestimating the number of violations of each type based onthe detected violations of that type and the probability ofinspecting each action of that type following the defender’saction.

For each defender action (expert) ~s, we define the randomvariable

Xt~s = Lt(~s)− Lt(~s).

Intuitively, Xt~s is a random variable representing our esti-

mation error at time t after the actions 〈~vt,~at〉 and ~st havebeen fixed by the adversary and the defender respectively.

Because we have assumed that our loss functions arescaled so that Lt(~s),Lt(~s) ∈ [0, 1] we have Xt

~s ∈ [−1, 1].This property of Xt

~s is useful in bounding the regret as wediscuss later.

Formally, we assume the following properties about est:

1) Accuracy: E[Xj~s

]= 0 for 0 ≤ j ≤ T .

2) Independence: ∀~s, X1~s, . . . ,X

T~s are all independent

random variables.Any estimation scheme est that satisfies both properties

can be plugged into RMA yielding the regret bound inTheorem 1. Informally, accuracy captures the idea that theestimate is accurate in an expected sense while independencecaptures the idea that the error in the estimate in each roundis independent of the error in all other rounds. We motivatethese properties by way of an example.

Remark 3. In fact if our estimation scheme only satisfied δ-accuracy, i.e.,

∣∣∣E [Xj~s

]∣∣∣ < δ, then we could still guaranteethat the average regret bounds from Theorem 1 still hold withan extra additive term δ. Formally, the following propertyholds: for all ε ∈ (0, 1)

Pr

∃~s, Regret(RMA,~s)T ≥ δ + 2

√2 lnN

T +

2

√2 ln( 4N

ε )T + 2

T lnN

≤ ε .Example Loss Function: We return to our running

example of the hospital. We use the example reputation(loss) function from the previous section:

Lt(~s) = R−(~Otint · ~Rint + ~Ot

ext · ~Rext + ~C · ~s).

To simplify our presentation we assume that there is onlyone type of violation. It is easy to generalize the loss functionthat we present in this example to include multiple types ofviolations.

Lt(s) = R−(Otint ×Rint +Ot

ext ×Rext + C × s).

Here Otint represents the number of violations caught in-

ternally after the actions 〈vt, at〉 and st are played by theadversary and the defender respectively, Rint (resp. Rext)captures the damage to the hospital’s reputation when aviolation is caught internally (resp. externally), and C isthe cost of performing one inspection. Notice that

E[Otext ×Rext

]= p

(vt − E

[Otint

])×Rext ,

where p is the probability that an undetected violation getscaught externally. Therefore,

E[Lt(s)

]= R−

(E[Otint

](Rint − p×Rext)

+p× vt ×Rext + C × s).

We can set R′ = (Rint − p×Rext) and then

E[Lt(s)

]= R−

(E[Otint

]×R′ + p× vt ×Rext + C × s

).

In our loss model, we allow the defender to use any recom-mendation algorithm REC that sorts all at actions at time tand probabilistically recommends st actions to inspect. Welet pd ≤ 1 denote the probability that the dth inspectionresults in a detected violation, where this probability isover the coin flips of the recommendation algorithm REC.Because this probability is taken over the coin flips of RECthe outcome Ot

int is independent of previous outcomes once〈~at, ~vt〉, ~st have been fixed.

For example, a naive recommendation algorithm RECmight just select a few actions uniformly at random andrecommend that the defender inspect these actions. In thiscase pj = vt

at for each j. (Remember that in this examplewe consider only one type of violation, so vt and at arescalars). If REC is more clever, then we will have p1 > vt

at .In this case the pj’s will also satisfy diminishing returns(pj > pj+1).

We assume that inspection is perfect, i.e., if we inspect aviolation it will be caught with probability 1. Thus, if weinspect all at actions we would catch all vt violations, i.e.,

at∑j=1

pj = vt.

Set pj = vt(

1−β1−βat

)βj−1, where the parameter β could be

any value in (0, 1). Notice thata∑j=1

pj = vt(

1− β1− βat

) at−1∑j=0

βj = vt ,

and pj > pj+1 so our model does satisfy diminishingreturns. Furthermore, if β = max{1− 1

at ,12} then we have

pj ≤ 1 for each j. We can express E [Oint] =∑st

i=1 pj .

E[Lt(s)

]= R−

R′ st∑i=1

pj + p× vt ×Rext + C × s

.

Example Loss Estimator: Our loss function estimatorest (Ot

int, st) is given in Algorithm 2.

Algorithm 2 Example: est (Otint, s

t)

• Input: Otint, s

t.

• Estimate vt: Set vt := 1−βat

1−βst Otint.

• Compute L: Set

L(x) := R−

(R′ × vt ×

∑xj=1

(1−β1−βa β

j−1)

+p× vt ×Rext + C × x

).

• Output: Lt.

Assuming that the defender understands the accuracy ofhis recommendation algorithm REC1 β is a known quantity

1If the algorithm recommends actions uniformly at random, then theaccuracy is certainly understood.

so that this computation is feasible and can be performedquickly. Independence of the random variables Xt

s followsfrom the independence of Ot

int. Now we verify that ourestimator satisfies our accuracy condition.

Claim 1. When Lt = est (Otint, s

t) from Algorithm 2E [Xt

s] = 0.

The proof can be found in Appendix A. The main insighthere is that because of the way the probabilities pj’s arescaled, the expectation of the estimated number of violationsis equal to the number of actual violations, i.e. E [vt] = vt.Consequently, the expected value of the error turns out tobe 0, thus satisfying the accuracy property.

Remark 4. If there are multiple types of actions then wecan estimate vtk, the number of violations of type k at timet, separately for each k. To do this est should substitute~Otint[k] and ~st[k] for Ot

int and vt and set

vtk :=

(1− βat

1− βst

)Otint .

Then we have

Lt(x) = R−

(~R′ · 〈vtk〉k ×

1−βx[k]1−βa

+p× 〈vtk〉k · ~Rext + ~C · ~x

).

VI. PROOF OUTLINE

In this section, we present the outline of the proof ofour main theorem (Theorem 1) which establishes highprobability regret bounds for RMA. The complete proof isin the appendices. The proof proceeds in two steps:

1) We first prove that RMA achieves low regret withrespect to the estimated loss function using standardresults from the literature on regret minimization [5],[11].

2) We then prove that with high probability the differencebetween regret with respect to the actual loss functionand regret with respect to the estimated loss functionis small. This step makes use of the two properties—accuracy and independence—of the estimated lossfunction est presented in the previous section and isthe novel part of the proof. The key technique used inthis step are the Hoeffding inequalities [16].

Let T denote the total number of rounds played, T~s denotethe total number of rounds that the action ~s was awake andI as before is a time selector function. We define

Loss (RMA, I) =T∑t=1

∑~s

I(t)pt~sLt(~s) ,

to be our total estimated loss. Notice that Loss is the sameas Loss except that we replaced the actual loss function Lt

with the estimated loss function Lt. We define Loss (~s, I)

and Regret (RMA, ~s) in a similar manner by using theestimated loss function.

Lemma 1 and Lemma 2 below bound the regret withrespect to the estimated loss function. They are based onstandard results from the literature on regret minimiza-tion [5], [11]. We provide full proofs of these results inAppendix C.

Lemma 1. For each expert ~s we have

Regret(RMA, ~s) ≤ 1

L− 1T + 2L lnN ,

where N is the total number of experts and γ, our learningparameter has been set to γ = 1− 1

L

Remark 5. We would like to bound our average regret:

Regret(RMA, ~s)

T~s.

There is a trade-off here in the choice of L. If L is too large,then L lnN will be large. If L is too small, then 1

L−1T willbe large. If we know T in advance, then we can tune ourlearning parameter γ to obtain the best bound by setting

L =

√T

2 lnN+ 1 .

After substituting this value for L in Lemma 1, weimmediately obtain the following result:

Lemma 2. For each expert ~s we have

Regret(RMA, ~s) ≤ 2√2T lnN + 2 lnN ,

where N is the total number of experts and learningparameter γ = 1−

√2 lnNT .

Remark 6. This shows that RMA can achieve low regretwith respect to the estimated loss functions Lt. This com-pletes step 1 of the proof. We now move on to step 2.

Notice that we can write our actual loss function(Loss (RMA, I)) in terms of our estimated loss function(Loss (RMA, I)

)and Xt

~s.

Fact 1.

Loss (RMA, I) = Loss (RMA, I) +T∑t=1

I(t)Xt~s .

We know that E [Xt~s] = 0 (from the accuracy property

of the loss estimation function) and that X1~s, . . .X

T~s are

independent (from the independence property), so we canapply the Hoeffding inequalities to bound

∑tX

t~s obtaining

Lemma 3. Appendix B contains a description of the inequal-ities and the full proof of the following lemma.

Lemma 3.

Pr[∃~s,Regret (RMA, ~s)− Regret (RMA, ~s) ≥ 2K

]≤ ε ,

where K =√

2T ln(4Nε

).

After straightforward algebraic substitution we can obtainour main result in Theorem 1 by combining Lemma 3 withLemma 2 (see Appendix D).

Observe that the optimal value of γ is dependent on T .But, it is conceivable that the time T for which the gameis played is not known in advance. The following remarkshows that we can overcome this problem by choosing adynamic value for γ. This makes RMA usable in the realworld.

Remark 7. Even if we don’t know T in advance we cantune γ dynamically using a technique from [17]. We set

γt =1

1− αt,

where

αt =

√2

lnN

Lt − 1.

where Lt is the minimum loss of any expert till time t(calculated using the estimated loss). Before playing round twe recompute the weights wt~s, pretending that our learningparameter γ had been set to γt from the beginning i.e.

wt~s = γ∑ti=1 I~s(t)(L

t(~s)−γtLt(RMA))t .

In this case our final guarantee (similar to Theorem 1) wouldbe that:

Pr

∃~s,Regret(RMA,~s)

T ≥ 2√

2 lnNT +

2

√2 ln( 4N

ε )T + 10

T lnN+4T (lnN) (ln(1 + T ))

≤ ε .VII. RELATED WORK

A. Auditing in Computer Security

A line of work in computer security uses evidencerecorded in audit logs to understand why access was grantedand to revise access control policies if unintended accessesare detected [6], [18], [19]. In contrast, we use audits todetect violations of policies, such as those restricting infor-mation use to specified purposes, that cannot be enforcedusing access control mechanisms.

Cederquist et al. [20] present logical methods for enforc-ing a class of policies, which cannot be enforced usingpreventive access control mechanisms, based on evidencerecorded on audit logs. The evidence demonstrating policycompliance is presented to the auditor in the form of a proofin a logic and can be checked mechanically. In contrast, ourfocus is on policies that cannot be mechanically enforced intheir entirety, but require involvement of human auditors. Inaddition, the new challenges in our setting arises from theimperfect and repeated nature of audits.

Zhao et al. [21] recognize that rigid access control cancause loss in productivity in certain types of organizations.They propose an access control regime that allows all accessrequests, but marks accesses not permitted by the policy for

posthoc audit coupled with punishments for violating policy.They assume that the utility function for the organizationand the employees are known and use a single shot game toanalyze the optimal behavior of the players. Our approachof using a permissive access control policy coupled withaudits is a similar idea. However, we consider a worst-caseadversary (employee) because we believe that it is difficultto identify the exact incentives of the employee. We furtherrecognize that the repeated nature of interaction in audits isnaturally modeled as a repeated game rather than a one-shotgame. Finally, we restrict the amount of audit inspectionsbecause of budgetary constraints. Thus, our game modelis significantly more realistic than the model of Zhao etal. [21].

Cheng et al. [22], [23] also start from the observation thatrigid access control is not desirable in many contexts. Theypropose a risk-based access control approach. Specifically,they allocate a risk budget to each agent, estimate therisk of allowing an access request, and permit an agentto access a resource if she can pay for the estimated riskof access from her budget. Further, they use metaheuristicssuch as genetic programming to dynamically change thesecurity policy, i.e. change the risk associated with accessesdynamically. We believe that the above mechanism mitigatesthe problem of rigid access control in settings such as ITsecurity risk management, but is not directly applicablefor privacy protection in settings such as hospitals wheredenying access based on privacy risks could have negativeconsequences on the quality of care. Our approach to theproblem is fundamentally different: we use a form of risk-based auditing instead of risk-based access control. Also,genetic programming is a metaheuristic, which is knownto perform well empirically, but does not have theoreticalguarantees [24]. In contrast, we provide mechanisms withprovable guarantees. Indeed an interesting topic for futurework is to investigate the use of learning-theoretic techniquesto dynamically adjust the risk associated with accesses in aprincipled manner.

Guts et al. [25] consider an orthogonal problem in thisspace. They provide a characterization of auditable proper-ties and describe how to type-check protocols to guaranteethat they log sufficient evidence to convince a judge that anauditable property was satisfied on a protocol run.

Garg et al. [26] present an algorithm that mechanically en-forces objective parts of privacy policies like HIPAA basedon evidence recorded in audit logs and outputs subjectivepredicates (such as beliefs) that have to be checked byhuman auditors. Combining the their algorithm with oursprovides an end-to-end enforcement mechanism for policiesof this form.

B. Regret Minimization

A regret minimization algorithm ALG is a randomizedalgorithm for playing in a repeated game. Our algorithm

RMA is based on the weighted majority algorithm [10]for regret minimization. The weighted majority maintainsweights w~s for each of the N fixed actions of the defender.wt~s is the weight of the expert before round t has beenplayed. The weights determine a probability distribution overactions, pt~s denotes the probability of playing ~s at timet. In any given round the algorithm attempts to learn theoptimal distribution over actions by increasing the weightsof experts that performed better than its current distributionand decreasing the weights of experts that performed worse.

1) Sleeping Experts: In the setting of [10] all of theactions are available all of the time. However, we areworking in the sleeping experts model where actions maynot be available every round due to budget constraints.Informally, in the sleeping experts setting the regret of RMAwith respect to a fixed action ~s in hindsight is the expecteddecrease in our total loss had we played ~s in each of the T~srounds when ~s was available.

There are variations of the weighted majority algorithmthat achieve low regret in the sleeping experts setting [5],[11]. These algorithms achieve average regret bounds:

∀~s, Regret (Alg, s)

T~s= O

(√T logN

T~s

).

In fact RMA is very similar to these algorithms. However,we are interested in finding exact (not asymptotic) bounds.We also have to deal with the imperfect information in ourgame.

2) Imperfect Information: In order to update its weightafter round t, the weighted majority algorithm needs to knowthe loss of ever available defender action ~s . Formally, thealgorithm needs to know Lt(~s) for each ~s ∈ AWAKEt.However, we only observe an outcome ~Ot, which allowsus to compute

Lt(~st) = R(~Ot)− ~C · ~st,

the loss for the particular action ~st played by the defenderat time t. There are several existing algorithms for regretminimization in games with imperfect information [9], [12]–[14]. For example, [9] provides an average regret bound of

∀~s, Regret(Alg, ~s)

T= O

(N1/3 logN

3√T

).

It is acceptable to have logN in the numerator, but the N1/3

term will make the algorithm impractical in our setting. Theaverage regret still does tend to 0 as T →∞, but the rate ofconvergence is much slower compared to the case when onlylogN in present in the numerator. Other algorithms [12]–[14] improve this bound slightly, but we still have the N1/3

term in the numerator. Furthermore, [9] assumes that eachaction ~s is available in every round. There are algorithms thatdeal with sleeping experts in repeated games with imperfectinformation, but the convergence bounds get even worse.

Regret minimization techniques have previously beenapplied in computer security by Barth et al. [27]. However,that paper addresses a different problem. They show thatreactive security is not worse than proactive security inthe long run. They propose a regret minimizing algorithm(reactive security) for allocation of budget in each round sothat the attacker’s “return on attack” does not differ muchfrom the case when a fixed allocation (proactive security)is chosen. Their algorithm is not suitable for our auditsetting due to imperfect information and sleeping experts.In their work, the defender learns the attack path playedby the adversary after each round, and by extension hasperfect knowledge of the loss function for that round. Bycontrast, RMA must work in the imperfect informationsetting (see section VII-B2). Also, their model considersunknown attack paths that get discovered over time. Thisis a special subcase of the sleeping experts setting, wherean expert is awake in every round after she wakes up.They extend the multiplicative weight update algorithm [10]to handle the special case. In our setting experts may beavailable in one round and unavailable in next. RMA wasdesigned to work in this more general setting.

VIII. CONCLUSION AND FUTURE WORK

We presented a principled approach to audits in organi-zations, such as hospitals, with permissive access controlregimes. We modeled the interaction between the defender(e.g., hospital auditors) and the adversary (e.g., hospitalemployees) as a repeated game. The model takes pragmaticconsiderations into account, and considers a powerful worst-case adversary. We formulate a desirable property of theaudit mechanism in this model based on the concept ofregret in learning theory and present an efficient audit mech-anism that provably minimizes regret for the defender. Thismechanism learns from experience to guide the defender’sauditing efforts. The regret bound is significantly better thanprior results in the learning literature. The stronger boundis important from a practical standpoint because it impliesthat the recommendations from the mechanism will convergefaster to the best fixed auditing strategy for the defender.

There are several directions for future work. We plan todevelop similar results with a weaker (but still realistic)adversary model. Specifically, the regret bounds guaranteedby our algorithm hold even if an adversary controls theactions of all the employees in a hospital. It is reasonable tobelieve that not all employees behave adversarially. We planto consider an alternative model in which some employeesare adversarial, some are selfish and others are well-behaved(cf. [28]). Such a model could enable us to develop auditmechanisms that provide better bounds on the organization’sregret. We also plan to implement such an audit mechanismand evaluate its performance over real hospital audit logs.

REFERENCES

[1] G. Hulme, “Steady Bleed: State of HealthCare DataBreaches,” September 2010, InformationWeek.

[2] HIPPA Enforcement, 2010 (accessed November 19,2010).[Online]. Available: http://www.hhs.gov/ocr/privacy/hipaa/enforcement/index.html

[3] H. DeYoung, D. Garg, L. Jia, D. Kaynar, and A. Datta,“Experiences in the logical specification of the HIPAA andGLBA privacy laws,” in Proceedings of the 9th annual ACMWorkshop on Privacy in the Electronic Society (WPES), 2010.

[4] US Congress, “Health Insurance Portability and Account-ability Act of 1996, Privacy Rule,” 45 CFR 164, 2002,available at http://www.access.gpo.gov/nara/cfr/waisidx 07/45cfr164 07.html.

[5] A. Blum and Y. Mansour, “Learning, regret minimization, andequilibria,” Algorithmic Game Theory, pp. 79–102, 2007.

[6] B. W. Lampson, “Computer security in the real world,” IEEEComputer, vol. 37, no. 6, pp. 37–46, 2004.

[7] D. J. Weitzner, H. Abelson, T. Berners-Lee, J. Feigenbaum,J. A. Hendler, and G. J. Sussman, “Information accountabil-ity,” Commun. ACM, vol. 51, no. 6, pp. 82–87, 2008.

[8] D. Fudenberg and J. Tirole, Game theory. MIT Press, 1991.

[9] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. Schapire, “Thenonstochastic multiarmed bandit problem,” SIAM Journal onComputing, vol. 32, no. 1, pp. 48–77, 2003.

[10] N. Littlestone and M. K. Warmuth, “The weighted majorityalgorithm,” Inf. Comput., vol. 108, no. 2, pp. 212–261, 1994.

[11] A. Blum and Y. Mansour, “From external to internal regret,”in COLT, 2005, pp. 621–636.

[12] V. Dani and T. Hayes, “Robbing the bandit: Less regret inonline geometric optimization against an adaptive adversary,”in Proceedings of the seventeenth annual ACM-SIAM sympo-sium on Discrete algorithm. ACM, 2006, p. 943.

[13] M. Zinkevich, M. Johanson, M. Bowling, and C. Piccione,“Regret minimization in games with incomplete information,”Advances in Neural Information Processing Systems, vol. 20,pp. 1729–1736, 2008.

[14] B. Awerbuch and R. Kleinberg, “Online linear optimizationand adaptive routing,” Journal of Computer and SystemSciences, vol. 74, no. 1, pp. 97–114, 2008.

[15] M. Feldman, C. H. Papadimitriou, J. Chuang, and I. Sto-ica, “Free-riding and whitewashing in peer-to-peer systems,”IEEE Journal on Selected Areas in Communications, vol. 24,no. 5, pp. 1010–1019, 2006.

[16] W. Hoeffding, “Probability inequalities for sums of boundedrandom variables,” Journal of the American Statistical Asso-ciation, vol. 58, no. 301, pp. 13–30, 1963.

[17] P. Auer, N. Cesa-Bianchi, and C. Gentile, “Adaptive and self-confident on-line learning algorithms,” Journal of Computerand System Sciences, vol. 64, no. 1, pp. 48–75, 2002.

[18] J. A. Vaughan, L. Jia, K. Mazurak, and S. Zdancewic,“Evidence-based audit,” in CSF, 2008, pp. 177–191.

[19] L. Bauer, S. Garriss, and M. K. Reiter, “Detecting andresolving policy misconfigurations in access-control systems,”in SACMAT, 2008, pp. 185–194.

[20] J. G. Cederquist, R. Corin, M. A. C. Dekker, S. Etalle, J. I. denHartog, and G. Lenzini, “Audit-based compliance control,”Int. J. Inf. Sec., vol. 6, no. 2-3, pp. 133–151, 2007.

[21] X. Zhao and M. E. Johnson, “Access governance: Flexibilitywith escalation and audit,” in HICSS, 2010, pp. 1–13.

[22] P.-C. Cheng and P. Rohatgi, “IT Security as Risk Manage-ment: A Research Perspective,” IBM Research Report, vol.RC24529, April 2008.

[23] P.-C. Cheng, P. Rohatgi, C. Keser, P. A. Karger, G. M.Wagner, and A. S. Reninger, “Fuzzy Multi-Level Security :An Experiment on Quantified Risk-Adaptive Access Control,”in Proceedings of the IEEE Symposium on Security andPrivacy, 2007.

[24] M. D. Vose, A. H. Wright, and J. E. Rowe, “Implicit paral-lelism,” in IN GECCO (2003), 2003, pp. 1505–1517.

[25] N. Guts, C. Fournet, and F. Z. Nardelli, “Reliable evidence:Auditability by typing,” in ESORICS, 2009, pp. 168–183.

[26] D. Garg, L. Jia, and A. Datta, “Policy Monitoring overEvolving Audit Logs: A Logical Method for Privacy PolicyEnforcement,” CMU, Tech. Rep. CMU-CyLab-11-002, Jan-uary 2011.

[27] A. Barth, B. Rubinstein, M. Sundararajan, J. Mitchell,D. Song, and P. Bartlett, “A Learning-Based Approach to Re-active Security,” Financial Cryptography and Data Security,pp. 192–206, 2010.

[28] A. S. Aiyer, L. Alvisi, A. Clement, M. Dahlin, J.-P.Martin, and C. Porth, “Bar fault tolerance for cooperativeservices,” SIGOPS Oper. Syst. Rev., vol. 39, pp. 45–58,October 2005. [Online]. Available: http://doi.acm.org/10.1145/1095809.1095816

APPENDIX

Our main goal is to prove our main theorem from sectionIII-B, showing that our audit mechanism (RMA) achieveslow regret with high probability.

First, we prove an orthogonal result in appendix A. Weprove that our example estimator function from section V isaccurate for the example loss function that we provided.

In appendix B we prove that Lemma 3 holds for anyestimator function est satisfies the accuracy and indepen-dence properties outlined in section V. Therefore, with highprobability the defender’s actual regret will be close to hisestimated regret.

In appendix C we review standard regret bounds from theliterature on regret minimization [5], [11]. We prove that ouralgorithm achieves low regret with respect to our estimatedloss function.

Finally, in appendix D we combine our results to proveour main theorem in section III-B. This theorem shows that,except with probability ε, RMA will achieve low regret.

A. Estimating Losses

Recall that our regret bounds for algorithm 1 depended onthe accuracy of the loss function estimator est. We provethat our example estimator (Algorithm 2) from section V isaccurate.

Reminder of Claim 1. When Lt = est (Otint, s

t) fromalgorithm 2

E[Xts

]= 0 .

Proof: First observe that

E[vt]

=1− βat

1− βstE[Otint

]=

1− βat

1− βstst∑i=1

pj

=1− βat

1− βstst∑i=1

vt1− β1− βat

βj−1

=1− β1− βst

vtst∑i=1

βj−1

= vt .

From which it follows that

E[Xts

]= E

[Lt(~s)

]− E

[Lt(~s)

]= R−R′

x∑i=1

pj − p× vt ×Rext

−C × s− E[Lt(~s)

]= R−R′

x∑i=1

pj − p× vt ×Rext − C × s

−R+R′ × E[vt]×

x∑j=1

(1− β1− βa

βj−1)

+p× E[vt]×Rext + C × s

= R′ × E[vt]×

x∑j=1

(1− β1− βat

βj−1)−R′

x∑i=1

pj

=

R′ × x∑j=1

(1− β1− βat

βj−1v

)−R′ x∑i=1

pj

=

R′ × x∑j=1

pj

−R′ x∑i=1

pj

= 0 .

B. Hoeffding Bounds

Hoeffding Bound [16] bounds the probability of thedeviation of a sum of independent random variables fromthe mean of the sum of random variables. The statementof Hoeffding Bound is as follows: if X1, X2, ..., Xn areindependent real valued random variables and ai ≤ Xi ≤ bi,then for t > 0,

Pr

[∣∣∣∣∣n∑i=1

Xi − E

[n∑i=1

Xi

]∣∣∣∣∣ > t

]≤ 2 exp

(−2t2∑n

i=1(bi − ai)2

).

Claim 2. For every ~s

Pr[∣∣∣Loss (~s, I~s)− Loss (~s, I~s)

∣∣∣ ≥ K] ≤ ε

2N,

where K =√2T ln 4N

ε .

Proof: Notice that we can rewrite

Loss (~s, I~s)− Loss (~s, I~s) =

T∑t=1

I~s(t)Xt~s .

By the independence property of of loss estimator est,the random variables Xt

~s are independent. By the accuracyproperty of our loss function estimator est we have

E

[T∑t=1

I~s(t)Xt~s

]= 0 .

By definition there are exactly T~s times when I~s(t) = 1 sothe sum contains T~s independent random variables. We alsohave −1 ≤ Xt

~s ≤ 1 so we can apply Hoeffding Boundsdirectly to obtain.

Pr

[∣∣∣∣∣T∑t=1

I~s(t)Xt~s

∣∣∣∣∣ ≥ K]≤ 2 exp

(−2K2

22 × T~s

).

Plugging in for K and using the fact that T~s ≤ T ,

Pr

[∣∣∣∣∣T∑t=1

I~s(t)Xt~s

∣∣∣∣∣ ≥ K]≤ 2 exp

(−T ln 4N

ε

T~s

)

≤ 2 exp

(− ln

4N

ε

)=

ε

2N.

Claim 3. For every ~s

Pr[∣∣∣Loss (RMA, ~s)− Loss (RMA, ~s)

∣∣∣ ≥ K] ≤ ε

2N,

where K =√2T ln 4N

ε .

Proof: Notice that we can rewrite

Loss (RMA, ~s)− Loss (RMA, ~s) =T∑t=1

∑~s

I~s(t)pt~sX

t~s .

Set Yt =∑~s p

t~sX

t~s, and observe that the random variables

Yt are independent and that Yt ∈ [−1, 1]. Substituting weget

Loss (RMA, ~s)− Loss (RMA, ~s) =T∑t=1

I~s(t)Yt .

Applying Hoeffding Bounds we have

Pr

[∣∣∣∣∣T∑t=1

I~s(t)Yt

∣∣∣∣∣ > K

]≤ 2 exp

(−2K2

22T~s

).

Set K =√

2T ln 4Nε . Then,

Pr[∣∣∣Loss (RMA, ~s)− Loss (RMA, ~s)

∣∣∣ > K]

= Pr

[∣∣∣∣∣T∑t=1

I~s(t)Yt

∣∣∣∣∣ > K

]

≤ 2 exp

(−2K2

22T~s

)≤ 2 exp

(− ln

4N

ε

)≤ ε

2N.

Lemma 4. Except with probability ε, for all ~s we have∣∣∣Loss (~s, I~s)− Loss (~s, I~s)∣∣∣ < K ,

and ∣∣∣Loss (RMA, ~s)− Loss (RMA, ~s)∣∣∣ < K ,

where K =√2T ln 4N

ε .

Proof: There are N fixed actions ~s and thus 2N totalevents. Applying the union bound to claims 2 and 3 yieldsthe desired result immediately.

Claim 4. Suppose that for all ~s we have∣∣∣Loss (~s, I~s)− Loss (~s, I~s)∣∣∣ < K ,

and ∣∣∣Loss (RMA, ~s)− Loss (RMA, ~s)∣∣∣ < K ,

where K =√

2T ln 4Nε . then for every ~s we have

Regret (RMA, ~s)− Regret (RMA, ~s) ≤ 2K .

Proof: We use the definition of Regret (RMA, ~s) andRegret, and then apply Lemma 4.

Regret (RMA, ~s)− Regret (RMA, ~s)

= (Loss (RMA, ~s)− Loss (~s, I~s))

−(Loss (RMA, ~s)− Loss (~s, I~s)

)≤

∣∣∣Loss (RMA, ~s)− Loss (RMA, ~s)∣∣∣

+∣∣∣Loss (~s, I~s)− Loss (~s, I~s)

∣∣∣≤ 2K .

We are now ready to prove Lemma 3 from section VI.

Reminder of Lemma 3.

Pr[∃~s,Regret (RMA, ~s)− Regret (RMA, ~s) ≥ 2K

]≤ ε ,

where K =√

2T ln(4Nε

).

Proof: We combine claim 4 and Lemma 4.

C. Standard Regret Bounds

We prove upper bounds on our estimated Regret. Theproof techniques used in this section are standard [5], [11].We include them to be thorough. The following claims willbe useful in our proofs.

Claim 5. ∑~s∈AWAKEt

wt~sLt(~s) =

∑~s∈AWAKEt

wt~sLt(RMA) .

Proof: We plug in the definition of Lt(RMA):∑~s∈AWAKEt

wt~sLt(RMA) =

∑~s∈AWAKEt

wt~s∑~σ

Lt (~σ)

=∑

~s∈AWAKEt

wt~s∑~σ

pt~σLt (~σ)

=∑~σ

∑~s∈AWAKEt

wt~spt~σL

t (~σ)

=∑~σ

Lt (~σ)

∑~s∈AWAKEt

wt~spt~σ

=

∑~σ

Lt (~σ)(wt~σ)

Relabel ~σ=

∑~s

Lt (~s)(wt~s).

Claim 6. For all times t,∑~s

wt~s ≤ N .

Proof: Initially, w0~s = 1 so initially the claim holds,∑~s

w0~s = N .

The sum of weights can only decrease. At time t we onlyupdate the weights for those experts ~s ∈ AWAKEt.∑~s∈AWAKEt

wt+1~s =

∑~s∈AWAKEt

wt~sγLt(~s)−γLt(RMA)

=∑

~s∈AWAKEt

wt~sγLt(~s)γ−γL

t(RMA)

≤∑

~s∈AWAKEt

wt~s

(1− (1− γ) Lt(~s)

)(1 + (1− γ) Lt(RMA)

)

≤

∑~s∈AWAKEt

wt~s

−(1− γ)

∑~s∈AWAKEt

wt~sLt(~s)

+(1− γ)

∑~s∈AWAKEt

wt~sLt(RMA)

Apply Claim 5

≤∑

~s∈AWAKEt

wt~s ,

where we used the following two facts

Fact 2.∀γ, y ∈ [0, 1], γy ≤ 1− (1− γ)y ,

and

Fact 3.

∀γ, y ∈ [0, 1], γ−y ≤ 1 + (1− γ)yγ.

Therefore,∑~s

wt+1~s =

∑~s/∈AWAKEt

wt+1~s +

∑~s∈AWAKEt

wt+1~s

=∑

~s/∈AWAKEt

wt~s +∑

~s∈AWAKEt

wt+1~s

≤∑

~s/∈AWAKEt

wt~s +∑

~s∈AWAKEt

wt~s

≤ N .

Claim 7. For each expert ~s we have

Loss(RMA, I~s) ≤Loss(~s, I~s) +

lnNln 1γ

γ,

where N is the total number of experts.

Proof: By assumption, Lt(~s) is independent of ~st sowe can think of Lt(~s) as being fixed before the defenderselects its action ~st. Notice that for all times j we have

N ≥ wj~s = γ∑jt=1 I~s(t)(L

t(~s)−γLt(RMA)) .

Taking log 1γ

we obtain:

log 1γN =

lnN

ln 1γ

≥ −j∑t=1

(I~s(t)L

t (~s)− γLt (RMA))

= −Loss (~s, I~s) + γLoss (RMA, Is) .

Hence,

Loss(RMA, I~s) ≤Loss(~s, I~s) +

lnNln 1γ

γ.

We are now ready to prove Lemma 1.

Reminder of Lemma 1. For each expert ~s we have

Regret(RMA, ~s) ≤ 1

L− 1T + 2L lnN .

where N is the total number of experts and γ, our learningparameter has been set to γ = 1− 1

L .Proof: Set γ = 1− 1

L and apply claim 7. We have

Loss(RMA, I~s) ≤L

L− 1Loss(~s, I~s) +

L

L− 1

lnN

ln LL−1

.

Using the definition of Regret(RMA, ~s) we get

Regret(RMA, ~s) ≤ 1

L− 1Loss(~s, I~s) +

L

L− 1

lnN

ln LL−1

.

We will use the following fact

Fact 4.1

L− 1< 2 ln

(L

L− 1

),

to get

Regret(RMA, ~s) ≤ 1

L− 1Loss(~s, I~s) +

L

L− 1

logN

log LL−1

≤ 1

L− 1Loss(~s, I~s) +

L

L− 1

logN1

2L−2

≤ 1

L− 1Loss(~s, I~s) + 2L logN .

Lemma 2 follows immediately from 1.

Reminder of Lemma 2. For each expert ~s we have

Regret(RMA, ~s) ≤ 2√2T lnN + 2 lnN ,

where N is the total number of experts and where ourlearning parameter has been set to

γ = 1−√

2 lnN

T.

D. Main Theorem

We are finally ready to prove our main theorem fromsection III-B.

Reminder of Theorem 1. For all ε > 0,

Pr

∃~s, Regret(RMA,~s)T ≥ 2

√2 lnN

T +

2

√2 ln( 4N

ε )T + 2

T lnN

≤ ε .Proof: Lemma 2 tells us that for each expert ~s we have

Regret(RMA, ~s) ≤ 2√2T lnN + 2 lnN .

and Lemma 3 tells us that except with probability ε, for allactions ~s we have

Regret(RMA, ~s)−Regret(RMA, ~s) ≤ 2

√2T ln

(4N

ε

).

Combining Lemma 2 with Lemma 3 we obtain the desiredresult.