designing cybersecurity policies with field experiments

Designing Cybersecurity Policies with Field Experiments

Gene Moo LeeUniversity of Texas at Austin

Joint work with Shu He, John S. Quarterman, Andrew B. Whinston

Supported by NSF 1228990

February 25, 2015KAIST

Gene Moo Lee, KAIST, Feb 2015

“Although the threats are serious and they constantly evolve, I believe that if we address them effectively, we can ensure that the Internet remains an engine for economic growth and a platform for the free exchange of ideas.”

—Barack Obama


Motivation• Inadequate cybersecurity is a serious threat

• avg cost $3.5 million in 2013, 15% increase

• # of compromises increased by 25%

• data breaches of 2.6 million Target consumers

• U.S. government’s measures

• Cybersecurity Policy Review (2009)

• Executive Order 13636 (2013) “Improving Critical Infrastructure Cybersecurity”


Approaches• Technical approaches:

• spam filtering, intrusion detection systems (IDS), digital forensics

• Sahami et al. (1998), Cormack and Lynam (2007), Denning (1987), Lee and Stolfo (1998), Casey (2011), Taylor et al. (2014)

• Economic approaches:

• underinvestment due to (1) information asymmetry, (2) network externalities, (3) moral hazards

• van Eeten et al. (2011), Moore and Clayton (2011), Arora et al. (2004), D’Arcy et al. (2009), Wood and Rowe (2011)


Our approach• We found evidence that spam evaluation publication help improving

security levels in country level

• Quarterman et al. (2012), Qian et al. (2013)

• Use outbound spam to estimate latent security level

• 90% spam is from compromised computers controlled by botnets (Rao and Reiley 2012, Moore and Clayton 2011)

• Ultimate goal:

• Evaluate the effectiveness in organizational level

• government sponsored institution to monitor and evaluate organizational security levels (Moody’s, S&P for bonds)

• Counterfactual policy analysis with randomized field experiments


Research questions

1. Our goal is to set up an independent institution to evaluate and monitor all organizations’ cybersecurity level

2. Does information disclosure change organizational behaviour? In other words, spam reduce?

• Method: Randomized field experiment

• Two treatment groups with different info disclosure

• Two cycles of emails at January/March 2014

• A website built on Google cloud


Experimental design

• 7919 US organizations, three groups: control, private, public

• Private treatment: email with spam volume, rank, IP addr

• Public treatment: email + publication in public website


Randomization• Stratification with industry sectors and IP counts

• Pair-wise matching with pre-experimental spam volume

• Re-randomization: 10,000 times and power calculation


Treatment channel: email


Website: search engine• http://cloud.spamrankings.net

http://cloud.spamrankings.net/


Website: overall stats


Website: detail charts


System implementation

• Back end: data collector, peer ranker, web generator, MySQL, JSON

• Front end: Google cloud, search engine, analytics


Data: CBL and PSBL• A spam blocklist uses spamtraps to collect IP adresses

sending out spams:

• CBL: http://cbl.abuseat.org/

• PSBL: http://psbl.org/

• Spamtrap

• honeypot used to collect spam

• email addresses not for legit communications

• CBL daily avg data

• 8 million IP, 190K netblocks, 21K ASNs, 200 countries

http://cbl.abuseat.org/

http://psbl.org/


Organizational spam data• IP > netblock > ASN > organization

• IP > netblock: IP lookup

• netblock > ASN: Team Cymru

• ASN > org: algorithm + manual inspection

• Organization data from LexisNexis

• 7919 U.S. organizations identified

• Industry codes: SIC, NAICS

• Public/private, # employees


Org level spam volume and IP address


Industry sectors


Industry level spam volume/host


Hypothesis development

1. Information disclosure effect

2. Publicity effect

3. Pre-experimental security level

4. Industry competition level


Info sharing and publicity effects (H1, 2)


Large spammers (H3)


Competition (H4)


Empirical analysis summary

1. Private info sharing doesn’t work

2. Publicity matters

3. Organizations with (1) large spam, (2) less competition reacted

4. Peer effect exists after the treatments. Stronger with treatment groups.


Robustness check

1. Placebo test: change experiment time

2. Subsample analysis: only include moderate spammers

3. Alternative pre-experimental spam measure: 6, 4, 2, months

4. Control variables


Directions

1. Robust security evaluation: spam, phishing, DDoS, etc.

2. Different environment: China, Korea

3. Treatment channel: social media

4. Cybersecurity insurance

5. Cloud security


Thank you!

Contact: [email protected]

mailto:[email protected]


References (1)[1] Adelsman, Rony M., and Andrew B. Whinston (1977). "Sophisticated voting with informationfor two voting functions." Journal of Economic Theory 15, no. 1: pp. 145-159.[2] Anderson, Axel, and Lones Smith. "Dynamic Deception." American Economic Review 103, no.7 (2013): 2811-47.[3] Anderson, Ross (2001). "Why information security is hard: An economic perspective." IEEEComputer Security Applications Conference, pp. 358-365.[4] Aral, Sinan, and Dylan Walker. "Identifying influential and susceptible members of socialnetworks." Science 337, no. 6092 (2012): pp. 337-341.[5] Arora, Ashish, Ramayya Krishnan, Anand Nandkumar, Rahul Telang, and Yubao Yang (2004)."Impact of vulnerability disclosure and patch availability-an empirical analysis." Workshop onEconomics of Information Security, vol. 24, pp. 1268-1287.[6] Bauer, Johannes, and Michael van Eeten (2009). “Cybersecurity: Stakeholder incentives, externalities,and policy options.” Telecommunications Policy, Vol. 33, pp. 706-719.[7] Blei, David M., Andrew Y. Ng, and Michael I. Jordan (2003). "Latent dirichlet allocation."Journal of Machine Learning Research 3: pp. 993-1022.[8] Bratko, Andrej, Gordon V. Cormack, Bogdan Filipic, Thomas R. Lynam, and Blaz Zupan(2006). Journal of Machine Learning Research 6: pp. 2673-2698.[9] Bruhn, Miriam, and David McKenzie (2008). "In pursuit of balance: Randomization in practicein development field experiments." World Bank Policy Research Working Paper Series.[10] Casey, Eoghan (2011). Digital evidence and computer crime: Forensic science, computers andthe Internet. Academic Press.[11] Cormack, Gordon V., and Thomas R. Lynam (2007). “Online supervised spam filter evaluation.”ACM Transaction on Information Systems, Vol. 25(3)


References (2)[12] D’Arcy, John, Anat Hovav, and Dennis Galletta (2009). "User awareness of security countermeasuresand its impact on information systems misuse: A deterrence approach." InformationSystems Research 20, no. 1: pp. 79-98.[13] Denning, Dorothy E. (1987). “An intrusion-detection model.” IEEE Transactions on SoftwareEngineering, Vol. 13(2): pp. 222-232.[14] Dharmapurikar, Sarang, Praveen Krishnamurthy, and David E. Taylor (2003). “Longest prefixmatching using bloom filters.” Proceedings of the ACM SIGCOMM Conference: pp. 201-212.[15] Dice, Lee R. (1945). “Measures of the amount of ecologic association between species.” Ecology26(3): pp. 297-302.[16] Duflo, Esther, Rachel Glennerster, and Michael Kremer. "Using randomization in developmenteconomics research: A toolkit." Handbook of development economics 4 (2007): 3895-3962.[17] Fracassi, Cesare (2014). "Corporate finance policies and social networks." In AFA 2011 DenverMeetings Paper.[18] Festinger, Leon. "A theory of social comparison processes." Human relations 7, no. 2 (1954):117-140.[19] Gal-Or, Esther, and Anindya Ghose (2005). "The economic incentives for sharing securityinformation." Information Systems Research 16, no. 2: pp. 186-208.[20] Graham, Bryan S. (2008). "Identifying social interactions through conditional variance restrictions."Econometrica 76, no. 3: pp. 643-660.[21] Harper, Yan Chen, F. Maxwell, Joseph Konstan, and Sherry Xin Li. "Social comparisons andcontributions to online communities: A field experiment on movielens." The American economicreview (2010): 1358-1398.[22] Harrison, Glenn W., and John A. List (2004). "Field experiments." Journal of Economic Literature:pp. 1009-1055.[23] Kugler, Logan (2014). “Online Privacy: Regional Differences.” Communications of the ACM,Vol. 58 No. 2, pp. 18-20.


References (3)[24] Krebs, Brian (2014). Spam Nation: The Inside Story of Organized Cybercrime - from GlobalEpidemic to Your Front Door. Sourcebooks, Inc.[25] Lee, Wenke, and Salvatore J. Stolfo (1998). “Data mining approaches for intrusion detection.”Proceedings of 7th USENIX Security Symposium.[26] Levchenko, Kirill, Andreas Pitsillidis, Neha Chachra, Brandon Enright, Márk Félegyházi, ChrisGrier, Tristan Halvorson, Chris Kanich, Christian Kreibich, He Liu, Damon McCoy, NicholasWeaver, Vern Paxson, Geoffrey M. Voelker, and Stefan Savage (2011). "Click Trajectories:End-to-End Analysis of the Spam Value Chain." IEEE Symposium on Security and Privacy.[27] Moore, Tyler and Richard Clayton (2011). "The Impact of Public Information on PhishingAttack and Defense." Communications & Strategies 81.[28] Morgan, Kari Lock, and Donald B. Rubin (2012). "Rerandomization to improve covariatebalance in experiments." Annals of Statistics 40, no. 2: pp. 1263-1282.[29] Popadak, Jillian A. (2012). "Dividend Payments as a Response to Peer Influence." Availableat SSRN 2170561, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2170561.[30] Pitsillidis, Andreas, Chris Kanich, Geoffrey M Voelker, Kirill Levchenko, Stefan Savage (2012).“Taster’s choice: A comparative analysis of spam feeds.” Proceedings of the 2012 ACM InternetMeassure Conference: pp. 427-440.[31] Rao, Justin M., and David H. Reiley (2012). "The economics of spam." Journal of EconomicPerspectives 26, no. 3: pp. 87-110.[32] Roesch, Martin (1999). “SNORT: Lightweight intrusion detection for networks.” Proceedingsof 13th Large Installation System Administration Conference, pp. 229-238.[33] Rothschild, Michael, and Joseph Stiglitz (1992). “Equilibrium in competitive insurance markets:An essay on the economics of imperfect information.” Springer Netherlands.[34] Sahami, Mehran, Susan Dumais, David Heckerman, and Eric Horvitz (1998). “A Bayesianapproach to filtering junk e-mail.” Learning for Text Categorization 62: pp. 98-105.


References (4)

[35] Shue, Kelly (2013). "Executive networks and firm policies: Evidence from the random assignmentof MBA peers." Review of Financial Studies 26, no. 6: pp. 1401-1442.[36] Tang, Qian, Leigh Linden, John S. Quarterman, and Andrew B. Whinston (2013). “ImprovingInternet security through social information and social comparison: A field quasi-experiment.”In Workshop on the Economics of Information Security.[37] Taylor, Robert W., Eric J. Fritsch, and John Liederbach (2014). Digital crime and digitalterrorism. Prentice Hall Press.[38] Taylor, Shelley E., and Marci Lobel (1989). "Social comparison activity under threat: downwardevaluation and upward contacts." Psychological review 96, no. 4: p. 569.[39] van Eeten, M., H. Asghari, J. M. Bauer, and S. Tabatabaie (2011). "Internet service providersand botnet mitigation: A fact-finding study on the Dutch market." Delft University of Technology.[40] Wood, Dallas, and Brent Rowe (2011). "Assessing home Internet users’ demand for security:Will they pay ISPs?" Workshop of Economics of Information Security.