making sense of cyberspace, keynote for software engineering institute cyber intelligence...

Mellon U

Making Sense of Cyberspace

Cyber Intel Research Consortium

May 5, 2015

Jason Hong

ComputerHumanInteraction:MobilityPrivacySecurity

Mellon U

•Bandwidth

Computing Trends

Mellon U

•Bandwidth

•Storage

Computing Trends

Mellon U

Computing Trends

•Bandwidth

•Storage

•Computing Power

Mellon U

Computing Trends

•Bandwidth

•Storage

•Computing Power

•Information

Mellon U

Human Capabilities

Cognitive Processing

Mellon U

Cognitive Processing

Visual acuity

Communication bandwidth

Attention

Human Capabilities

Mellon U

Evidence suggests it’s more like 4

Mellon U

How to Scale Up Our Ability to Understand and Analyze?

• Research in mobility, privacy, security

– Anti-phishing

– Smartphone apps, Internet of Things

• Key themes today

– Information visualization

– Machine learning

– Wisdom of crowds

Mellon U

•Which state has highest income?

• Is there a relationship between

income and education?

•Are there any outliers?

Mellon U

Visualization for ElicitingKnowledge from Data

Mellon U

How You Represent Info Matters

• What is VIII x XCI = ???

• Another example:

– Game with two players

– Alternately choose numbers between 1-15

– First to get three numbers to 15 wins

– Each number can be taken only once

Mellon U

How You Represent Info Matters

• It’s actually Tic-Tac-Toe

– Trivial in this representation

Mellon U

Other Design Principles?

1. Start out going Southwest on ELLSWORTH AVE

Towards BROADWAY by turning right.

2: Turn RIGHT onto BROADWAY.

3. Turn RIGHT onto QUINCY ST.

4. Turn LEFT onto CAMBRIDGE ST.

5. Turn SLIGHT RIGHT onto MASSACHUSETTS AVE.

6. Turn RIGHT onto RUSSELL ST.

Mellon U

Some Lessons

1. How you represent info matters

2. InfoViz is what you do show as well as what you don’t

Mellon U

Focus + Context

• More pixels to stuff that matters, fewer pixels to supporting context

Mellon U

Focus + Context

Mellon U

Appropriate Use of Color

Mellon U

Appropriate Use of Color

Mellon U

Some Lessons

3. Focus + Context

4. Appropriate use of colors (perception)

Mellon U

US Election 2004

Mellon U

InfoViz’s Can Show and Hide Info

Mellon U

All Viz’s Show and Hide Info

Mellon U

InfoViz’s Can Show and Hide Info

Mellon U

All Viz’s Show and Hide Info

Mellon U

Some Lessons

3. Focus + Context

4. Appropriate use of colors (perception)

5. All visualizations have inherent biases

Mellon U

Visualizations Don’t Have to Be Complex to be Effective

Mellon U

Example from Jeff Heer

Mellon U

Work by Jeff Heer

Mellon U

Collaborative Analysis

• Many Eyes (by IBM)

Mellon U

Can We Have Crowds of People Help Analyze Data?

• What are better ways of getting groups of people working together?

• One approach is to use lots of crowd workers to improve accuracy and time

– Common structure in your problem?

– Can you break it down into small tasks?

Mellon U

Wisdom of Crowds Approach

• Mechanics of PhishTank

– Submissions require at least 4 votes and 70% agreement

– Some votes weighted more

• Total stats (Oct2006 – Feb2011)

– 1.1M URL submissions from volunteers

– 4.3M votes

– resulting in about 646k identified phish

• Why so many votes for only 646k phish?

Mellon U

PhishTank Statistics

Jan 2011

Submissions 16019

Total Votes 69648

Valid Phish 12789

Invalid Phish 549

Median Time 2hrs 23min

• 69648 votes max of 17412 labels

– But only 12789 phish and 549 legitimate identified

– 2681 URLs not identified at all

• Median delay of 2+ hours still has room for improvement (Jan 2015 is 14+ hours)

Mellon U

Ways of Smartening the Crowd

• Change the order URLs are shown

– Ex. most recent vs closest to completion

• Change how submissions are shown

– Ex. show one at a time or in groups

• Adjust threshold for labels

– PhishTank is 4 votes and 70%

– Ex. vote weights, algorithm also votes

• Motivating people / allocating work

– Filtering by brand, competitions, teams of voters, leaderboards

Mellon U

Clustering Phish

• Motivations

– Most phish are generated by toolkits and thus similar

– Labeling single sites as phish can be hard, easier if multiple examples

– Reduce labor by labeling suspicious sites in bulk

Mellon U

Clustering Phish

• Motivations

– Most phish are generated by toolkits and thus similar

– Labeling single sites as phish can be hard, easier if multiple examples

– Reduce labor by labeling suspicious sites in bulk

Mellon U

Most Phish Can be Clustered

• With all data over two weeks, 3180 of 3973 web pages can be grouped (80%)

– Used shingling and DBSCAN (see paper)

– 392 clusters, size from 2 to 153 URLs

Mellon U

Comparing Coverage and Time

Mellon U

CrowdVerify• This is the New York Times Privacy Policy• Can we use crowd to help summarize?

Flag surprises? • Can we build predictive language models?

Mellon U

Which statement is more important?

Privacy and SecurityCrowdVerify

We receive data whenever you visit a game, application, or website that uses Facebook Platform …

When you sign up for Facebook, you are required to provide information such as your name, email address, birthday, and gender.

Mellon U

Privacy and SecurityCrowdVerify on Google’s Policies

Statement ID

Frequency Experiment Sum of Scores

Google 08

Google 13

Google 07

Google 30

Google 10

Google 12

Google 04

Google 16

Google 03

Google 11

Google 05

Google 09

Google 28

Google 02

Google 06

Google 23

Google 14

Mellon U

Top Statement - Google

• Google 08: When you use our services or view content provided by Google, we may automatically collect and store certain information in server logs. This may include: telephony log information like your phone number, calling-party number, forwarding numbers, time and date of calls, duration of calls, SMS routing information and types of calls.

Mellon U

Bottom 3 Statements - Google

• Google 29: Whenever you use our services, we aim to provide you with access to your personal information.

• Google 26: For example, you can: Take information out of many of our services.

• Google 24: People have different privacy concerns. Our goal is to be clear about what information we collect, so that you can make meaningful choices about how it is used.

Mellon U

Designing Crowd Systems

• Motivation for contributing?

– Money, altruism, fun?

• Quality control?

– How to ensure good quality? Prevent bozos?

• How is crowd wisdom aggregated?

– How are people and computers organized?

– How are decisions made?

• Skill level required?

– Novice, intermediate, expert?

Mellon U

One Real-World SuccessClickworkers

• 100k+ volunteers identified 2M Mars craters from space photographs (14k hours of free work!!)

• Aggregate results “virtually indistinguishable” from experts

Mellon U

Another Real-World SuccessGalaxy Zoo

• Problem: you are an astronomer looking at the Sloan Digital Sky survey, which has images of over 1M galaxies. You want to classify these by galaxy type.

Mellon U

DARPA Red Balloon

• 2009 challenge to find 10 red weather balloons at unknown locations in the USA

• Winner gets $40k

• Surprise: top team wins in < 9 hours

– Other teams did well too

– 9 balloons in 9 hours (1)

– 8 balloons (2 teams)

– 7 balloons (5 teams)

Mellon U

DARPA Red Balloon

Mellon U

DARPA Red Balloon

• MIT Team (10 balloons in less than 9 hours)

• Overall team strategy

– Get as many people as possible (over 5k people)

– Use multiple approaches to verify

• How people were organized

– Core team / recruit / spotters

• “Social engineering”

– Recursive financial structure to get more people

– Incentivizes people who might not be able to help

• Technologies used

– Surprisingly little, one centralized web site + google maps

Mellon U

MIT Reward Structure

Mellon U

Not Quite as SuccessfulSearch for Steve Fossett

Mellon U

DARPA Network Challenge

Mellon U

Summary

• Collecting info is easy,understanding it is hard

• How to scale?

– Information visualization

– Machine learning

– Wisdom of crowds

Mellon U

More Resources

• A Tour through the Visualization Zoo– https://queue.acm.org/detail.cfm?id=1805128

• Interactive Dynamics for Visual Analysis

– https://queue.acm.org/detail.cfm?id=2146416

• Tableau Software– http://www.tableau.com/

• Many Eyes– http://www-01.ibm.com/software/analytics/many-eyes/

Mellon U

More Resources

• DARPA Red Balloon Challenge– http://cacm.acm.org/magazines/2011/4/106587-reflecting-

on-the-darpa-red-balloon-challenge/fulltext

• Pirolli and Card, The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis

• Apolo, combining data mining with infoviz– https://www.youtube.com/watch?v=EFxYVWkj1aE

Mellon U

Patterns in Crowd Analysis

Unit of Work

Voting(Ex. PhishTank)

MapReduce(Ex. CrowdVerify)

Unit of Work

Funnel

Unit of Work

Unit of WorkDecision

Mellon U

Visualization of the Internet

Mellon U

Combine Data Mining + Viz

User can specify exemplars of a groupBelief Propagation to find more nodes

Mellon U

making sense of cyberspace, keynote for software engineering institute cyber intelligence...

chris harrison

visual processing

wonderful book

wonderful title

highest income

computer human interaction

peter kinnaird

marti hearst

Technology

cyberspace the final frontier. future of cyberspace & new...

yber intelligence tradecraft ertification - cyber...

strategic intelligence, analytic tradecraft, and agent-based...

cyberspace addiction

securing cyberspace

ga cyberspace

cyber intelligence tradecraft project

main sponsor: bronze sponsors: supporting partner:...

top secret - synergizing network analysis tradecraft

developing an understanding of analytic tradecraft...

open sesame bypassing building management controls and...

a tradecraft primer: structured analytic techniques for

developing an understanding of analytic tradecraft...

sei innovation center report: cyber intelligence tradecraft...

strategic intelligence, analytic tradecraft, and agent...

sales cycle and sales roles - tradecraft

offense in depth a developer’s perspective on hacker...

tradecraft & analysis learning using intelligence...

yber intelligence tradecraft ertification - treadstone...

oswald inks to spy tradecraft microdots