big data, big bets, big opportunities - college of science · launch of facebook modification....

Big Data, Big Bets,Big Opportunities

DAT

A

Accelerate Discoverywith Visual Analytics

Limit Surprisewith Smart Data

Detect Patternswith Mass Correlation

FY11

FY12FY13+

Ira A. (Gus) HuntChief Technology Officer

18 October 2011

Our Mission

We are the nation's first line of defense. We accomplish what others cannot accomplish and go where others cannot go. We carry out our mission by:

Collecting information that reveals the plans, intentions and capabilities of our adversaries and provides the basis for decision and action.

Producing timely analysis that provides insight, warning and opportunity to the President and decisionmakers charged with protecting and advancing America's interests.

Conducting covert action at the direction of the President to preempt threats or achieve US policy objectives.

3 Prime Directives

3

!

"

– CIA doesn’t work when IT doesn’t workSystems UpKeep our

Cutting EdgeKeep us on the

Mission First– Focus on the business of CIA

– Innovate and deliver solutions for Mission

#

DCIA Panetta

!

"

4 Big Bets

– Acquire, federate, and position for multiple constituencies to securely exploit. Grow the haystack, magnify the needles.

DataBig

ExcellenceOperational

Serve CIA ICby supporting the

ManagementTalent

– Assume a leadership role in IC activities that benefit the CIA

– Innovate infrastructure operations and provisioning, create an authoritative source on our asset base, and run IT like a business.

– Focus on continuous learning and diversity of thought, experience, background

#

$

!"

5 Key Technology Enablers

– World-class abilities to discover patterns, correlate information, understand plans and intentions, and find and identify operational targets in a sea of data

Mission AnalyticsAdvanced

Widgets and ServicesEnterprise

Security Serviceas a

Data HarborEnterprise Data Management--the

– One environment, all data, protected and secure using common security services such as: ubiquitous encryption, enterprise authentication, audit, DRM, secure ID propagation, and Gold Version C&A.

– A customizable, integrated and adaptive webtop that lets analysts, ops officers, and targeters to “have it their way”.

– An ultra-high performance data environment that enables CIA missions to acquire, federate, and position and securely exploit huge volumes data.

#

$Cloud Computing– Ruthlessly standardized, rigorously automated, dynamic and elastic commodity

computing environment. Massive capacity ahead of demand. Speed for mission need.

%

!

Change the Conversation

– Time and capacity were not constraints– All data was securely and appropriately available to those who need it– Data security was automatic, seamless, and persistent– Answers were computed ahead of the questions– Users had the tools they need to do their job their way– Sophisticated data engines worked for people not vice versa– IT was Frictionless

Just Imagine if:

We would Change the Conversation with Mission – EIT becomes an enabler– The need for Mission to seek alternatives evaporates– Mission can focus on mission execution, not IT management– Overall Agency IT costs decline

#

!

"

KTE Focus

– Speed of delivery, capacity ahead of demand, security ahead of need, answers ahead of questions

Provide the Opportunity to Change the Conversation

“Two-fer”Deliver the

– Zero wait, zero effort– Like 110 out of the wall, plug in and go

– Make Big Data real, while – Enabling Operational Excellence

#

Make IT Frictionless for Mission

8

Capacity ahead of demand

Deliver

!"

#Security ahead of need

Answers ahead of questions

Change the ConversationMake IT Frictionless for Mission

9

BIGOur world by the numbers

7/25/2008

10

Google passes1 Trillion Indexed

URLs(1,000,000,000,000)

11

$187

7/25/2008—Google passes 1 trillion URLs

12

Cost per second of Ebay’s last outage

$16,156,800/Day

13

789,400,000,000,000,000

7/25/2008—Google passes 1 trillion URLs

$187/second—Cost of last Ebay outage

14

Current size of YouTube in bytes

(789.4 PB)

15

2/4/2011

7/25/2008—Google passes 1 trillion URLs

$187/second—Cost of last Ebay outage

789.4 PB—Size of YouTube

16

IPv4 address space is exhausted, 4.3 billion addresses

have been allocated

17

340 undecillion (1038)34,000,000,000,000,000,000,000,000,000,000,000,000,000

7/25/2008—Google passes 1 trillion URLs

$187/second—Cost of last Ebay outage

789.4 PB—Size of YouTube

2/4/2011—IPv4 address space exhausted

18

Size of the IPv6 address space

(340x1038)

19

100 million gigabytes

7/25/2008—Google passes 1 trillion URLs

$187/second—Cost of last Ebay outage

789.4 PB—Size of YouTube

2/4/2011—IPv4 address space exhausted

340x1038—Size of IPv6 address space

20

Size of Google’s index

21

144 million

7/25/2008—Google passes 1 trillion URLs

$187/second—Cost of last Ebay outage

789.4 PB—Size of YouTube

2/4/2011—IPv4 address space exhausted

340x1038—Size of IPv6 address space

100 million gigabytes—Size of Google’s index

22

Number of Tweets per day

23

1,700,000,000,000

7/25/2008—Google passes 1 trillion URLs

$187/second—Cost of last Ebay outage

789.4 PB—Size of YouTube

2/4/2011—IPv4 address space exhausted

340x1038—Size of IPv6 address space100 million gigabytes—Size of Google’s index

144 million—Number of Tweets per day

24

Number of objects in a small private startup database

25

90,000,000,000,000,000

7/25/2008—Google passes 1 trillion URLs

$187/second—Cost of last Ebay outage

789.4 PB—Size of YouTube

2/4/2011—IPv4 address space exhausted

340x1038—Size of IPv6 address space100 million gigabytes—Size of Google’s index

144 million—Number of Tweets per day

1.7 trillion—Items in a startup’s DB

26

Facebook data holdings (90PB)

27

4,600,000,000

7/25/2008—Google passes 1 trillion URLs

$187/second—Cost of last Ebay outage

789.4 PB—Size of YouTube

2/4/2011—IPv4 address space exhausted

340x1038—Size of IPv6 address space100 million gigabytes—Size of Google’s index

144 million—Number of Tweets per day

1.7 trillion—Items in a startup’s DB

90PB—Facebook data holdings

28

Number of wireless devices in the world

29

1,000,000,000

7/25/2008—Google passes 1 trillion URLs

$187/second—Cost of last Ebay outage

789.4 PB—Size of YouTube

2/4/2011—IPv4 address space exhausted

340x1038—Size of IPv6 address space100 million gigabytes—Size of Google’s index

144 million—Number of Tweets per day

1.7 trillion—Items in a startup’s DB

90PB—Facebook data holdings

4.3 Billion—Mobile devices

30

PCs and Laptops

31

6,904,000,000 People4,600,000,000 Phones

1,000,000,000 PCs

32

6.9 Billion—World1. 1.3 Billion—China2. 1.2 Billion—India

5. 311 Million—US6. 237 Million—Indonesia

33

6.9 Billion—World1. 1.3 Billion—China2. 1.2 Billion—India3. 800 Million—Facebook

5. 311 Million—US6. 237 Million—Indonesia

34

6.9 Billion—World1. 1.3 Billion—China2. 1.2 Billion—India3. 800 Million—Facebook4. 500 Million—Microsoft Online5. 311 Million—US6. 237 Million—Indonesia

35

Connected World=Big Data

7/25/2008—Google passes 1 trillion URLs

$187/second—Cost of last Ebay outage

1 Billion—PCs and Laptops

789.4 PB—Size of YouTube

2/4/2011—IPv4 address space exhausted

340x1038—Size of IPv6 address space100 million gigabytes—Size of Google’s index

144 million—Number of Tweets per day

1.7 trillion—Items in a startup’s DB

90PB—Facebook data holdings

4.3 Billion—Mobile devices

6.9 billion people

800 million Facebook users

36

37

2Driving Forces

38

Mobility Cloud

39

Simple to useAlways onEverywhereUser added functionality

Continuous ServicesUnbounded Scaling

=

Provides

Requires

See Cloud

40

=

Continuous Services Unbounded Scale User added functionality

Big Data Advanced Analytics

Radical Efficiency Rigorous Standardization Ruthless Automation Failure Tolerant Redundant by Design Available by Default

41

Makes IT Frictionless

42

Frictionless ITThe new revolution in IT

Most significant paradigm shift to date

Bigger than mainframe to PC

Bigger than PC to LAN

Bigger than the Internet

Innovation engine of the future

Start Fast, Build Fast, Succeed Fast, Scale FastOr

Fail Cheap

43

+ + =

44

Animoto and Amazon EC2N

umbe

r of E

C2

Inst

ance

s

4/12/2008

Launch of Facebook modification.

Amazon EC2 easily scaled to handle additional traffic

Peak of 5000 instances

4/14/2008 4/15/2008 4/16/2008 4/18/2008 4/19/2008 4/20/20084/17/20084/13/2008

Steady state of ~40 instances

45

+ +=

+

46

47

!

"

Our Job

Leverage the Big Data world

Find the Information that Matters

Connect the Dots

Understand the Plans of our Adversaries– Prevent an attack, Save lives, Safeguard our national security

#

$

49

Why We Care

50

Why We Care

TheProblem

51

!

"

Our Problem: Which 5K

Don’t know the future value of a dot today

We cannot connect dots we don’t have

The old collect, winnow, dissem model fails spectacularly in the Big Data world

The few cannot know the needs of the many

#

Secure the data, Connect the data, Empower the user

53

Let’s talk Cloud

54

! Cloud, n. A visible mass of vapor, especially one suspended in the sky

! Cloud, v. To darken; obscure; threaten

! Cloud, it. The single most over-hyped term in the history of Information Technology

! Cloud = 42

Cloud Definitions

55

Cloud Definitions

Public Private Hybrid

Commerciallyavailable compute

storage and hosting

Private cloud that expands into the publiccloud for occasionalpeak demand workloads

Operated, managedand controlled by aprivate enterprise

56

Cloud@CIA

Public Private Hybrid

Private cloud that expands into the publiccloud for occasionalpeak demand workloads

Managed, secured audited, & operated under our control in our spaces!!!

XCommercially

available computestorage and hosting

X

57

Cloud, cia: Private,

Cloud@CIA

Large Scale

X

58

Cloud, cia: Private, large scale,

Cloud@CIA

Dynamic, ElasticComputing, storage, and networking resources can be rapidly provisioned and released

59

Cloud, cia: Private, large scale, dynamic, elastic

Cloud@CIA

AutomatedMinimal management effort or service provider interaction--“Zero touch”

60

Cloud, cia: Private, large scale, dynamic, elastic, automated,

Cloud@CIA

CommodityBuilt from commodity components, period. Follow Moore’s Law for both cost and performance

61

Cloud, cia: Private, large scale, dynamic, elastic, automated, commodity computing environment

Cloud@CIA

An essential investment to enable our Missions

Why Cloud?

62

!Strategic and technical direction!Inexorable trend in computing!Commodity wins—PERIOD!

!Efficiency and effectiveness!Commodity pricing, commodity cost curve !Highly-automated management and provisioning!Rigorous standards at the OS and middle-ware levels!Green for “Free”

!Speed and agility!Capacity and scale ahead of demand!Dynamic and elastic provisioning

Why Cloud?

63

Unprecedented capabilities for mission! Peta-scale capacity for peta-scale problems! Big data, big analytics, big performance! Connect the dots—find/stop the next underwear bomber or IED! Accelerated targeting, Enhanced asset validation! Correlation not search; verbs as well as nouns; patterns not just

words! Boolean is broken

! Enhanced data analytics and visualization—volume is our friend! Better security—encrypted enterprise! COOP vs DR

64

Let’s talk Security

Security Matters

#They really are out to get us!

Share with assurance

" It is a life and death matter

Security Maxims You Know

#People are the weak link!

Absolute security is absolutely impossible

" Current methods are insufficient and unsustainable

(by Heart)

A Different Approach

#Minimize the Opportunity!

Assume you’ll be had (ie: the trusted insider)

" Minimize the Loss

$ Elevate protections and Control your world

Security as a Service

#Build common security services!

How many times do you build security?

" Use them everywhere

$ Ruthlessly protect and control them

Security Services

#Data—Ingestion and Tagging, Signing, and DRM!

People—Directory Services, Identity Management, PKI, Secure Identity Propagation

"People & Data—Access Control Service, Audit

$ Enterprise Encryption

Necessary Steps

#

Separate!

Separatedata from applications

applications from securitysecurity from rules

rules from data

roles and responsibilities from accesses

Necessary Steps-2

" Implement transparent end-to-end encryption

$%

Encrypt and sign all objects—data, people, devices

Enable shades of gray—its not a yes/no world

& Distribute permission authority

Leverage the Cloud:Control Your World, Change Your Appearance

#Constantly reimage from the vault!

Build gold images then sign and vault them

" Embrace elastic computing—make it a shell game

$ Take advantage of replication management

Missing Pieces

#Distributed key management in the Cloud!

Transparent end-to-end encryption

" Secure identity propagation

$ Vaulted image management

Questions

The journey of a thousand miles begins with a single step.

Lao Tzu