2011-08-10 in-q-tel technology focus day, trends & observations in open source cloud computing

1

Trends & Observations inOpen Source Cloud Computing

IQT Technology Focus DayAugust 10, 2011 | Arlington, Virginia

Shawn WellsTechnical Director, U.S Intelligence [email protected] / 443.534.0130

mailto:[email protected]

2

Cloud Jeopardy!

? ? ? ? ? ?

Cloud Type SaaS IaaS IaaSSaaS / PaaS

SaaSSaaS / PaaS

Customers55k Corp1.5m Pers

100k+ 50k+ 1m+ 750m+ 10k+

Cloud Revenue $1.5B $500M $56M $50M N/A N/A

http://www.thetechtrendz.com/2010/07/top-5-cloud-computing-providers.html

3

Cloud Jeopardy!

SalesForce ? ? ? ? ?


SaaSSaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+



4

Cloud Jeopardy!

SalesForce Amazon ? ? ? ?


SaaSSaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+



5

Cloud Jeopardy!

SalesForce Amazon Rackspace ? ? ?


SaaSSaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+



6

Cloud Jeopardy!

SalesForce Amazon Rackspace Google ? ?


SaaSSaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+



7

Cloud Jeopardy!

SalesForce Amazon Rackspace Google Facebook ?


SaaSSaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+



8

Cloud Jeopardy!

SalesForce Amazon Rackspace Google Facebook Microsoft


SaaSSaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+



9

Cloud Jeopardy!



SaaSSaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+


We're using commodity hardware running RHEL and it has been phenomenal It's 10x less

expensive running RHEL and orders of magnitude faster.

- Parker Harris, Executive VP of Technology at SF.com

10

Cloud Jeopardy!



SaaSSaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+


Amazon cut technology expenses by about 25% from $71M to $54M. The reduction was attributed primarily to Amazon's migration to

Linux.- http://news.cnet.com/2100-1001-275155.html

11

Cloud Jeopardy!



SaaSSaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+


“VMWare has not built something that can scale up to Amazon level scalability of

hundreds of thousands to millions of servers.” So, Rackspace went open source.

- Jim Curry, VP of Corporate Developoment at Rackspace

12

Cloud Jeopardy!



SaaSSaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+


“Google announced last week that its cloud will run nothing but open source software.”

- http://www.zdnet.com/blog/open-source/google-goes-all-in-with-an-open-source-cloud/5334

13

Cloud Jeopardy!



SaaSSaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+


“Facebook was built from the ground up on open source software.

- https://developers.facebook.com/opensource/

14

Cloud Jeopardy!



SaaSSaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+


Microsoft is proprietary, but note how they follow the open source clouds with things like Daytona

(Hadoop replacement).- http://research.microsoft.com/en-us/news/headlines/daytona-

071811.aspx

15

How Does Open Source Compete?

16

Red Hat VMWare0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

Organization

Em

plo

yees

17

Red Hat VMWare Google0

5,000

10,000

15,000

20,000

25,000

Organization

Em

plo

yees

18

Red Hat VMWare Google Microsoft0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

90,000

Organization

Em

plo

yees

19

Red Hat VMWare Google Microsoft Oracle0

20000

40000

60000

80000

100000

120000

Organization

Em

plo

yees

20

Red Hat VMWare GoogleMicrosoft Oracle IBM0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

Organization

Em

plo

yees

21

Red HatVMWare

GoogleMicrosoft

OracleIBM

SourceForge

0200,000400,000600,000800,000

1,000,0001,200,0001,400,0001,600,0001,800,0002,000,000

Organization

Em

plo

yees

22

Open Source Development Models Mitigate Risk

Coverity has tracked open source code quality since 2004. They have found propriatary software, on average, has 20-30,000 defects per million lines of code. This has been true since 1960.

● 2004: Linux has 172 defects per MLOC (0.02%)

● 2005: Linux grew 4.7%, but defect density dropped to 133 defects per MLOC (0.01%)

● 2006: DHS funds study, adds LAMP stack, finds 0.01% defect density (even though 325,000+ lines of code added)

● 2009: Coverity now covers 280 projects, finds average defect density of 111 defects per MLOC(proprietary has 20,000 to 30,000)

25

Evolution of Warfighter Applications

26


27


28


29


30


31


32


33

KVM SPECvirt Scalability & Performance

Leading Propriatary SPECvirt2010

35

● Capable of 1B+ messages 14 minutes

● JPMorgan only sends ~1B AMQP messages per day

37

Included Security Controls

● Type 1 vs Type 2 (classified as Type 1 by the IC)

● Common Criteria (CAPP, RBAC, LSPP)

● sVirt

● Certifiable Linux Integration Project (CLIP)● DCID 6/3 PL4● NSS 1253● DOD 8500.2● DISA STIG V5R1● http://oss.tresys.com/projects/clip

38

What's the Current Problem?

40

Red Hat Cloud Foundations

41

ADMINS BUILD PRIVATE PAAS

46

DEVELOPERS USE PRIVATE PAAS

51

WHO'S DOING THIS?

52

Cloud Jeopardy!



SaaS SaaS / PaaS


100k+ 50k+ 1m+ 750m+ 10k+

Cloud Revenue

$1.5B $500M $56M $50M N/A N/A

Open Source Providers

53

“Our decade-long partnership with Red Hat has always been focused on customer value and innovation. Today, we are extending this partnership to include cloud computing – broadening our reach and answering the strong customer demand for cloud computing services, Red Hat Enterprise Virtualization is an optimal hypervisor technology for the infrastructure offerings on the IBM cloud.”

- Maria Azua, VP Cloud Computing Enablement at IBM

RED HAT CLOUD CUSTOMERS

54

“Red Hat gets it. In kernel virtualization, virtualization management, the application orchestration and cloud abstraction layers, Red Hat gets it. Red Hat is giving customers the opportunity to innovate [in different ways] and leverage them where it makes sense.”

- Derek Chan, Digital Operations, DreamWorks

“At DreamWorks Animation, cutting-edge technology is a key enabler to our creative ambition. Red Hat’s Cloud Foundations provides a compelling platform to enable our cloud strategy giving us the ability to quickly scale our compute needs to meet the demands of our productions."

— Ed Leonard, DreamWorks Animation’s CTO

RED HAT CLOUD CUSTOMERS

55

What About the Data?

56

Structured Data Services

CFDB

CSDS DMDC

GSORTSIDE/AVNGA

FLIS

CSDS_PL

GDSS JOPESClassic

JOPES4.0

Physical Layer (PL)

GTN

57


CFDB

CSDS DMDC

GSORTSIDE/AVNGA

FLIS

CSDS_PL

CSDS_VBL

GDSS JOPESClassic

JOPES4.0

Virtual Base Layer

(VBL)

Physical Layer (PL)

GTN

58


CFDB

CSDS DMDC

GSORTSIDE/AVNGA

FLIS

CSDS_PL

CSDS_VBL

Facilities_VMLMaterial_VML

GDSS

Priv

ate

Dat

a a

nd M

etad

ata

Virtual Mid Layer (VML)

JOPESClassic

JOPES4.0

Virtual Base Layer

(VBL)

Physical Layer (PL)

GTN

Plans_VML

59


CFDB

CSDS DMDC

GSORTSIDE/AVNGA

FLIS

CSDS_PL

CSDS_VBL

Facilities_VMLMaterial_VML

Facilities_VQLMaterial_VQL

GDSS

Plans_VQL

Priv

ate

Dat

a a

nd M

etad

ata

Pub

lic D

ata

Virtual Mid Layer (VML)

Virtual Query Layer (VQL)

(Exposed Views)

JOPESClassic

JOPES4.0

Virtual Base Layer

(VBL)

Physical Layer (PL)

GTN

Plans_VML

60

DISA ADNET: Anti-Drug Network

Challenge Counter-narcotics and counter-narcoterrorism Statutory detection and monitoring Data is heterogenous & on multiple systems

Solution MetaMatrix provides an abstracted view across

multiple State/Local Law enforcement agencies. The virtual Database enables BI tools to get a

complete picture of a "person of interest" from any history, warrants, jail, crimes, vehicles, etc...

Also, MM is used as a federated search layer looking for possible persons of interest given general details (cars, addresses, license, aliases, etc...) ‏

Benefit Enable ADNET to deliver on its mission

Data Services Platform

BI tools, Portal, Federated Search

Data Service

Disparate, heterogenous State/Local databases

61

IC Persons of Interest

Challenge Need to find Person of Interest among disparate

systems Adherence/mapping to common schema Data integration for SOA enablement

Solution Created abstracted view of a Enterprise Schema that

is focused on Master Data Entities (Domains) like Person, Organizations, etc.

Provide data services layer of the SOA stack, feeding the ESB

ESB facilitates sync/async capabilities and provides integrated enterprise data efficiently and rapidly to multiple consumers

Benefit Simplified data access and decoupled services and

apps from the underlying complex data infrastructure Single view of data enables migration of external

sources into the Enterprise repository seamlessly and without application impact

Data Services Platform

Portal, ESB, Federated Search

Data Service

Disparate, heterogenous data sources with varying

schemas/representations

62

How Can the Government (and the IC) Get Involved?

63

Efforts at NGA

● NGA Project Executive: Bert Beaulieu (InnoVision)

● NGA Project Manager: Michael W. Cline

2011-08-10 in-q-tel technology focus day, trends & observations in open source cloud computing

Software