structure data 2014: inverting 80/20: beyond bespoke big data, ari gesher
DESCRIPTION
Presentation from Ari Gesher, Engineering Ambassador, Palantir Technologies #gigaomlive More at http://events.gigaom.com/structuredata-2014/TRANSCRIPT
Wednesday, March 26, 14
INVERTING 80/20: BEYOND BESPOKE BIG DATA
Ari GesherPalantir Technologies
Wednesday, March 26, 14SLIDES BY 3/14 at NOON - TBD.....
INVERTING 80/20Beyond Bespoke Big Data
(or how operating systems change everything)
mailto: [email protected]
twitter: @alephbass
Ari Gesher, Engineering Ambassador
Wednesday, March 26, 14! •! Hiring, hiring, hiring - the only way to have this problem (and it’s a good one to have) is to have a hiring machine that’s kicking ass.! •! Design for growth - make sure your systems (IT, food, etc) can handle exponential growth. Build small, but design scalable! •! Be most worried about long-lead time resources: hiring, leadership, and real estate! •! Culture: how it changes, how to preserve it! •! Discard your old biases - project managers enable scale, but process hinders at small scale
Wednesday, March 26, 14Artillery tables, calculated by people called ‘computers’
Wednesday, March 26, 14Eniac, 1945
Wednesday, March 26, 14
OS/360 becomes THE computer
Wednesday, March 26, 14
Yak Shavers, Inc
BEFORE OPERATING SYSTEMS
Wednesday, March 26, 14
Yak Shavers, Inc
BEFORE OPERATING SYSTEMS
• Programmers had to write the minutiae of hardware, error codes, etc
Wednesday, March 26, 14
Yak Shavers, Inc
BEFORE OPERATING SYSTEMS
• Programmers had to write the minutiae of hardware, error codes, etc
• Writing directly for the hardware of a single computer
Wednesday, March 26, 14
Yak Shavers, Inc
BEFORE OPERATING SYSTEMS
• Programmers had to write the minutiae of hardware, error codes, etc
• Writing directly for the hardware of a single computer
• Programmers started with nothing but primitive functionality
Wednesday, March 26, 14
Yak Shavers, Inc
BEFORE OPERATING SYSTEMS
• Programmers had to write the minutiae of hardware, error codes, etc
• Writing directly for the hardware of a single computer
• Programmers started with nothing but primitive functionality
• Very little of the effort went into building the actual intent of the software
Wednesday, March 26, 14
yak shaving, n
Wednesday, March 26, 14
yak shaving, n1. Any apparently useless activity which, by allowing you to overcome intermediate difficulties, allows you to solve a larger problem.
Wednesday, March 26, 14
yak shavingAny apparently useless activity which, by allowing you to overcome intermediate difficulties, allows
you to solve a larger problem.
Wednesday, March 26, 14
Yaks keep hair
AFTER OPERATING SYSTEMS
Wednesday, March 26, 14
Yaks keep hair
AFTER OPERATING SYSTEMS
• OS API is a clean abstraction, hardware agnostic
Wednesday, March 26, 14
Yaks keep hair
AFTER OPERATING SYSTEMS
• OS API is a clean abstraction, hardware agnostic
• Hardware can change and code still works
Wednesday, March 26, 14
Yaks keep hair
AFTER OPERATING SYSTEMS
• OS API is a clean abstraction, hardware agnostic
• Hardware can change and code still works
• Common functionality (GUI, networking, files) shared by all programs
Wednesday, March 26, 14
Yaks keep hair
AFTER OPERATING SYSTEMS
• OS API is a clean abstraction, hardware agnostic
• Hardware can change and code still works
• Common functionality (GUI, networking, files) shared by all programs
• Majority of effort goes into primary purpose
Wednesday, March 26, 14
THE RIGHT ABSTRACTION
Wednesday, March 26, 14
Wednesday, March 26, 14
Wednesday, March 26, 14
Wednesday, March 26, 14
abstraction creates efficiency
Wednesday, March 26, 14
abstraction creates SAFETY
Wednesday, March 26, 14
abstraction creates innovation
Wednesday, March 26, 14
big data, n1. Data sets that are too large and complex to manipulate or interrogate with standard methods or tools (OED, 2013).
2. A set of technologies and tools designed to address large-scale data analysis problems.
3. A IT buzzword from the early teens of the 21st century.
Wednesday, March 26, 14
Wednesday, March 26, 14
Wednesday, March 26, 14
OPERATING SYSTEM ABSTRACTIONS
Wednesday, March 26, 14
OPERATING SYSTEM ABSTRACTIONS
• Device Drivers
Wednesday, March 26, 14
OPERATING SYSTEM ABSTRACTIONS
• Device Drivers
• Virtual Machine
Wednesday, March 26, 14
OPERATING SYSTEM ABSTRACTIONS
• Device Drivers
• Virtual Machine
• File system
Wednesday, March 26, 14
OPERATING SYSTEM ABSTRACTIONS
• Device Drivers
• Virtual Machine
• File system
• Permissions / Access Controls
Wednesday, March 26, 14
OPERATING SYSTEM ABSTRACTIONS
• Device Drivers
• Virtual Machine
• File system
• Permissions / Access Controls
• Networking
Wednesday, March 26, 14
OPERATING SYSTEM ABSTRACTIONS
• Device Drivers
• Virtual Machine
• File system
• Permissions / Access Controls
• Networking
• GUI Toolkits / UI Frameworks
Wednesday, March 26, 14
Yaks keep hair
BIG DATA ABSTRACTIONS
Wednesday, March 26, 14
Yaks keep hair
BIG DATA ABSTRACTIONS
• Data Integration Standards / Pipelines
Wednesday, March 26, 14
Yaks keep hair
BIG DATA ABSTRACTIONS
• Data Integration Standards / Pipelines
• Virtual object model / Composition Rules
Wednesday, March 26, 14
Yaks keep hair
BIG DATA ABSTRACTIONS
• Data Integration Standards / Pipelines
• Virtual object model / Composition Rules
• Object revisioning
Wednesday, March 26, 14
Yaks keep hair
BIG DATA ABSTRACTIONS
• Data Integration Standards / Pipelines
• Virtual object model / Composition Rules
• Object revisioning
• Permissions / Access Control
Wednesday, March 26, 14
Yaks keep hair
BIG DATA ABSTRACTIONS
• Data Integration Standards / Pipelines
• Virtual object model / Composition Rules
• Object revisioning
• Permissions / Access Control
• Data Exchange / Translation
Wednesday, March 26, 14
Yaks keep hair
BIG DATA ABSTRACTIONS
• Data Integration Standards / Pipelines
• Virtual object model / Composition Rules
• Object revisioning
• Permissions / Access Control
• Data Exchange / Translation
• Interactive, integrated, extensible visualization & analysis environment
Wednesday, March 26, 14
Wednesday, March 26, 14
Wednesday, March 26, 14
Wednesday, March 26, 14
Wednesday, March 26, 14
Wednesday, March 26, 14MOVIE ON NEXT SLIDE - NO AUDIO
Wednesday, March 26, 14MOVIE FILE _ NO AUDIO
Wednesday, March 26, 14MOVIE FILE _ NO AUDIO
FAMILIAR
Wednesday, March 26, 14MOVIE FILE _ NO AUDIO
INTERACTIVE
Wednesday, March 26, 14MOVIE FILE _ NO AUDIO
EXPRESSIVE
Wednesday, March 26, 14MOVIE FILE _ NO AUDIO
Wednesday, March 26, 14