big data madison: architecting for big data
Post on 14-Jan-2017
55 Views
Preview:
TRANSCRIPT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Jordan@jordanbarrette
Josh@jherritz
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
DATA#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
DATAIS
EVERYWHERE
#CONTEXT
© Kheel Center, Cornell University “Crowds fill the street during the New York Dressmakers Strike of 1933”
Copyright 2014 MIOsoft Corporation. All rights reserved.
HADOOPZILLA
#CONTEXT
© Kheel Center, Cornell University “Crowds fill the street during the New York Dressmakers Strike of 1933”
Copyright 2014 MIOsoft Corporation. All rights reserved. © Tasmanian Archive and Heritage Office NS3245/1/249
THERE’S
GOLD IN
THEMHILLS
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
© Kheel Center, Cornell University “Crowds fill the street during the New York Dressmakers Strike of 1933”
Copyright 2014 MIOsoft Corporation. All rights reserved.
BE PROACTIVE. MOVE FASTER, BETTER, LEANER. REDUCE EMPLOYEE CHURN. REDUCE CUSTOMER CHURN.
INCREASE SALES. ENABLE UPSELL. REDUCE COSTS.
#CONTEXT
© Jim Pennucci “Email Under The Light”
Copyright 2014 MIOsoft Corporation. All rights reserved.
“Context is one of the most
important concepts to application architecture”
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
“Without contextual systems we will never get to the nirvana of what we envision around cognitive engagement"
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
FRAUD PREVENTIONRISK MITIGATIONINTELLIGENCESERVICE PERSONALIZATIONSECURITYCOMPLIANCECUSTOMER 360-DEGREE VIEWINVENTORY / SUPPLY CHAIN MANAGEMENT
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
© NASA Goddard Space Flight Center “Barred Spiral Galaxy”
Copyright 2014 MIOsoft Corporation. All rights reserved.
WE WANT TO MODEL CONTEXT, SERVE CONTEXT TO APPS ANDLEVERAGE CONTEXT TO UNDERSTAND PATTERNS ACROSS OUR DATA.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
WE WANT TO MODEL CONTEXT, SERVE CONTEXT TO APPS ANDLEVERAGE CONTEXT TO UNDERSTAND PATTERNS ACROSS OUR DATA.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
WE NEED TO EFFICIENTLY STORE ANDQUICKLY RETRIEVE CONTEXTUALLY RELATED DATA.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
Computer Storage Hierarchy
Registers
Cache
RAMFlashHard Disk
Cost Per Byte
Capacity
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
WE NEED TO EFFICIENTLY STORE ANDQUICKLY RETRIEVE CONTEXTUALLY RELATED DATA.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
COMMODITY HARDWARE $ << SPECIALIZED HARDWARE $
#CONTEXT
©Bill Wetzel “PDP-10”
Copyright 2014 MIOsoft Corporation. All rights reserved.
1 DELL R720XD = 48TBs
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
WE NEED TO EFFICIENTLY STORE PBs+
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
THINK DISTRIBUTED
#CONTEXT
© Candid Business“google-datacenter-tech-13”
Copyright 2014 MIOsoft Corporation. All rights reserved.
CONTEXT: A UNIT OF DISTRIBUTION
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
WE NEED TO EFFICIENTLY STORE ANDQUICKLY RETRIEVE CONTEXTUALLY RELATED DATA.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
Computer Storage Hierarchy
Registers
Cache
RAMFlashHard Disk
SpeedCost Per Byte
Capacity
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
© Dario Trimarchi “Computer Hard Disk Stock Image”
Copyright 2014 MIOsoft Corporation. All rights reserved.
1 MILLION SEEKS = 83 MINUTES1 BILLION SEEKS = 58 DAYS
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
CONTEXT: A PHYSICAL AND LOGICAL CLUSTER OF RELATED DATA. READ/WRITE WITH ONE SEEK TO DISK.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
WE WANT TO MODEL CONTEXT, SERVE CONTEXT TO APPS ANDLEVERAGE CONTEXT TO UNDERSTAND PATTERNS ACROSS DATA.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
© Dario Trimarchi “Computer Hard Disk Stock Image”
Copyright 2014 MIOsoft Corporation. All rights reserved.
OPERATIONAL v. ANALYTICAL
© University of Chicago, Adam Lisberg “Athletics”
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
© Mariano Garcia-Gaspar “Locked”
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
© University of Chicago, Capes Photo “World War II”
Copyright 2014 MIOsoft Corporation. All rights reserved.
GOODBYE LOCKS
#CONTEXT
© Brandon Doran “Once the dust settles”
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
© Province of British Columbia “002 Calgary Skyline”
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
WAITING…© University of Chicago, Capes Photo “World War II”
Copyright 2014 MIOsoft Corporation. All rights reserved.
COLLAPSE MULTIPLE SERIAL TRANSACTIONS
INTO ONE TRANSACTION. STILL 1 SEEK.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
NOW THE HARD PART
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
RELATIONSHIP DISCOVERY
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
NEAR vs. FAR RELATIONSHIPS
#CONTEXT
© NASA, Visible Earch
Copyright 2014 MIOsoft Corporation. All rights reserved.
“NEAR” =DATA IN THE SAME CONTEXT
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
DISTANCE-BASEDCLUSTERING
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
PHONETICNUMERICSEMANTICTEMPORALGEOSPATIALETC
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
http://accidents.apps.miosoft.com
Copyright 2014 MIOsoft Corporation. All rights reserved.
TRANSITIVITY
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
http://accidents.apps.miosoft.com
Copyright 2014 MIOsoft Corporation. All rights reserved.
PROVENANCE
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
ORIGIN
#CONTEXT
©Steve Corey “Creation of Adam, Sistine Chapel
Copyright 2014 MIOsoft Corporation. All rights reserved.
PROVENANCE – IT MIGHT NEED TO BE UPDATEDPRESERVATION
#CONTEXT
© Stephen Mitchell “Ligurian Bee’s”
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
“FAR” = RELATIONSHIP WITHANOTHER CONTEXT
#CONTEXT
© Granger Meador www.meador.org
Copyright 2014 MIOsoft Corporation. All rights reserved.
CONTEXT FRAGMENTS CANIMPLY RELATIONSHIPS WITHUNIDENTIFIED CONTEXTS
#CONTEXT
© Pablo OE, “V”
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
GRAPPLE
#CONTEXT
© John Piekos “Fishing Plug Boken”
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
ELASTICITYCOMMUNICATIONCONTAINERIZATIONCONFLICT RESOLUTIONRELATIONSHIP SUMMARY DATAADVANCED INDEXINGPARALLEL AGGREGATIONDATA CURATION TOOLSOPERATIONS AND MANAGEMENT TOOLS …
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
15+ YEARS IN THE MAKING
#CONTEXT
Copyright 2014 MIOsoft Corporation. All rights reserved.
www.miosoft.com
top related