middleware diarrhea and other ailments
DESCRIPTION
Middleware Diarrhea and Other Ailments. Michael Stonebraker Adjunct Professor Massachusetts Institute of Technology ([email protected]). Outline. Too much middleware XML ailments Web services ills Our professional sickness. Client-Server Got Replaced by N-Tier Computing. The Web - PowerPoint PPT PresentationTRANSCRIPT
1
Middleware Diarrhea and Other Ailments
Michael Stonebraker
Adjunct Professor
Massachusetts Institute of Technology
2
M.I.T.
OutlineOutlineOutlineOutline
Too much middlewareXML ailmentsWeb services illsOur professional sickness
Too much middlewareXML ailmentsWeb services illsOur professional sickness
3
M.I.T.
Client-Server Got Replaced by N-Tier Client-Server Got Replaced by N-Tier ComputingComputingClient-Server Got Replaced by N-Tier Client-Server Got Replaced by N-Tier ComputingComputing
The WebGizmosScalability and management problems
with client server
The WebGizmosScalability and management problems
with client server
4
M.I.T.
Humility LessonHumility LessonHumility LessonHumility Lesson
We all sold client-server hardduring the 80’sand even into the 90’s
Less than 10 years later it is the worst idea on the planet
We all sold client-server hardduring the 80’sand even into the 90’s
Less than 10 years later it is the worst idea on the planet
We should feel really dumb!
5
M.I.T.
N-Tier Computing Produced Lots of N-Tier Computing Produced Lots of MiddlewareMiddlewareN-Tier Computing Produced Lots of N-Tier Computing Produced Lots of MiddlewareMiddleware
App serversEAI/messagingETLFederatorsWorkflowCMSPortalsDBMS
App serversEAI/messagingETLFederatorsWorkflowCMSPortalsDBMS
6
M.I.T.
Middleware DiarrheaMiddleware DiarrheaMiddleware DiarrheaMiddleware Diarrhea
Average enterprise has one (or more) app serversone (or more) EAI packages one (or more) ETL packagesone (or more) portal productsone (or more) application packagesand maybe someday a federated
DBMS
Average enterprise has one (or more) app serversone (or more) EAI packages one (or more) ETL packagesone (or more) portal productsone (or more) application packagesand maybe someday a federated
DBMS
7
M.I.T.
All of these systemsAll of these systems All of these systemsAll of these systems
Contain transformation enginesAnd often do function activation (app service)And often have adapters to legacy systems
Contain transformation enginesAnd often do function activation (app service)And often have adapters to legacy systems
Huge overlap in functionality!!
8
M.I.T.
Less Moving PartsLess Moving PartsLess Moving PartsLess Moving Parts
Less systemsMore uniformityLess duplication
Less systemsMore uniformityLess duplication
9
M.I.T.
Less SystemsLess SystemsLess SystemsLess Systems
Less system administratorsLess trainingLess manualsLess bugsLess cross system issues
Less system administratorsLess trainingLess manualsLess bugsLess cross system issues
10
M.I.T.
More UniformityMore UniformityMore UniformityMore Uniformity
Every island has memory managementsecurity modelthreading model
Less is better
Every island has memory managementsecurity modelthreading model
Less is better
11
M.I.T.
Less DuplicationLess DuplicationLess DuplicationLess Duplication
Most of the islands support transformationsreasonable chance you will do each one 6
or more timesmaintenance headache
Most of the islands support transformationsreasonable chance you will do each one 6
or more timesmaintenance headache
12
M.I.T.
So How To Consolidate…… So How To Consolidate…… So How To Consolidate…… So How To Consolidate……
Converge app server into OR DBMSdumbest OR query is execute function
Converge app server into OR DBMSdumbest OR query is execute function
Remember that everything looks like a
nail to the guy with the hammer!
13
M.I.T.
PictoriallyPictorially
client
DBMS
DBMS
component
component
component
14
M.I.T.
This Requires….This Requires….This Requires….This Requires….
DBMS to send queries to other DBMSsI.e. be a data federatorLoad balance also requires a federator
DBMS to send queries to other DBMSsI.e. be a data federatorLoad balance also requires a federator
15
M.I.T.
Best of Breed FederatorsBest of Breed FederatorsBest of Breed FederatorsBest of Breed Federators
Support schema heterogeneityby executing OR functions
Support materialized viewsto cache static data
Support schema heterogeneityby executing OR functions
Support materialized viewsto cache static data
16
M.I.T.
Less Moving Parts….Less Moving Parts….Less Moving Parts….Less Moving Parts….
Federators dominate ETLETL only supports “push”federators do both “push” and “pull”
Federators dominate ETLETL only supports “push”federators do both “push” and “pull”
17
M.I.T.
WorkflowWorkflowWorkflowWorkflow
A collection of ruleswho’s allowed to buy whatand who must approve it
Best considered as a boxes and arrows
diagramAnd compiled into components to run on an
app server
A collection of ruleswho’s allowed to buy whatand who must approve it
Best considered as a boxes and arrows
diagramAnd compiled into components to run on an
app server
18
M.I.T.
Workflow Framework -- PO’s
managerPOPO Big? nono
Lavenyesyes
IT? nono
yesyes
19
M.I.T.
Data Intensive Workflow Should Move Inside an OR DBMSData Intensive Workflow Should Move Inside an OR DBMS
GUI for “boxes and arrows”Compiler for the diagram
processing steps become components business rules become triggers all data flow inside the DBMS
Worked great in Media/360
GUI for “boxes and arrows”Compiler for the diagram
processing steps become components business rules become triggers all data flow inside the DBMS
Worked great in Media/360
20
M.I.T.
Why?Why?
Big Big Big performance advantage no polling of the DBMS no data movement easy to change!
Big Big Big performance advantage no polling of the DBMS no data movement easy to change!
Watch for Informix product in this area!
21
M.I.T.
NirvanaNirvanaNirvanaNirvana
One integrated system that doesfederationEAI app service
With a single transformation systemBased on DBMS technology (or
something else….)
One integrated system that doesfederationEAI app service
With a single transformation systemBased on DBMS technology (or
something else….)
22
M.I.T.
XMLXMLXMLXML
Good for content storage and movementGood as “on the wire” format for data
movementas long as you don’t need to send a
lot of stuff fastBad for data storage!
Good for content storage and movementGood as “on the wire” format for data
movementas long as you don’t need to send a
lot of stuff fastBad for data storage!
23
M.I.T.
History LessonHistory LessonHistory LessonHistory Lesson
1960’s IMS and IDMS get tractioncustomers start complaining about
rewriting everything when schema
changes
1960’s IMS and IDMS get tractioncustomers start complaining about
rewriting everything when schema
changes
24
M.I.T.
History LessonHistory LessonHistory LessonHistory Lesson
1970Codd writes pioneering paperstarts a decade long argument between
IMS/CODASYL advocates and Codd
supporters
1970Codd writes pioneering paperstarts a decade long argument between
IMS/CODASYL advocates and Codd
supporters
25
M.I.T.
Net-Net of ArgumentNet-Net of ArgumentNet-Net of ArgumentNet-Net of Argument
Putting semantics into data order is badrestricts storage options
Hidden meaning bad no self-defining fields
Putting semantics into data order is badrestricts storage options
Hidden meaning bad no self-defining fields
26
M.I.T.
Net-Net of ArgumentNet-Net of ArgumentNet-Net of ArgumentNet-Net of Argument
Data independence is goodschemas change oftendon’t want to rewrite anything when this
happens
Data independence is goodschemas change oftendon’t want to rewrite anything when this
happens
27
M.I.T.
Net-Net of ArgumentNet-Net of ArgumentNet-Net of ArgumentNet-Net of Argument
Complexity is badhigh level query languages are goodKISS arguments
Call these three premises “Codd’s laws”
Complexity is badhigh level query languages are goodKISS arguments
Call these three premises “Codd’s laws”
28
M.I.T.
History LessonHistory Lesson History LessonHistory Lesson
1983(?)Codd wins Turing awardacknowledgement for being right
1983(?)Codd wins Turing awardacknowledgement for being right
29
M.I.T.
XML in This Historical LightXML in This Historical Light XML in This Historical LightXML in This Historical Light
Most of the bad features of IMS/Codasylallows semantics in data orderdata independence will be a challenge
try updates on inverted hierarchies look at IMS LDBs
more complex than Codasyl
Most of the bad features of IMS/Codasylallows semantics in data orderdata independence will be a challenge
try updates on inverted hierarchies look at IMS LDBs
more complex than Codasyl
30
M.I.T.
Our FieldOur Field Our FieldOur Field
We look a little silly sayingan idea renounced in the 1970’s is back
Leading our colleagues to ask “What’s
different?” if somebody disproved Codd’s laws; they
didn’t tell me…..
We look a little silly sayingan idea renounced in the 1970’s is back
Leading our colleagues to ask “What’s
different?” if somebody disproved Codd’s laws; they
didn’t tell me…..
31
M.I.T.
How to Win the Turing Award Circa 2020How to Win the Turing Award Circa 2020How to Win the Turing Award Circa 2020How to Win the Turing Award Circa 2020
2000’sXML data storage gets traction
2010dust off Codd’s paper
Wait 10 years to be proven right
2000’sXML data storage gets traction
2010dust off Codd’s paper
Wait 10 years to be proven right
32
M.I.T.
In Any CaseIn Any CaseIn Any CaseIn Any Case
In line tags turn 1Tbyte of EMP data into 10
Tbytes of EMP dataWon’t store anything big in native XML
will use something else…. like what?
In line tags turn 1Tbyte of EMP data into 10
Tbytes of EMP dataWon’t store anything big in native XML
will use something else…. like what?
33
M.I.T.
OR DBMSOR DBMS OR DBMSOR DBMS
XML is merely this year’s data typeNext year it will be WML or …
and there will be a next year….
XML is merely this year’s data typeNext year it will be WML or …
and there will be a next year….
34
M.I.T.
XMLSchemaXMLSchema XMLSchemaXMLSchema
Contains the kitchen sinkComplexity run amok
diarrhea from the SGML typesIncludes lots of known hard stuff
e.g. union types
Contains the kitchen sinkComplexity run amok
diarrhea from the SGML typesIncludes lots of known hard stuff
e.g. union types
35
M.I.T.
XqueryXquery XqueryXquery
Mostly syntactic sugar on OR SQL // is a user-defined function in
Informix OR engineTry to keep the semantics close to OR
SQL
Mostly syntactic sugar on OR SQL // is a user-defined function in
Informix OR engineTry to keep the semantics close to OR
SQL
36
M.I.T.
Another History LessonAnother History LessonAnother History LessonAnother History Lesson
Typical enterprise wanted data integration
for business analysis badlyneeded data in a variety of systems in a variety of formatsoften with no unique idsoften with incompatible semantics
2 day delivery means lots of thingsoften dirty
Typical enterprise wanted data integration
for business analysis badlyneeded data in a variety of systems in a variety of formatsoften with no unique idsoften with incompatible semantics
2 day delivery means lots of thingsoften dirty
37
M.I.T.
ETL Warehouse Projects of the 90’sETL Warehouse Projects of the 90’sETL Warehouse Projects of the 90’sETL Warehouse Projects of the 90’s
Well into 8 digitsUsually a factor of three behind scheduleDelivering a factor of 3 less stuffEverybody dented their pick on semantic
heterogeneitywhich is hard, hard, hardand not solved by the blizzard of 3 letter
acronyms from Redmond
Well into 8 digitsUsually a factor of three behind scheduleDelivering a factor of 3 less stuffEverybody dented their pick on semantic
heterogeneitywhich is hard, hard, hardand not solved by the blizzard of 3 letter
acronyms from Redmond
38
M.I.T.
Web ServicesWeb ServicesWeb ServicesWeb Services
Will be a long time coming outside of
simple domains (where there is no data
integration to deal with)E.g. catalog management
Grainger perspiration….
Will be a long time coming outside of
simple domains (where there is no data
integration to deal with)E.g. catalog management
Grainger perspiration….
39
M.I.T.
The Depressing State of AffairsThe Depressing State of Affairs
~50-75% of IT projects fail if we built bridges, our profession
would be firedand the same mistakes are repeated
over and over (excessive ambition,
rolling specs, bad design, failure to
load a large data set early)
~50-75% of IT projects fail if we built bridges, our profession
would be firedand the same mistakes are repeated
over and over (excessive ambition,
rolling specs, bad design, failure to
load a large data set early)
40
M.I.T.
What To Do?What To Do?
We typically don’t teach this stuff (and
do a serious disservice to our students)probably because we don’t (can’t)
spend any time in industry to figure it
out
We typically don’t teach this stuff (and
do a serious disservice to our students)probably because we don’t (can’t)
spend any time in industry to figure it
out
Action item: at the very least read a couple of Robert L. Glass’s books
41
M.I.T.
The Depressing State of AffairsThe Depressing State of Affairs
Hardware “half-life” is 18 monthsSoftware half-life is 18 years (or more)!In 25 years we moved from
C to JavaSQL to Xquery
Hardware “half-life” is 18 monthsSoftware half-life is 18 years (or more)!In 25 years we moved from
C to JavaSQL to Xquery
42
M.I.T.
What To Do?What To Do?
Much higher level design environmentsvisworkflowspecial purpose languages (report
writers,…)And stop turning down papers on this
stuff
Much higher level design environmentsvisworkflowspecial purpose languages (report
writers,…)And stop turning down papers on this
stuff
43
M.I.T.
Grand ChallengeGrand Challenge
Improve application productivity
(probability of success * programmer
productivity) by 2 this decade
Improve application productivity
(probability of success * programmer
productivity) by 2 this decade