DistributedCompu/ngPrimer

UMBCCMSC491Hadoop-BasedDistributedCompu/ng

Spring2016AdamShook

SomecontentadaptedfromDr.Kalpakis’sCMSC621slides

Agenda

•  DistributedCompu/ng– Evolu/onofCompu/ngInfrastructure– NetworkingInfrastructure– Proper/esofDistributedSystems– ExampleSystemArchitectures

EVOLUTIONOFCOMPUTINGINFRASTRUCTURE

Mainframe–50sto70s

•  Customhardware•  Customlow-levelspecializedcode

•  Veryexpensivesolu/ons

Client/Server–80sto00s

•  IT-ledarchitectures•  Moreportablesolu/ons•  Scalablesolu/onsbasedondemand

•  ReignoftheEnterpriseDataWarehouse

Cloud–00stoToday

•  Consumer-gradeinfrastructure•  GrowingIaaSandPaaSmarkets•  Datarevolu/on

•  Focusonapplica/onsandnotinfrastructure

WheredoesHadoopfit?

•  Apieceofyourdatainfrastructure– Cancrunchdataforanaly/cs– Canexposedataforwebapplica/ons

•  Explora/onofrawdata•  Augmentstoday’sinfrastructure

•  IMO,abigtoolboxthatcandoabitofeverything

NETWORKINGINFRASTRUCTURE

SingleServer

HDDHDDCPU

CPURAMRAM

NICNIC

Server ScaleUp

ScaleOut

FasterCPUsBiggerStorage

MoreServers

Local-AreaNetwork(LAN)Rack

HDDHDDCPU

CPURAMRAM

NICNIC

Server

HDDHDDCPU

CPURAMRAM

NICNIC

Server

HDDHDDCPU

CPURAMRAM

NICNIC

Server

HDDHDDCPU

CPURAMRAM

NICNIC

Server

Rack

HDDHDDCPU

CPURAMRAM

NICNIC

Server

HDDHDDCPU

CPURAMRAM

NICNIC

Server

HDDHDDCPU

CPURAMRAM

NICNIC

Server

HDDHDDCPU

CPURAMRAM

NICNIC

Server

WAN

Gatew

ay

WideAreaNetwork(WAN)

London,England

Beijing,ChinaNewYork,NY

PROPERTIESOFDISTRIBUTEDSYSTEMS

DistributedSystems

•  Thedevelopmentoflow-costpowerfulmicroprocessors,togetherwiththeinven/onofhighspeednetworks,enableustoconstructcomputersystemsbyconnec/ngalargenumberofcomputers

•  Adistributedsystemisacollec/onofindependentcomputersthatappearstoitsusersasasinglecoherentsystem.

Proper/esofDistributedSystems

•  Reliability•  Scalability•  Availability•  Efficiency•  CAPTheorem

Reliability

•  Canthesystemdeliverservicesinfaceofseveralcomponentfailures?

Scalability

•  Canthesystemscaletosupportagrowingnumberoftasks?

Availability

•  Howmuchlatencyisimposedonthesystemwhenafailureoccurs?

Efficiency

•  Howefficientisthesystem,intermsoflatencyandthroughput?

CAPTheorem

•  Consistent•  Available•  Par//onTolerant

•  Trade-offbetweenConsistencyandAvailability

Statefulvs.Stateless

•  Whetherornotadistributedsystemsavestheirstateonanagacheddeviceforrecovery

EXAMPLESYSTEMARCHITECTURES

SimpleClient/Server

Mul/-TieredClient/Server

Round-RobinClient/Server

LinuxReference

•  Afreeandopensourceopera/ngsystem•  Inthiscourse,weliveinEclipseandthecommandline

•  Masteryof'vi'getsyou+4charismahgp://www.ibm.com/developerworks/library/l-lpic1-v3-103-1/

hgp://www.linuxdevcenter.com/excerpt/LinuxPG_quickref/linux.pdf

References

•  hgp://webdam.inria.fr/Jorge/html/wdmch15.html•  GoogleImages