SoftwareComplexityStevenM.Bellovin
https://www.cs.columbia.edu/~smb
1
WhatHappened?
• Whyisatrainarrivingin-2minutes?• Isthe10:26runningaheadofthe9:38?• (We’llignorethefactthatthey’rebothquitelate.)
2
Optimism
“Theprogrammer,likethepoet,worksonlyslightlyremovedfrompurethought-stuff.Hebuildshiscastlesintheair,fromair,creatingbyexertionoftheimagination.Fewmediaofcreationaresoflexible,soeasytopolishandrework,soreadilycapableofrealizinggrandconceptualstructures.”
FredBrooks,TheMythicalMan-Month
5
RealityCheck
“[O]nemustperformperfectly.Thecomputerresemblesthemagicoflegendinthisrespect,too.Ifonecharacter,onepause,oftheincantationisnotstrictlyinproperform,themagicdoesn’twork.Humanbeingsarenotaccustomedtobeingperfect,andfewareasofhumanactivitydemandit.”
FredBrooks,TheMythicalMan-Month
6
RealSoftware
• Softwareisbuggy• Softwareisalwaysbuggy• Thebiggertheprogram,thebuggierthesoftware—always
7
So?
• Therearelimitstohowgoodoursoftwarecanbe• Therearethereforethingswecan’tdo• Moreprecisely,whenweincreasecomplexitywe
a) Increasethebugratedramaticallyb) Increasethedevelopmentcostdramaticallyc) Both!
10
ComplexityandCurrentEvents
• Webankonline• Webuythingsonline• Wecommunicateonline• Whycan’twevoteonline?
11
ProbablyJustaBug
(PhotobyEdFelten) 12
Avotingmachinetapefromthe2008presidentialprimaryinaNewJerseyprecinct.
GoingDark
“Asaresult,althoughthegovernmentmayobtainacourtorderauthorizingthecollectionofcertaincommunications,itoftenservesthatorderonaproviderwhodoesnothaveanobligationunderCALEAtobepreparedtoexecuteit.”
ValerieCaproni,GeneralCounseloftheFBI
14
TheFBI’sSolution
• Allcommunicationsystemsneedsomeformofaccessforlawenforcement• Allencryptionsystemsneeda“backdoor”(whichtheycalla“goldenkey”)• Canwedoit?
15
WiretapInterfacesareHard
• Someyearsago,theNSAevaluatedthestandardizedwiretapinterfaceon26differentphoneswitches• Allhadsecurityflaws
• Someone(probablyanintelligenceagency)hackedacellphoneswitchinAthensandabusedthewiretapinterface• About100phoneswereillegallytapped,includingthePrimeMinister’s
16
CryptographyisHard
“Finally,protocolssuchasthosedevelopedherearepronetoextremelysubtleerrorsthatareunlikelytobedetectedinnormaloperation.”
RogerNeedhamandMichaelSchroeder,“UsingEncryptionforAuthenticationinLargeNetworksofComputers”
17
From“KeysUnderDoormats”
“Wehavefoundthatthedamagethatcouldbecausedbylawenforcementexceptionalaccessrequirementswouldbeevengreatertodaythanitwouldhavebeen20yearsago.Inthewakeofthegrowingeconomicandsocialcostofthefundamentalinsecurityoftoday’sInternetenvironment,anyproposalsthatalterthesecuritydynamicsonlineshouldbeapproachedwithcaution...Thecomplexityoftoday’sInternetenvironment,withmillionsofappsandgloballyconnectedservices,meansthatnewlawenforcementrequirementsarelikelytointroduceunanticipated,hardtodetectsecurityflaws.”
Abelsonetal.
18
WhyTechnologistsOpposeGoldenKeys
• IthasnothingtodowithdislikeoftheFBIortheNSA• Technologistscanbevictimsofcriminalsandterrorists,too• Rather,it’saquestionofcrimeprevention—thesoftwarenecessarytopermitlawenforcementaccesshasahighprobabilityofopeningupnewsecurityholes• Therootcauseisthecomplexityofsoftware
19
TheInternetofThings
• We’reconnectingmoreandmore“things”totheInternet• Theserunonsoftware;thissoftwareisoftenpoorlywrittenandneverpatched
20
Self-DrivingCars
• Almostcertainly,wewillseesomecrashesduetobuggycode• Possibly(thoughnotcertainly),therewillbecrashesduetohacking
• Eventoday’s“dumb”carscontain50-75networkedcomputers• Amoderncarisactuallyamobiledatacenter!
• But—self-drivingcars,flawsandall,willalmostcertainlybesaferthanhuman-drivencars• Carsdon’tgetdrunk,sleepy,distracted,etc.
21
UsersDon’tSeeMostoftheComplexity
• Goodsoftwareoftenhideshowcomplexitis• But—thecomplexityisstillthere• Often,it’sthepartsyoudon’tknowaboutthatcancausethemosttrouble
22
SoWhatDoWeDo?
• Giveup?• No;thatsacrificesthebenefitsofcomputers.Therearereasons(andgenerallygoodones)whywerelyonsoftware
24
SoWhatDoWeDo?
• Giveup?• No;thatsacrificesthebenefitsofcomputers.Therearereasons(andgenerallygoodones)whywerelyonsoftware• Often,somesmallrateoffailureisquiteacceptable—nothingelseisperfect,either
25
SoWhatDoWeDo?
• Giveup?• No;thatsacrificesthebenefitsofcomputers.Therearereasons(andgenerallygoodones)whywerelyonsoftware• Often,somesmallrateoffailureisquiteacceptable—nothingelseisperfect,either• Thetrickisknowinghowtodecide.Wewantmajorbenefits,comparativelylowrisks,andacceptableconsequencesifthereisafailure
26
“Thecompetentprogrammerisfullyawareofthestrictlylimitedsizeofhisownskull;thereforeheapproachestheprogrammingtaskinfullhumility…”
EdsgerDijkstra,“TheHumbleProgrammer”
27
SomeSuggestions
Good
• Self-drivingcars• Communicationsapps
• Thesmartgrid?
Bad
• (Residential)lightbulbs• Bikelocks• Anti-missilesystems• Votingmachines
• Networkedsextoys?
28