LearningPython

TableofContents

LearningPython

Credits

AbouttheAuthor

Acknowledgements

AbouttheReviewers

www.PacktPub.com

Supportfiles,eBooks,discountoffers,andmore

Whysubscribe?

FreeaccessforPacktaccountholders

Preface

Whatthisbookcovers

Whatyouneedforthisbook

Whothisbookisfor

Conventions

Readerfeedback

Customersupport

Downloadingtheexamplecode

Errata

Piracy

Questions

1.IntroductionandFirstSteps–TakeaDeepBreath

Aproperintroduction

EnterthePython

AboutPython

Portability

Coherence

Developerproductivity

Anextensivelibrary

Softwarequality

Softwareintegration

Satisfactionandenjoyment

Whatarethedrawbacks?

WhoisusingPythontoday?

Settinguptheenvironment

Python2versusPython3–thegreatdebate

InstallingPython

SettingupthePythoninterpreter

Aboutvirtualenv

Yourfirstvirtualenvironment

Yourfriend,theconsole

HowyoucanrunaPythonprogram

RunningPythonscripts

RunningthePythoninteractiveshell

RunningPythonasaservice

RunningPythonasaGUIapplication

HowisPythoncodeorganized

Howdoweusemodulesandpackages

Python’sexecutionmodel

Namesandnamespaces

Scopes

Objectandclasses

Guidelinesonhowtowritegoodcode

ThePythonculture

AnoteontheIDEs

Summary

2.Built-inDataTypes

Everythingisanobject

Mutableorimmutable?Thatisthequestion

Numbers

Integers

Booleans

Reals

Complexnumbers

Fractionsanddecimals

Immutablesequences

Stringsandbytes

Encodinganddecodingstrings

Indexingandslicingstrings

Tuples

Mutablesequences

Lists

Bytearrays

Settypes

Mappingtypes–dictionaries

Thecollectionsmodule

Namedtuples

Defaultdict

ChainMap

Finalconsiderations

Smallvaluescaching

Howtochoosedatastructures

Aboutindexingandslicing

Aboutthenames

Summary

3.IteratingandMakingDecisions

Conditionalprogramming

Aspecializedelse:elif

Theternaryoperator

Looping

Theforloop

Iteratingoverarange

Iteratingoverasequence

Iteratorsanditerables

Iteratingovermultiplesequences

Thewhileloop

Thebreakandcontinuestatements

Aspecialelseclause

Puttingthisalltogether

Example1–aprimegenerator

Example2–applyingdiscounts

Aquickpeekattheitertoolsmodule

Infiniteiterators

Iteratorsterminatingontheshortestinputsequence

Combinatoricgenerators

Summary

4.Functions,theBuildingBlocksofCode

Whyusefunctions?

Reducecodeduplication

Splittingacomplextask

Hideimplementationdetails

Improvereadability

Improvetraceability

Scopesandnameresolution

Theglobalandnonlocalstatements

Inputparameters

Argumentpassing

Assignmenttoargumentnamesdon’taffectthecaller

Changingamutableaffectsthecaller

Howtospecifyinputparameters

Positionalarguments

Keywordargumentsanddefaultvalues

Variablepositionalarguments

Variablekeywordarguments

Keyword-onlyarguments

Combininginputparameters

Avoidthetrap!Mutabledefaults

Returnvalues

Returningmultiplevalues

Afewusefultips

Recursivefunctions

Anonymousfunctions

Functionattributes

Built-infunctions

Onefinalexample

Documentingyourcode

Importingobjects

Relativeimports

Summary

5.SavingTimeandMemory

map,zip,andfilter

map

zip

filter

Comprehensions

Nestedcomprehensions

Filteringacomprehension

dictcomprehensions

setcomprehensions

Generators

Generatorfunctions

Goingbeyondnext

Theyieldfromexpression

Generatorexpressions

Someperformanceconsiderations

Don’toverdocomprehensionsandgenerators

Namelocalization

Generationbehaviorinbuilt-ins

Onelastexample

Summary

6.AdvancedConcepts–OOP,Decorators,andIterators

Decorators

Adecoratorfactory

Object-orientedprogramming

ThesimplestPythonclass

Classandobjectnamespaces

Attributeshadowing

I,me,andmyself–usingtheselfvariable

Initializinganinstance

OOPisaboutcodereuse

Inheritanceandcomposition

Accessingabaseclass

Multipleinheritance

Methodresolutionorder

Staticandclassmethods

Staticmethods

Classmethods

Privatemethodsandnamemangling

Thepropertydecorator

Operatoroverloading

Polymorphism–abriefoverview

Writingacustomiterator

Summary

7.Testing,Profiling,andDealingwithExceptions

Testingyourapplication

Theanatomyofatest

Testingguidelines

Unittesting

Writingaunittest

Mockobjectsandpatching

Assertions

Aclassicunittestexample

Makingatestfail

Interfacetesting

Comparingtestswithandwithoutmocks

Boundariesandgranularity

Amoreinterestingexample

Test-drivendevelopment

Exceptions

ProfilingPython

Whentoprofile?

Summary

8.TheEdges–GUIsandScripts

Firstapproach–scripting

Theimports

Parsingarguments

Thebusinesslogic

Secondapproach–aGUIapplication

Theimports

Thelayoutlogic

Thebusinesslogic

Fetchingthewebpage

Savingtheimages

Alertingtheuser

Howtoimprovetheapplication?

Wheredowegofromhere?

Thetkinter.tixmodule

Theturtlemodule

wxPython,PyQt,andPyGTK

Theprincipleofleastastonishment

Threadingconsiderations

Summary

9.DataScience

IPythonandJupyternotebook

Dealingwithdata

Settingupthenotebook

Preparingthedata

Cleaningthedata

CreatingtheDataFrame

Unpackingthecampaignname

Unpackingtheuserdata

Cleaningeverythingup

SavingtheDataFrametoafile

Visualizingtheresults

Wheredowegofromhere?

Summary

10.WebDevelopmentDoneRight

WhatistheWeb?

HowdoestheWebwork?

TheDjangowebframework

Djangodesignphilosophy

Themodellayer

Theviewlayer

Thetemplatelayer

TheDjangoURLdispatcher

Regularexpressions

Aregexwebsite

SettingupDjango

Startingtheproject

Creatingusers

AddingtheEntrymodel

Customizingtheadminpanel

Creatingtheform

Writingtheviews

Thehomeview

Theentrylistview

Theformview

TyingupURLsandviews

Writingthetemplates

Thefutureofwebdevelopment

WritingaFlaskview

BuildingaJSONquoteserverinFalcon

Summary

11.DebuggingandTroubleshooting

Debuggingtechniques

Debuggingwithprint

Debuggingwithacustomfunction

Inspectingthetraceback

UsingthePythondebugger

Inspectinglogfiles

Othertechniques

Profiling

Assertions

Wheretofindinformation

Troubleshootingguidelines

Usingconsoleeditors

Wheretoinspect

Usingteststodebug

Monitoring

Summary

12.SummingUp–ACompleteExample

Thechallenge

Ourimplementation

ImplementingtheDjangointerface

Thesetup

Themodellayer

Asimpleform

Theviewlayer

Importsandhomeview

Listingallrecords

Creatingrecords

Updatingrecords

Deletingrecords

SettinguptheURLs

Thetemplatelayer

Homeandfootertemplates

Listingallrecords

Creatingandeditingrecords

TalkingtotheAPI

Deletingrecords

ImplementingtheFalconAPI

Themainapplication

Writingthehelpers

Codingthepasswordvalidator

Codingthepasswordgenerator

Writingthehandlers

Codingthepasswordvalidatorhandler

Codingthepasswordgeneratorhandler

RunningtheAPI

TestingtheAPI

Testingthehelpers

Testingthehandlers

Wheredoyougofromhere?

Summary

Awordoffarewell

Index

LearningPython

LearningPythonCopyright©2015PacktPublishing

Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.

PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

Firstpublished:December2015

Productionreference:1171215

PublishedbyPacktPublishingLtd.

LiveryPlace

35LiveryStreet

BirminghamB32PB,UK.

ISBN978-1-78355-171-2

www.packtpub.com

CreditsAuthor

FabrizioRomano

Reviewers

SimoneBurol

JulioVicenteTrigoGuijarro

VeitHeller

CommissioningEditor

AkramHussain

AcquisitionEditor

IndrajitDas

ContentDevelopmentEditors

SamanthaGonsalves

AdrianRaposo

TechnicalEditor

SiddhiRane

CopyEditors

JanbalDharmaraj

KevinMcGowan

ProjectCoordinator

KinjalBari

Proofreader

SafisEditing

Indexer

PriyaSane

Graphics

KirkD’Penha

AbhinashSahu

ProductionCoordinator

MelwynD’sa

CoverWork

MelwynD’sa

AbouttheAuthorFabrizioRomanowasborninItalyin1975.Heholdsamaster’sdegreeincomputerscienceengineeringfromtheUniversityofPadova.HeisalsoacertifiedScrummaster.

BeforePython,hehasworkedwithseveralotherlanguages,suchasC/C++,Java,PHP,andC#.

In2011,hemovedtoLondonandstartedworkingasaPythondeveloperforGlassesDirect,oneofEurope’sleadingonlineprescriptionglassesretailers.

HethenworkedasaseniorPythondeveloperforTBG(nowSprinklr),oneoftheworld’sleadingcompaniesinsocialmediaadvertising.AtTBG,heandhisteamcollaboratedwithFacebookandTwitter.TheywerethefirstintheworldtogetaccesstotheTwitteradvertisingAPI.Hewrotethecodethatpublishedthefirstgeo-narrowcastedpromotedtweetintheworldusingtheAPI.

HecurrentlyworksasaseniorplatformdeveloperatStudent.com,acompanythatisrevolutionizingthewayinternationalstudentsfindtheirperfecthomeallaroundtheworld

HehasdeliveredtalksonTeachingPythonandTDDwithPythonatthelasttwoeditionsofEuroPythonandatSkillsmatterinLondon.

AcknowledgementsIwouldliketothankAdrianRaposoandIndrajitDasfromPacktPublishingfortheirhelpandsupportandgivingmetheopportunitytolivethisadventure.IwouldalsoliketothankeveryoneatPacktPublishingwhohavecontributedtotherealizationofthisbook.SpecialthanksgotoSiddhiRane,mytechnicaleditor.Thankyouforyourkindness,forworkingveryhard,andforgoingtheextramilejusttomakemehappy.

IwouldliketoexpressmydeepestgratitudetoSimoneBurolandJulioTrigo,whohavegiftedmewithsomeoftheirpreciousfreetime.Theyhavereviewedthebookandprovidedmewithinvaluablefeedback.

Abigthankyoutomyteammates,MattBennettandJakubKubaBorys,fortheirinterestinthisbookandfortheirsupportandfeedbackthatmakesmeabettercodereveryday.

AheartfeltthankyoutoMarco“Tex”Beri,whointroducedmetoPythonwithanenthusiasmsecondtonone.

AspecialthankstoDr.NaomiCeder,fromwhomIlearnedsomuchoverthelastyear.Shehasgivenmeprecioussuggestionsandhasencouragedmetoembracethisopportunity.

Finally,Iwouldliketothankallmyfriendswhohavesupportedmeinanyway.

AbouttheReviewersSimoneBurolisanItaliansoftwaredeveloperwhowasborninTreviso(Italy)in1978.Heobtainedamaster’sdegreeincomputerscienceengineeringfromtheUniversityofPadua(Italy),andsincethenworkedinbankingfor5yearsinVenice(Italy).In2010,hemovedtoLondon(UnitedKingdom),whereheworkedinwarehouseautomationforOcadoTechnologyandtheninbankingforAlgomi.

JulioVicenteTrigoGuijarroisacomputerscientistandsoftwareengineerwithalmostadecadeofexperienceinsoftwaredevelopment.HeisalsoacertifiedScrummaster,whoenjoysthebenefitsofusingagilesoftwaredevelopment(ScrumandXP).

HecompletedhisstudiesincomputerscienceandsoftwareengineeringfromtheUniversityofAlicante,Spain,in2007.Sincethen,hehasworkedwithseveraltechnologiesandlanguages,includingMicrosoftDynamicsNAV,Java,JavaScript,andPython.

SomeoftheapplicationscoveredbyJulioduringhiscareerincludeRESTfulAPIs,ERPs,billingplatforms,paymentgateways,ande-commercewebsites.

HehasbeenusingPythononbothpersonalandprofessionalprojectssince2012,andheispassionateaboutsoftwaredesign,softwarequality,andcodingstandards.

Iwouldliketothankmyparentsfortheirlove,goodadvice,andcontinuoussupport.

IwouldalsoliketothankallmyfriendsthatImetalongtheway,whoenrichedmylife,formotivatingmeandhelpingmeprogress.

VeitHellerisafullstackdeveloper,mostlyworkingonthebackendsideofwebprojects.HecurrentlyresidesinBerlinandworksforaprototypicalPythonistacompanynamedBright.Inhisfreetime,hewritesinterpretersforvariousprogramminglanguages.

IwouldliketothankthepeopleatBrightforbeingawelcomingcompanythatsupportsmeinallmyendeavors,myfriendsandmyfamilyforcopingwithmystrangeness,andmanufacturersofcaffeinateddrinksworldwide.

www.PacktPub.com

Supportfiles,eBooks,discountoffers,andmoreForsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.

DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusat<service@packtpub.com>formoredetails.

Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

https://www2.packtpub.com/books/subscription/packtlib

DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigitalbooklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks.

Whysubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,print,andbookmarkcontentOndemandandaccessibleviaawebbrowser

FreeaccessforPacktaccountholdersIfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccessPacktLibtodayandview9entirelyfreebooks.Simplyuseyourlogincredentialsforimmediateaccess.

ToAlanTuring,thefatherofComputerScience.

ToGuidoVanRossum,thefatherofPython.

ToAdrianoRomano,myfather,mybiggestfan.

PrefaceShortlyafterIstartedwriting,afriendaskedmeiftherereallywasaneedofanotherLearningPythonbook.

Anexcellentquestionthatwecouldalsoexpressinanotherform:Whathasthisbooktooffer?WhatmakesthisbookdifferentfromtheaverageintroductorybookonPython?

Ithinktherearetwomaindifferencesandmanygoodreasonswhyyouwouldwanttoreadit.

Firstly,westartwithintroducingsomeimportantprogrammingconcepts.Webuildasolidfoundationbycoveringthecriticalaspectsofthiswonderfullanguage.

Thepacegraduallyincreases,alongwiththedifficultyofthesubjectspresented.BytheendofChapter7,Testing,Profiling,andDealingwithExceptions,wewillcoverallthefundamentals.

FromChapter8,TheEdges–GUIsandScripts,onward,thebooktakesasteepturn,whichbringsustodifferencenumbertwo.

Toconsolidatetheknowledgeacquired,thereisnothinglikeworkingonasmallproject.So,inthesecondpartofthebook,eachchapterdeliversaprojectonadifferentsubject.Weexplorescripting,graphicalinterfaces,datascience,andwebprogramming.

Eachprojectissmallenoughtofitwithinachapterandyetbigenoughtoberelevant.Eachchapterisinteresting,conveysamessage,andteachessomethingvaluable.

Afterashortsectionondebugging,thebookendswithacompleteexamplethatwrapsthingsup.Itriedtocraftitsothatyouwillbeabletoexpanditinseveralways.

So,thisisdefinitelynottheusualLearningPythonbook.Itsapproachismuchmore“hands-on”andpractical.

IwantedtoempoweryoutohelpyoubecomeatruePythonninja.ButIalsodidmybesttoentertainyouandfosteryourlogicalthinkingandcreativityalongtheway.

Now,haveIansweredthequestion?

WhatthisbookcoversChapter1,IntroductionandFirstSteps–TakeaDeepBreath,introducesyoutofundamentalprogrammingconcepts.ItguidesyoutogettingPythonupandrunningonyourcomputerandintroducesyoutosomeofitsconstructs.

Chapter2,Built-inDataTypes,introducesyoutoPythonbuilt-indatatypes.Pythonhasaveryrichsetofnativedatatypesandthischapterwillgiveyouadescriptionandashortexampleforeachofthem.

Chapter3,IteratingandMakingDecisions,teachesyouhowtocontroltheflowofyourcodebyinspectingconditions,applyinglogic,andperformingloops.

Chapter4,Functions,theBuildingBlocksofCode,teachesyouhowtowritefunctions.Functionsarethekeystoreusingcode,toreducingdebuggingtime,andingeneral,towritingbettercode.

Chapter5,SavingTimeandMemory,introducesyoutothefunctionalaspectsofPythonprogramming.Thischapterteachesyouhowtowritecomprehensionsandgenerators,whicharepowerfultoolsthatyoucanusetospeedupyourcodeandsavememory.

Chapter6,AdvancedConcepts–OOP,Decorators,andIterators,teachesyouthebasicsofobject-orientedprogrammingwithPython.Itshowsyouthekeyconceptsandallthepotentialsofthisparadigm.ItalsoshowsyouoneofthemostbelovedcharacteristicsofPython:decorators.Finally,italsocoverstheconceptofiterators.

Chapter7,Testing,Profiling,andDealingwithExceptions,teachesyouhowtomakeyourcodemorerobust,fast,andstableusingtechniquessuchastestingandprofiling.Italsoformallydefinestheconceptofexceptions.

Chapter8,TheEdges–GUIsandScripts,guidesyouthroughanexamplefromtwodifferentpointsofview.Theyareattheextremitiesofaspectrum:oneimplementationisascriptandtheotheroneapropergraphicaluserinterfaceapplication.

Chapter9,DataScience,introducesafewkeyconceptsandaveryspecialtool,theJupyterNotebook.

Chapter10,WebDevelopmentDoneRight,introducesthefundamentalsofwebdevelopmentanddeliversaprojectusingtheDjangowebframework.Theexamplewillbebasedonregularexpressions.

Chapter11,DebuggingandTroubleshooting,showsyouthemainmethodstodebugyourcodeandsomeexamplesonhowtoapplythem.

Chapter12,SummingUp–ACompleteExample,presentsaDjangowebsitethatactsasaninterfacetoanunderlyingslimAPIwrittenwiththeFalconwebframework.Thischaptertakesalltheconceptscoveredinthebooktothenextlevelandsuggestswheretogotodigdeeperandtakethenextsteps.

WhatyouneedforthisbookYouareencouragedtofollowtheexamplesinthisbook.Inordertodoso,youwillneedacomputer,anInternetconnection,andabrowser.ThebookiswritteninPython3.4,butitshouldalsoworkwithanyPython3.*version.IhavewritteninstructionsonhowtoinstallPythononthethreemainoperatingsystemsusedtoday:Windows,Mac,andLinux.Ihavealsoexplainedhowtoinstallalltheextralibrariesusedinthevariousexamplesandprovidedsuggestionsifthereaderfindsanyissuesduringtheinstallationofanyofthem.Noparticulareditorisrequiredtotypethecode;however,Isuggestthatthosewhoareinterestedinfollowingtheexamplesshouldconsideradoptingapropercodingenvironment.Ihavegivensuggestionsonthismatterinthefirstchapter.

WhothisbookisforPythonisthemostpopularintroductoryteachinglanguageinthetopcomputerscienceuniversitiesintheUS,soifyouarenewtosoftwaredevelopmentorifyouhavelittleexperienceandwouldliketostartoffontherightfoot,thenthislanguageandthisbookarewhatyouneed.Itsamazingdesignandportabilitywillhelpyoubecomeproductiveregardlessoftheenvironmentyouchoosetoworkwith.

IfyouhavealreadyworkedwithPythonoranyotherlanguage,thisbookcanstillbeusefultoyoubothasareferencetoPython’sfundamentalsandtoprovideawiderangeofconsiderationsandsuggestionscollectedovertwodecadesofexperience.

ConventionsInthisbook,youwillfindanumberoftextstylesthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheirmeaning.

Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“OpenupaPythonconsole,andtypeimportthis.”

Ablockofcodeissetasfollows:

#wedefineafunction,calledlocal

deflocal():

m=7

print(m)

m=5

print(m)

Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:

#wedefineafunction,calledlocal

deflocal():

m=7

print(m)

m=5

print(m)

Anycommand-lineinputoroutputiswrittenasfollows:

>>>frommathimportfactorial

>>>factorial(5)

120

Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,forexample,inmenusordialogboxes,appearinthetextlikethis:“ToopentheconsoleonWindows,gototheStartmenu,chooseRun,andtypecmd.”

NoteWarningsorimportantnotesappearinaboxlikethis.

TipTipsandtricksappearlikethis.

ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook—whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsusdeveloptitlesthatyouwillreallygetthemostoutof.

Tosendusgeneralfeedback,simplye-mail<feedback@packtpub.com>,andmentionthebook’stitleinthesubjectofyourmessage.

Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideatwww.packtpub.com/authors.

CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.

DownloadingtheexamplecodeYoucandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.

ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthecode—wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataundertheErratasectionofthattitle.

Toviewthepreviouslysubmittederrata,gotohttps://www.packtpub.com/books/content/supportandenterthenameofthebookinthesearchfield.TherequiredinformationwillappearundertheErratasection.

PiracyPiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.

Pleasecontactusat<copyright@packtpub.com>withalinktothesuspectedpiratedmaterial.

Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluablecontent.

QuestionsIfyouhaveaproblemwithanyaspectofthisbook,youcancontactusat<questions@packtpub.com>,andwewilldoourbesttoaddresstheproblem.

Chapter1.IntroductionandFirstSteps–TakeaDeepBreath “Giveamanafishandyoufeedhimforaday.Teachamantofishandyoufeedhimforalifetime.”

—Chineseproverb

AccordingtoWikipedia,computerprogrammingis:

“…aprocessthatleadsfromanoriginalformulationofacomputingproblemtoexecutablecomputerprograms.Programminginvolvesactivitiessuchasanalysis,developingunderstanding,generatingalgorithms,verificationofrequirementsofalgorithmsincludingtheircorrectnessandresourcesconsumption,andimplementation(commonlyreferredtoascoding)ofalgorithmsinatargetprogramminglanguage”.

Inanutshell,codingistellingacomputertodosomethingusingalanguageitunderstands.

Computersareverypowerfultools,butunfortunately,theycan’tthinkforthemselves.Sotheyneedtobetoldeverything.Theyneedtobetoldhowtoperformatask,howtoevaluateaconditiontodecidewhichpathtofollow,howtohandledatathatcomesfromadevicesuchasthenetworkoradisk,andhowtoreactwhensomethingunforeseenhappens,say,somethingisbrokenormissing.

Youcancodeinmanydifferentstylesandlanguages.Isithard?Iwouldsay“yes”and“no”.It’sabitlikewriting.Everybodycanlearnhowtowrite,andyoucantoo.Butwhatifyouwantedtobecomeapoet?Thenwritingaloneisnotenough.Youhavetoacquireawholeothersetofskillsandthiswilltakealongerandgreatereffort.

Intheend,itallcomesdowntohowfaryouwanttogodowntheroad.Codingisnotjustputtingtogethersomeinstructionsthatwork.Itissomuchmore!

Goodcodeisshort,fast,elegant,easytoreadandunderstand,simple,easytomodifyandextend,easytoscaleandrefactor,andeasytotest.Ittakestimetobeabletowritecodethathasallthesequalitiesatthesametime,butthegoodnewsisthatyou’retakingthefirststeptowardsitatthisverymomentbyreadingthisbook.AndIhavenodoubtyoucandoit.Anyonecan,infact,weallprogramallthetime,onlywearen’tawareofit.

Wouldyoulikeanexample?

Sayyouwanttomakeinstantcoffee.Youhavetogetamug,theinstantcoffeejar,ateaspoon,water,andthekettle.Evenifyou’renotawareofit,you’reevaluatingalotofdata.You’remakingsurethatthereiswaterinthekettleaswellasthekettleisplugged-in,thatthemugisclean,andthatthereisenoughcoffeeinthejar.Then,youboilthewaterandmaybeinthemeantimeyouputsomecoffeeinthemug.Whenthewaterisready,youpouritintothecup,andstir.

So,howisthisprogramming?

Well,wegatheredresources(thekettle,coffee,water,teaspoon,andmug)andweverifiedsomeconditionsonthem(kettleisplugged-in,mugisclean,thereisenoughcoffee).Thenwestartedtwoactions(boilingthewaterandputtingcoffeeinthemug),andwhenbothofthemwerecompleted,wefinallyendedtheprocedurebypouringwaterinthemugandstirring.

Canyouseeit?Ihavejustdescribedthehigh-levelfunctionalityofacoffeeprogram.Itwasn’tthathardbecausethisiswhatthebraindoesalldaylong:evaluateconditions,decidetotakeactions,carryouttasks,repeatsomeofthem,andstopatsomepoint.Cleanobjects,putthemback,andsoon.

Allyouneednowistolearnhowtodeconstructallthoseactionsyoudoautomaticallyinreallifesothatacomputercanactuallymakesomesenseofthem.Andyouneedtolearnalanguageaswell,toinstructit.

Sothisiswhatthisbookisfor.I’lltellyouhowtodoitandI’lltrytodothatbymeansofmanysimplebutfocusedexamples(myfavoritekind).

AproperintroductionIlovetomakereferencestotherealworldwhenIteachcoding;Ibelievetheyhelppeopleretaintheconceptsbetter.However,nowisthetimetobeabitmorerigorousandseewhatcodingisfromamoretechnicalperspective.

Whenwewritecode,we’reinstructingacomputeronwhatarethethingsithastodo.Wheredoestheactionhappen?Inmanyplaces:thecomputermemory,harddrives,networkcables,CPU,andsoon.It’sawhole“world”,whichmostofthetimeistherepresentationofasubsetoftherealworld.

Ifyouwriteapieceofsoftwarethatallowspeopletobuyclothesonline,youwillhavetorepresentrealpeople,realclothes,realbrands,sizes,andsoonandsoforth,withintheboundariesofaprogram.

Inordertodoso,youwillneedtocreateandhandleobjectsintheprogramyou’rewriting.Apersoncanbeanobject.Acarisanobject.Apairofsocksisanobject.Luckily,Pythonunderstandsobjectsverywell.

Thetwomainfeaturesanyobjecthasarepropertiesandmethods.Let’stakeapersonobjectasanexample.Typicallyinacomputerprogram,you’llrepresentpeopleascustomersoremployees.Thepropertiesthatyoustoreagainstthemarethingslikethename,theSSN,theage,iftheyhaveadrivinglicense,theire-mail,gender,andsoon.Inacomputerprogram,youstoreallthedatayouneedinordertouseanobjectforthepurposeyou’reserving.Ifyouarecodingawebsitetosellclothes,youprobablywanttostoretheheightandweightaswellasothermeasuresofyourcustomerssothatyoucansuggesttheappropriateclothesforthem.So,propertiesarecharacteristicsofanobject.Weusethemallthetime:“Couldyoupassmethatpen?”–“Whichone?”–“Theblackone.”Here,weusedthe“black”propertyofapentoidentifyit(mostlikelyamongstablueandaredone).

Methodsarethingsthatanobjectcando.Asaperson,Ihavemethodssuchasspeak,walk,sleep,wake-up,eat,dream,write,read,andsoon.AllthethingsthatIcandocouldbeseenasmethodsoftheobjectsthatrepresentsme.

So,nowthatyouknowwhatobjectsareandthattheyexposemethodsthatyoucanrunandpropertiesthatyoucaninspect,you’rereadytostartcoding.Codinginfactissimplyaboutmanagingthoseobjectsthatliveinthesubsetoftheworldthatwe’rereproducinginoursoftware.Youcancreate,use,reuse,anddeleteobjectsasyouplease.

AccordingtotheDataModelchapterontheofficialPythondocumentation:

“ObjectsarePython’sabstractionfordata.AlldatainaPythonprogramisrepresentedbyobjectsorbyrelationsbetweenobjects.”

We’lltakeacloserlookatPythonobjectsinChapter6,AdvancedConcepts–OOP,Decorators,andIterators.Fornow,allweneedtoknowisthateveryobjectinPythonhasanID(oridentity),atype,andavalue.

Oncecreated,theidentityofanobjectisneverchanged.It’sauniqueidentifierforit,andit’susedbehindthescenesbyPythontoretrievetheobjectwhenwewanttouseit.

Thetypeaswell,neverchanges.Thetypetellswhatoperationsaresupportedbytheobjectandthepossiblevaluesthatcanbeassignedtoit.

We’llseePython’smostimportantdatatypesinChapter2,Built-inDataTypes.

Thevaluecaneitherchangeornot.Ifitcan,theobjectissaidtobemutable,whilewhenitcannot,theobjectissaidtobeimmutable.

Howdoweuseanobject?Wegiveitanameofcourse!Whenyougiveanobjectaname,thenyoucanusethenametoretrievetheobjectanduseit.

Inamoregenericsense,objectssuchasnumbers,strings(text),collections,andsoonareassociatedwithaname.Usually,wesaythatthisnameisthenameofavariable.Youcanseethevariableasbeinglikeabox,whichyoucanusetoholddata.

So,youhavealltheobjectsyouneed:whatnow?Well,weneedtousethem,right?Wemaywanttosendthemoveranetworkconnectionorstoretheminadatabase.Maybedisplaythemonawebpageorwritethemintoafile.Inordertodoso,weneedtoreacttoauserfillinginaform,orpressingabutton,oropeningawebpageandperformingasearch.Wereactbyrunningourcode,evaluatingconditionstochoosewhichpartstoexecute,howmanytimes,andunderwhichcircumstances.

Andtodoallthis,basicallyweneedalanguage.That’swhatPythonisfor.Pythonisthelanguagewe’llusetogetherthroughoutthisbooktoinstructthecomputertodosomethingforus.

Now,enoughofthistheoreticalstuff,let’sgetstarted.

EnterthePythonPythonisthemarvelouscreatureofGuidoVanRossum,aDutchcomputerscientistandmathematicianwhodecidedtogifttheworldwithaprojecthewasplayingaroundwithoverChristmas1989.Thelanguageappearedtothepublicsomewherearound1991,andsincethenhasevolvedtobeoneoftheleadingprogramminglanguagesusedworldwidetoday.

IstartedprogrammingwhenIwas7yearsold,onaCommodoreVIC20,whichwaslaterreplacedbyitsbiggerbrother,theCommodore64.ThelanguagewasBASIC.Lateron,IlandedonPascal,Assembly,C,C++,Java,JavaScript,VisualBasic,PHP,ASP,ASP.NET,C#,andotherminorlanguagesIcannotevenremember,butonlywhenIlandedonPython,Ifinallyhadthatfeelingthatyouhavewhenyoufindtherightcouchintheshop.Whenallofyourbodypartsareyelling,“Buythisone!Thisoneisperfectforus!”

Ittookmeaboutadaytogetusedtoit.ItssyntaxisabitdifferentfromwhatIwasusedto,andingeneral,Iveryrarelyworkedwithalanguagethatdefinesscopingwithindentation.Butaftergettingpastthatinitialfeelingofdiscomfort(likehavingnewshoes),Ijustfellinlovewithit.Deeply.Let’sseewhy.

AboutPythonBeforewegetintothegorydetails,let’sgetasenseofwhysomeonewouldwanttousePython(IwouldrecommendyoutoreadthePythonpageonWikipediatogetamoredetailedintroduction).

Tomymind,Pythonexposesthefollowingqualities.

PortabilityPythonrunseverywhere,andportingaprogramfromLinuxtoWindowsorMacisusuallyjustamatteroffixingpathsandsettings.Pythonisdesignedforportabilityandittakescareofoperatingsystem(OS)specificquirksbehindinterfacesthatshieldyoufromthepainofhavingtowritecodetailoredtoaspecificplatform.

CoherencePythonisextremelylogicalandcoherent.Youcanseeitwasdesignedbyabrilliantcomputerscientist.Mostofthetimeyoucanjustguesshowamethodiscalled,ifyoudon’tknowit.

Youmaynotrealizehowimportantthisisrightnow,especiallyifyouareatthebeginning,butthisisamajorfeature.Itmeanslessclutteringinyourhead,lessskimmingthroughthedocumentation,andlessneedformappinginyourbrainwhenyoucode.

DeveloperproductivityAccordingtoMarkLutz(LearningPython,5thEdition,O’ReillyMedia),aPythonprogramistypicallyone-fifthtoone-thirdthesizeofequivalentJavaorC++code.Thismeansthejobgetsdonefaster.Andfasterisgood.Fastermeansafasterresponseonthemarket.Lesscodenotonlymeanslesscodetowrite,butalsolesscodetoread(andprofessionalcodersreadmuchmorethantheywrite),lesscodetomaintain,todebug,andtorefactor.

AnotherimportantaspectisthatPythonrunswithouttheneedoflengthyandtimeconsumingcompilationandlinkagesteps,soyoudon’thavetowaittoseetheresultsofyourwork.

AnextensivelibraryPythonhasanincrediblywidestandardlibrary(it’ssaidtocomewith“batteriesincluded”).Ifthatwasn’tenough,thePythoncommunityallovertheworldmaintainsabodyofthirdpartylibraries,tailoredtospecificneeds,whichyoucanaccessfreelyatthePythonPackageIndex(PyPI).WhenyoucodePythonandyourealizethatyouneedacertainfeature,inmostcases,thereisatleastonelibrarywherethatfeaturehasalreadybeenimplementedforyou.

SoftwarequalityPythonisheavilyfocusedonreadability,coherence,andquality.Thelanguageuniformityallowsforhighreadabilityandthisiscrucialnowadayswherecodeismoreofacollectiveeffortthanasoloexperience.AnotherimportantaspectofPythonisitsintrinsicmulti-paradigmnature.Youcanuseitasscriptinglanguage,butyoualsocanexploitobject-oriented,imperative,andfunctionalprogrammingstyles.Itisversatile.

SoftwareintegrationAnotherimportantaspectisthatPythoncanbeextendedandintegratedwithmanyotherlanguages,whichmeansthatevenwhenacompanyisusingadifferentlanguageastheirmainstreamtool,Pythoncancomeinandactasaglueagentbetweencomplexapplicationsthatneedtotalktoeachotherinsomeway.Thisiskindofanadvancedtopic,butintherealworld,thisfeatureisveryimportant.

SatisfactionandenjoymentLastbutnotleast,thefunofit!WorkingwithPythonisfun.Icancodefor8hoursandleavetheofficehappyandsatisfied,alientothestruggleothercodershavetoendurebecausetheyuselanguagesthatdon’tprovidethemwiththesameamountofwell-designeddatastructuresandconstructs.Pythonmakescodingfun,nodoubtaboutit.Andfunpromotesmotivationandproductivity.

ThesearethemajoraspectswhyIwouldrecommendPythontoeveryonefor.Ofcourse,therearemanyothertechnicalandadvancedfeaturesthatIcouldhavetalkedabout,buttheydon’treallypertaintoanintroductorysectionlikethisone.Theywillcomeupnaturally,chapterafterchapter,inthisbook.

Whatarethedrawbacks?Probably,theonlydrawbackthatonecouldfindinPython,whichisnotduetopersonalpreferences,istheexecutionspeed.Typically,Pythonisslowerthanitscompiledbrothers.ThestandardimplementationofPythonproduces,whenyourunanapplication,acompiledversionofthesourcecodecalledbytecode(withtheextension.pyc),whichisthenrunbythePythoninterpreter.Theadvantageofthisapproachisportability,whichwepayforwithaslowdownduetothefactthatPythonisnotcompileddowntomachinelevelasareotherlanguages.

However,Pythonspeedisrarelyaproblemtoday,henceitswideuseregardlessofthissuboptimalfeature.Whathappensisthatinreallife,hardwarecostisnolongeraproblem,andusuallyit’seasyenoughtogainspeedbyparallelizingtasks.Whenitcomestonumbercrunchingthough,onecanswitchtofasterPythonimplementations,suchasPyPy,whichprovidesanaverage7-foldspeedupbyimplementingadvancedcompilationtechniques(checkhttp://pypy.org/forreference).

Whendoingdatascience,you’llmostlikelyfindthatthelibrariesthatyouusewithPython,suchasPandasandNumpy,achievenativespeedduetothewaytheyareimplemented.

Ifthatwasn’tagoodenoughargument,youcanalwaysconsiderthatPythonisdrivingthebackendofservicessuchasSpotifyandInstagram,whereperformanceisaconcern.Nonetheless,Pythondoesitsjobperfectlyadequately.

WhoisusingPythontoday?Notyetconvinced?Let’stakeaverybrieflookatthecompaniesthatareusingPythontoday:Google,YouTube,Dropbox,Yahoo,ZopeCorporation,IndustrialLight&Magic,WaltDisneyFeatureAnimation,Pixar,NASA,NSA,RedHat,Nokia,IBM,Netflix,Yelp,Intel,Cisco,HP,Qualcomm,andJPMorganChase,justtonameafew.

EvengamessuchasBattlefield2,Civilization4,andQuArKareimplementedusingPython.

Pythonisusedinmanydifferentcontexts,suchassystemprogramming,webprogramming,GUIapplications,gamingandrobotics,rapidprototyping,systemintegration,datascience,databaseapplications,andmuchmore.

SettinguptheenvironmentBeforewetalkaboutinstallingPythononyoursystem,letmetellyouaboutwhichPythonversionI’llbeusinginthisbook.

Python2versusPython3–thegreatdebatePythoncomesintwomainversions—Python2,whichisthepast—andPython3,whichisthepresent.Thetwoversions,thoughverysimilar,areincompatibleonsomeaspects.

Intherealworld,Python2isactuallyquitefarfrombeingthepast.Inshort,eventhoughPython3hasbeenoutsince2008,thetransitionphaseisstillfarfrombeingover.ThisismostlyduetothefactthatPython2iswidelyusedintheindustry,andofcourse,companiesaren’tsokeenonupdatingtheirsystemsjustforthesakeofupdating,followingtheifitain’tbroke,don’tfixitphilosophy.YoucanreadallaboutthetransitionbetweenthetwoversionsontheWeb.

Anotherissuethatwashinderingthetransitionistheavailabilityofthird-partylibraries.Usually,aPythonprojectreliesontensofexternallibraries,andofcourse,whenyoustartanewproject,youneedtobesurethatthereisalreadyaversion3compatiblelibraryforanybusinessrequirementthatmaycomeup.Ifthat’snotthecase,startingabrandnewprojectinPython3meansintroducingapotentialrisk,whichmanycompaniesarenothappytotake.

Atthetimeofwriting,themajorityofthemostwidelyusedlibrarieshavebeenportedtoPython3,andit’squitesafetostartaprojectinPython3formostcases.Manyofthelibrarieshavebeenrewrittensothattheyarecompatiblewithbothversions,mostlyharnessingthepowerofthesix(2x3)library,whichhelpsintrospectingandadaptingthebehavioraccordingtotheversionused.

OnmyLinuxbox(Ubuntu14.04),IhavethefollowingPythonversion:

>>>importsys

>>>print(sys.version)

3.4.0(default,Apr112014,13:05:11)

[GCC4.8.2]

SoyoucanseethatmyPythonversionis3.4.0.TheprecedingtextisalittlebitofPythoncodethatItypedintomyconsole.We’lltalkaboutitinamoment.

AlltheexamplesinthisbookwillberunusingthisPythonversion.MostofthemwillrunalsoinPython2(Ihaveversion2.7.6installedaswell),andthosethatwon’twilljustrequiresomeminoradjustmentstocaterforthesmallincompatibilitiesbetweenthetwoversions.AnotherreasonbehindthischoiceisthatIthinkit’sbettertolearnPython3,andthen,ifyouneedto,learnthedifferencesithaswithPython2,ratherthangoingtheotherwayaround.

Don’tworryaboutthisversionthingthough:it’snotthatbiganissueinpractice.

InstallingPythonIneverreallygotthepointofhavingasetupsectioninabook,regardlessofwhatitisthatyouhavetosetup.Mostofthetime,betweenthetimetheauthorwritestheinstructionandthetimeyouactuallytrythemout,monthshavepassed.Thatis,ifyou’relucky.Oneversionchangeandthingsmaynotworkthewayitisdescribedinthebook.Luckily,wehavetheWebnow,soinordertohelpyougetupandrunning,I’lljustgiveyoupointersandobjectives.

TipIfanyoftheURLsorresourcesI’llpointyoutoarenolongertherebythetimeyoureadthisbook,justremember:Googleisyourfriend.

SettingupthePythoninterpreterFirstofall,let’stalkaboutyourOS.PythonisfullyintegratedandmostlikelyalreadyinstalledinbasicallyalmosteveryLinuxdistribution.IfyouhaveaMac,it’slikelythatPythonisalreadythereaswell(however,possiblyonlyPython2.7),whereasifyou’reusingWindows,youprobablyneedtoinstallit.

GettingPythonandthelibrariesyouneedupandrunningrequiresabitofhandiwork.LinuxhappenstobethemostuserfriendlyOSforPythonprogrammers,Windowsontheotherhandistheonethatrequiresthebiggesteffort,Macbeingsomewhereinbetween.Forthisreason,ifyoucanchoose,IsuggestyoutouseLinux.Ifyoucan’t,andyouhaveaMac,thengoforitanyway.IfyouuseWindows,you’llbefinefortheexamplesinthisbook,butingeneralworkingwithPythonwillrequireyouabitmoretweaking.

MyOSisUbuntu14.04,andthisiswhatIwillusethroughoutthebook,alongwithPython3.4.0.

TheplaceyouwanttostartistheofficialPythonwebsite:https://www.python.org.ThiswebsitehoststheofficialPythondocumentationandmanyotherresourcesthatyouwillfindveryuseful.Takethetimetoexploreit.

TipAnotherexcellent,resourcefulwebsiteonPythonanditsecosystemishttp://docs.python-guide.org.

FindthedownloadsectionandchoosetheinstallerforyourOS.IfyouareonWindows,makesurethatwhenyouruntheinstaller,youchecktheoptioninstallpip(actually,Iwouldsuggesttomakeacompleteinstallation,justtobesafe,ofallthecomponentstheinstallerholds).We’lltalkaboutpiplater.

NowthatPythonisinstalledinyoursystem,theobjectiveistobeabletoopenaconsoleandrunthePythoninteractiveshellbytypingpython.

NotePleasenotethatIusuallyrefertothePythoninteractiveshellsimplyasPythonconsole.

ToopentheconsoleinWindows,gototheStartmenu,chooseRun,andtypecmd.Ifyouencounteranythingthatlookslikeapermissionproblemwhileworkingontheexamplesofthisbook,pleasemakesureyouarerunningtheconsolewithadministratorrights.

OntheMacOSX,youcanstartaterminalbygoingtoApplications|Utilities|Terminal.

IfyouareonLinux,youknowallthatthereistoknowabouttheconsole.

NoteIwillusethetermconsoleinterchangeablytoindicatetheLinuxconsole,theWindowscommandprompt,andtheMacterminal.Iwillalsoindicatethecommand-linepromptwiththeLinuxdefaultformat,likethis:

$sudoapt-getupdate

Whateverconsoleyouopen,typepythonattheprompt,andmakesurethePythoninteractiveshellshowsup.Typeexit()toquit.Keepinmindthatyoumayhavetospecifypython3ifyourOScomeswithPython2.*preinstalled.

ThisishowitshouldlookonWindows7:

AndthisishowitshouldlookonLinux:

NowthatPythonissetupandyoucanrunit,it’stimetomakesureyouhavetheothertoolthatwillbeindispensabletofollowtheexamplesinthebook:virtualenv.

AboutvirtualenvAsyouprobablyhaveguessedbyitsname,virtualenvisallaboutvirtualenvironments.Letmeexplainwhattheyareandwhyweneedthemandletmedoitbymeansofasimpleexample.

YouinstallPythononyoursystemandyoustartworkingonawebsiteforclientX.Youcreateaprojectfolderandstartcoding.Alongthewayyoualsoinstallsomelibraries,forexampletheDjangoframework,whichwe’llseeindepthinChapter10,WebDevelopmentDoneRight.Let’ssaytheDjangoversionyouinstallforprojectXis1.7.1.

Now,yourwebsiteissogoodthatyougetanotherclient,Y.Hewantsyoutobuildanotherwebsite,soyoustartprojectYand,alongtheway,youneedtoinstallDjangoagain.TheonlyissueisthatnowtheDjangoversionis1.8andyoucannotinstallitonyoursystembecausethiswouldreplacetheversionyouinstalledforprojectX.Youdon’twanttoriskintroducingincompatibilityissues,soyouhavetwochoices:eitheryoustickwiththeversionyouhavecurrentlyonyourmachine,oryouupgradeitandmakesurethefirstprojectisstillfullyworkingcorrectlywiththenewversion.

Let’sbehonest,neitheroftheseoptionsisveryappealing,right?Definitelynot.So,here’sthesolution:virtualenv!

virtualenvisatoolthatallowsyoutocreateavirtualenvironment.Inotherwords,itisatooltocreateisolatedPythonenvironments,eachofwhichisafolderthatcontainsallthenecessaryexecutablestousethepackagesthataPythonprojectwouldneed(thinkofpackagesaslibrariesforthetimebeing).

SoyoucreateavirtualenvironmentforprojectX,installallthedependencies,andthenyoucreateavirtualenvironmentforprojectY,installingallitsdependencieswithouttheslightestworrybecauseeverylibraryyouinstallendsupwithintheboundariesoftheappropriatevirtualenvironment.Inourexample,projectXwillholdDjango1.7.1,whileprojectYwillholdDjango1.8.

NoteItisofvitalimportancethatyouneverinstalllibrariesdirectlyatthesystemlevel.LinuxforexamplereliesonPythonformanydifferenttasksandoperations,andifyoufiddlewiththesysteminstallationofPython,youriskcompromisingtheintegrityofthewholesystem(guesstowhomthishappened…).Sotakethisasarule,suchasbrushingyourteethbeforegoingtobed:always,alwayscreateavirtualenvironmentwhenyoustartanewproject.

Toinstallvirtualenvonyoursystem,thereareafewdifferentways.OnaDebian-baseddistributionofLinuxforexample,youcaninstallitwiththefollowingcommand:

$sudoapt-getinstallpython-virtualenv

Probably,theeasiestwayistousepipthough,withthefollowingcommand:

$sudopipinstallvirtualenv#sudomaybyoptional

pipisapackagemanagementsystemusedtoinstallandmanagesoftwarepackageswritteninPython.

Python3hasbuilt-insupportforvirtualenvironments,butinpractice,theexternallibrariesarestillthedefaultonproductionsystems.Ifyouhavetroublegettingvirtualenvupandrunning,pleaserefertothevirtualenvofficialwebsite:https://virtualenv.pypa.io.

YourfirstvirtualenvironmentItisveryeasytocreateavirtualenvironment,butaccordingtohowyoursystemisconfiguredandwhichPythonversionyouwantthevirtualenvironmenttorun,youneedtorunthecommandproperly.Anotherthingyouwillneedtodowithavirtualenv,whenyouwanttoworkwithit,istoactivateit.ActivatingavirtualenvbasicallyproducessomepathjugglingbehindthescenessothatwhenyoucallthePythoninterpreter,you’reactuallycallingtheactivevirtualenvironmentone,insteadofthemeresystemone.

I’llshowyouafullexampleonbothLinuxandWindows.Wewill:

1. Createafoldernamedlearning.pythonunderyourprojectroot(whichinmycaseisafoldercalledsrv,inmyhomefolder).Pleaseadaptthepathsaccordingtothesetupyoufancyonyourbox.

2. Withinthelearning.pythonfolder,wewillcreateavirtualenvironmentcalled.lpvenv.

NoteSomedevelopersprefertocallallvirtualenvironmentsusingthesamename(forexample,.venv).Thiswaytheycanrunscriptsagainstanyvirtualenvbyjustknowingthenameoftheprojecttheydwellin.ThisisaverycommontechniquethatIuseaswell.Thedotin.venvisbecauseinLinux/Macprependinganamewithadotmakesthatfileorfolderinvisible.

3. Aftercreatingthevirtualenvironment,wewillactivateit(thisisslightlydifferentbetweenLinux,Mac,andWindows).

4. Then,we’llmakesurethatwearerunningthedesiredPythonversion(3.4.*)byrunningthePythoninteractiveshell.

5. Finally,wewilldeactivatethevirtualenvironmentusingthedeactivatecommand.

Thesefivesimplestepswillshowyouallyouhavetodotostartanduseaproject.

Here’sanexampleofhowthosestepsmightlooklikeonLinux(commandsthatstartwitha#arecomments):

NoticethatIhadtoexplicitlytellvirtualenvtousethePython3.4interpreterbecauseonmyboxPython2.7isthedefaultone.HadInotdonethat,IwouldhavehadavirtualenvironmentwithPython2.7insteadofPython3.4.

Youcancombinethetwoinstructionsforstep2inonesinglecommandlikethis:

$virtualenv-p$(whichpython3.4).lpvenv

Ipreferredtobeexplicitlyverboseinthisinstance,tohelpyouunderstandeachbitoftheprocedure.

Anotherthingtonoticeisthatinordertoactivateavirtualenvironment,weneedtorunthe/bin/activatescript,whichneedstobesourced(whenascriptis“sourced”,itmeansthatitseffectsstickaroundwhenit’sdonerunning).Thisisveryimportant.Alsonoticehowthepromptchangesafterweactivatethevirtualenvironment,showingitsnameontheleft(andhowitdisappearswhenwedeactivate).InMacOS,thestepsarethesamesoIwon’trepeatthemhere.

Nowlet’shavealookathowwecanachievethesameresultinWindows.Youwillprobablyhavetoplayaroundabit,especiallyifyouhaveadifferentWindowsorPythonversionthanI’musinghere.Thisisallgoodexperiencethough,sotryandthinkpositivelyattheinitialstrugglethateverycoderhastogothroughinordertogetthingsgoing.

Here’showitshouldlookonWindows(commandsthatstartwith::arecomments):

NoticethereareafewsmalldifferencesfromtheLinuxversion.Apartfromthecommandstocreateandnavigatethefolders,oneimportantdifferenceishowyouactivateyourvirtualenv.Also,inWindowsthereisnowhichcommand,soweusedthewherecommand.

Atthispoint,youshouldbeabletocreateandactivateavirtualenvironment.Pleasetryandcreateanotheronewithoutmeguidingyou,getacquaintedtothisprocedurebecauseit’ssomethingthatyouwillalwaysbedoing:weneverworksystem-widewithPython,remember?It’sextremelyimportant.

So,withthescaffoldingoutoftheway,we’rereadytotalkabitmoreaboutPythonandhowyoucanuseit.Beforewedoitthough,allowmetospendafewwordsabouttheconsole.

Yourfriend,theconsoleInthiseraofGUIsandtouchscreendevices,itseemsalittleridiculoustohavetoresorttoatoolsuchastheconsole,wheneverythingisjustaboutoneclickaway.

Butthetruthiseverytimeyouremoveyourrighthandfromthekeyboard(ortheleftone,ifyou’realefty)tograbyourmouseandmovethecursorovertothespotyouwanttoclick,you’relosingtime.Gettingthingsdonewiththeconsole,counter-intuitivelyasitmaybe,resultsinhigherproductivityandspeed.Iknow,youhavetotrustmeonthis.

Speedandproductivityareimportantandpersonally,Ihavenothingagainstthemouse,butthereisanotherverygoodreasonforwhichyoumaywanttogetwellacquaintedwiththeconsole:whenyoudevelopcodethatendsuponsomeserver,theconsolemightbetheonlyavailabletool.Ifyoumakefriendswithit,Ipromiseyou,youwillnevergetlostwhenit’sofutmostimportancethatyoudon’t(typically,whenthewebsiteisdownandyouhavetoinvestigateveryquicklywhat’sgoingon).

Soit’sreallyuptoyou.Ifyou’reindoubt,pleasegrantmethebenefitofthedoubtandgiveitatry.It’seasierthanyouthink,andyou’llneverregretit.ThereisnothingmorepitifulthanagooddeveloperwhogetslostwithinanSSHconnectiontoaserverbecausetheyareusedtotheirowncustomsetoftools,andonlytothat.

Now,let’sgetbacktoPython.

HowyoucanrunaPythonprogramThereareafewdifferentwaysinwhichyoucanrunaPythonprogram.

RunningPythonscriptsPythoncanbeusedasascriptinglanguage.Infact,italwaysprovesitselfveryuseful.Scriptsarefiles(usuallyofsmalldimensions)thatyounormallyexecutetodosomethinglikeatask.Manydevelopersenduphavingtheirownarsenaloftoolsthattheyfirewhentheyneedtoperformatask.Forexample,youcanhavescriptstoparsedatainaformatandrenderitintoanotherdifferentformat.Oryoucanuseascripttoworkwithfilesandfolders.Youcancreateormodifyconfigurationfiles,andmuchmore.Technically,thereisnotmuchthatcannotbedoneinascript.

It’squitecommontohavescriptsrunningataprecisetimeonaserver.Forexample,ifyourwebsitedatabaseneedscleaningevery24hours(forexample,thetablethatstorestheusersessions,whichexpireprettyquicklybutaren’tcleanedautomatically),youcouldsetupacronjobthatfiresyourscriptat3:00A.M.everyday.

NoteAccordingtoWikipedia,thesoftwareutilityCronisatime-basedjobschedulerinUnix-likecomputeroperatingsystems.Peoplewhosetupandmaintainsoftwareenvironmentsusecrontoschedulejobs(commandsorshellscripts)torunperiodicallyatfixedtimes,dates,orintervals.

IhavePythonscriptstodoallthemenialtasksthatwouldtakememinutesormoretodomanually,andatsomepoint,Idecidedtoautomate.Forexample,Ihavealaptopthatdoesn’thaveaFnkeytotogglethetouchpadonandoff.Ifindthisveryannoying,andIdon’twanttogoclickingaboutthroughseveralmenuswhenIneedtodoit,soIwroteasmallscriptthatissmartenoughtotellmysystemtotogglethetouchpadactivestate,andnowIcandoitwithonesimpleclickfrommylauncher.Priceless.

We’lldevotehalfofChapter8,TheEdges–GUIsandScriptsonscriptingwithPython.

RunningthePythoninteractiveshellAnotherwayofrunningPythonisbycallingtheinteractiveshell.Thisissomethingwealreadysawwhenwetypedpythononthecommandlineofourconsole.

Soopenaconsole,activateyourvirtualenvironment(whichbynowshouldbesecondnaturetoyou,right?),andtypepython.Youwillbepresentedwithacoupleoflinesthatshouldlooklikethis(ifyouareonLinux):

Python3.4.0(default,Apr112014,13:05:11)

[GCC4.8.2]onlinux

Type"help","copyright","credits"or"license"formoreinformation.

Those>>>arethepromptoftheshell.TheytellyouthatPythoniswaitingforyoutotypesomething.Ifyoutypeasimpleinstruction,somethingthatfitsinoneline,that’sallyou’llsee.However,ifyoutypesomethingthatrequiresmorethanonelineofcode,theshellwillchangethepromptto...,givingyouavisualcluethatyou’retypingamultilinestatement(oranythingthatwouldrequiremorethanonelineofcode).

Goon,tryitout,let’sdosomebasicmaths:

>>>2+4

6

>>>10/4

2.5

>>>2**1024

179769313486231590772930519078902473361797697894230657273430081157732675805

500963132708477322407536021120113879871393357658789768814416622492847430639

474124377767893424865485276302219601246094119453082952085005768838150682342

462881473913110540827237163350510684586298239947245938479716304835356329624

224137216

Thelastoperationisshowingyousomethingincredible.Weraise2tothepowerof1024,andPythonishandlingthistaskwithnotroubleatall.TrytodoitinJava,C++,orC#.Itwon’twork,unlessyouusespeciallibrariestohandlesuchbignumbers.

Iusetheinteractiveshelleveryday.It’sextremelyusefultodebugveryquickly,forexample,tocheckifadatastructuresupportsanoperation.Ormaybetoinspectorrunapieceofcode.

WhenyouuseDjango(awebframework),theinteractiveshelliscoupledwithitandallowsyoutoworkyourwaythroughtheframeworktools,toinspectthedatainthedatabase,andmanymorethings.Youwillfindthattheinteractiveshellwillsoonbecomeoneofyourdearestfriendsonthejourneyyouareembarkingon.

Anothersolution,whichcomesinamuchnicergraphiclayout,istouseIDLE(IntegratedDeveLopmentEnvironment).It’squiteasimpleIDE,whichisintendedmostlyforbeginners.Ithasaslightlylargersetofcapabilitiesthanthenakedinteractiveshellyougetintheconsole,soyoumaywanttoexploreit.ItcomesforfreeintheWindowsPythoninstallerandyoucaneasilyinstallitinanyothersystem.YoucanfindinformationaboutitonthePythonwebsite.

GuidoVanRossumnamedPythonaftertheBritishcomedygroupMontyPython,soit’srumoredthatthenameIDLEhasbeenchoseninhonorofErikIdle,oneofMontyPython’sfoundingmembers.

RunningPythonasaserviceApartfrombeingrunasascript,andwithintheboundariesofashell,Pythoncanbecodedandrunaspropersoftware.We’llseemanyexamplesthroughoutthebookaboutthismode.Andwe’llunderstandmoreaboutitinamoment,whenwe’lltalkabouthowPythoncodeisorganizedandrun.

RunningPythonasaGUIapplicationPythoncanalsoberunasaGUI(GraphicalUserInterface).Thereareseveralframeworksavailable,someofwhicharecross-platformandsomeothersareplatform-specific.InChapter8,TheEdges–GUIsandScripts,we’llseeanexampleofaGUIapplicationcreatedusingTkinter,whichisanobject-orientedlayerthatlivesontopofTk(TkintermeansTkInterface).

NoteTkisagraphicaluserinterfacetoolkitthattakesdesktopapplicationdevelopmenttoahigherlevelthantheconventionalapproach.ItisthestandardGUIforTcl(ToolCommandLanguage),butalsoformanyotherdynamiclanguagesandcanproducerichnativeapplicationsthatrunseamlesslyunderWindows,Linux,MacOSX,andmore.

TkintercomesbundledwithPython,thereforeitgivestheprogrammereasyaccesstotheGUIworld,andforthesereasons,IhavechosenittobetheframeworkfortheGUIexamplesthatI’llpresentinthisbook.

AmongtheotherGUIframeworks,wefindthatthefollowingarethemostwidelyused:

PyQtwxPythonPyGtk

Describingthemindetailisoutsidethescopeofthisbook,butyoucanfindalltheinformationyouneedonthePythonwebsiteintheGUIProgrammingsection.IfGUIsarewhatyou’relookingfor,remembertochoosetheoneyouwantaccordingtosomeprinciples.Makesurethey:

OfferallthefeaturesyoumayneedtodevelopyourprojectRunonalltheplatformsyoumayneedtosupportRelyonacommunitythatisaswideandactiveaspossibleWrapgraphicdrivers/toolsthatyoucaneasilyinstall/access

HowisPythoncodeorganizedLet’stalkalittlebitabouthowPythoncodeisorganized.Inthisparagraph,we’llstartgoingdowntherabbitholealittlebitmoreandintroduceabitmoretechnicalnamesandconcepts.

Startingwiththebasics,howisPythoncodeorganized?Ofcourse,youwriteyourcodeintofiles.Whenyousaveafilewiththeextension.py,thatfileissaidtobeaPythonmodule.

TipIfyou’reonWindowsorMac,whichtypicallyhidefileextensionstotheuser,pleasemakesureyouchangetheconfigurationsothatyoucanseethecompletenameofthefiles.Thisisnotstrictlyarequirement,butaheartysuggestion.

Itwouldbeimpracticaltosaveallthecodethatitisrequiredforsoftwaretoworkwithinonesinglefile.Thatsolutionworksforscripts,whichareusuallynotlongerthanafewhundredlines(andoftentheyarequiteshorterthanthat).

AcompletePythonapplicationcanbemadeofhundredsofthousandsoflinesofcode,soyouwillhavetoscatteritthroughdifferentmodules.Better,butnotnearlygoodenough.Itturnsoutthatevenlikethisitwouldstillbeimpracticaltoworkwiththecode.SoPythongivesyouanotherstructure,calledpackage,whichallowsyoutogroupmodulestogether.Apackageisnothingmorethanafolder,whichmustcontainaspecialfile,__init__.pythatdoesn’tneedtoholdanycodebutwhosepresenceisrequiredtotellPythonthatthefolderisnotjustsomefolder,butit’sactuallyapackage(notethatasofPython3.3__init__.pyisnotstrictlyrequiredanymore).

Asalways,anexamplewillmakeallofthismuchclearer.Ihavecreatedanexamplestructureinmybookproject,andwhenItypeinmyLinuxconsole:

$tree-vexample

Igetatreerepresentationofthecontentsofthech1/examplefolder,whichholdsthecodefortheexamplesofthischapter.Here’showastructureofarealsimpleapplicationcouldlooklike:

example/

├──core.py

├──run.py

└──util

├──__init__.py

├──db.py

├──math.py

└──network.py

Youcanseethatwithintherootofthisexample,wehavetwomodules,core.pyandrun.py,andonepackage:util.Withincore.py,theremaybethecorelogicofourapplication.Ontheotherhand,withintherun.pymodule,wecanprobablyfindthelogictostarttheapplication.Withintheutilpackage,Iexpecttofindvariousutilitytools,and

infact,wecanguessthatthemodulestherearecalledbythetypeoftoolstheyhold:db.pywouldholdtoolstoworkwithdatabases,math.pywouldofcourseholdmathematicaltools(maybeourapplicationdealswithfinancialdata),andnetwork.pywouldprobablyholdtoolstosend/receivedataonnetworks.

Asexplainedbefore,the__init__.pyfileistherejusttotellPythonthatutilisapackageandnotjustamerefolder.

Hadthissoftwarebeenorganizedwithinmodulesonly,itwouldhavebeenmuchhardertoinferitsstructure.Iputamoduleonlyexampleunderthech1/files_onlyfolder,seeitforyourself:

$tree-vfiles_only

Thisshowsusacompletelydifferentpicture:

files_only/

├──core.py

├──db.py

├──math.py

├──network.py

└──run.py

Itisalittlehardertoguesswhateachmoduledoes,right?Now,considerthatthisisjustasimpleexample,soyoucanguesshowmuchharderitwouldbetounderstandarealapplicationifwecouldn’torganizethecodeinpackagesandmodules.

HowdoweusemodulesandpackagesWhenadeveloperiswritinganapplication,itisverylikelythattheywillneedtoapplythesamepieceoflogicindifferentpartsofit.Forexample,whenwritingaparserforthedatathatcomesfromaformthatausercanfillinawebpage,theapplicationwillhavetovalidatewhetheracertainfieldisholdinganumberornot.Regardlessofhowthelogicforthiskindofvalidationiswritten,it’sverylikelythatitwillbeneededinmorethanoneplace.Forexampleinapollapplication,wheretheuserisaskedmanyquestion,it’slikelythatseveralofthemwillrequireanumericanswer.Forexample:

WhatisyourageHowmanypetsdoyouownHowmanychildrendoyouhaveHowmanytimeshaveyoubeenmarried

Itwouldbeverybadpracticetocopypaste(or,moreproperlysaid:duplicate)thevalidationlogicineveryplacewhereweexpectanumericanswer.ThiswouldviolatetheDRY(Don’tRepeatYourself)principle,whichstatesthatyoushouldneverrepeatthesamepieceofcodemorethanonceinyourapplication.Ifeeltheneedtostresstheimportanceofthisprinciple:youshouldneverrepeatthesamepieceofcodemorethanonceinyourapplication(gottheirony?).

Thereareseveralreasonswhyrepeatingthesamepieceoflogiccanbeverybad,themostimportantonesbeing:

Therecouldbeabuginthelogic,andtherefore,youwouldhavetocorrectitineveryplacethatlogicisapplied.Youmaywanttoamendthewayyoucarryoutthevalidation,andagainyouwouldhavetochangeitineveryplaceitisapplied.Youmayforgettofix/amendapieceoflogicbecauseyoumisseditwhensearchingforallitsoccurrences.Thiswouldleavewrong/inconsistentbehaviorinyourapplication.Yourcodewouldbelongerthanneeded,fornogoodreason.

Pythonisawonderfullanguageandprovidesyouwithallthetoolsyouneedtoapplyallthecodingbestpractices.Forthisparticularexample,weneedtobeabletoreuseapieceofcode.Tobeabletoreuseapieceofcode,weneedtohaveaconstructthatwillholdthecodeforussothatwecancallthatconstructeverytimeweneedtorepeatthelogicinsideit.Thatconstructexists,andit’scalledfunction.

I’mnotgoingtoodeepintothespecificshere,sopleasejustrememberthatafunctionisablockoforganized,reusablecodewhichisusedtoperformatask.Functionscanassumemanyformsandnames,accordingtowhatkindofenvironmenttheybelongto,butfornowthisisnotimportant.We’llseethedetailswhenweareabletoappreciatethem,lateron,inthebook.Functionsarethebuildingblocksofmodularityinyourapplication,andtheyarealmostindispensable(unlessyou’rewritingasupersimplescript,you’llusefunctionsallthetime).We’llexplorefunctionsinChapter4,Functions,theBuildingBlocksofCode.

Pythoncomeswithaveryextensivelibrary,asIalreadysaidafewpagesago.Now,maybeit’sagoodtimetodefinewhatalibraryis:alibraryisacollectionoffunctionsandobjectsthatprovidefunctionalitiesthatenrichtheabilitiesofalanguage.

Forexample,withinPython’smathlibrarywecanfindaplethoraoffunctions,oneofwhichisthefactorialfunction,whichofcoursecalculatesthefactorialofanumber.

NoteInmathematics,thefactorialofanon-negativeintegernumberN,denotedasN!,isdefinedastheproductofallpositiveintegerslessthanorequaltoN.Forexample,thefactorialof5iscalculatedas:

5!=5*4*3*2*1=120

Thefactorialof0is0!=1,torespecttheconventionforanemptyproduct.

So,ifyouwantedtousethisfunctioninyourcode,allyouwouldhavetodoistoimportitandcallitwiththerightinputvalues.Don’tworrytoomuchifinputvaluesandtheconceptofcallingisnotveryclearfornow,pleasejustconcentrateontheimportpart.

NoteWeusealibrarybyimportingwhatweneedfromit,andthenweuseit.

InPython,tocalculatethefactorialofnumber5,wejustneedthefollowingcode:

>>>frommathimportfactorial

>>>factorial(5)

120

NoteWhateverwetypeintheshell,ifithasaprintablerepresentation,willbeprintedontheconsoleforus(inthiscase,theresultofthefunctioncall:120).

So,let’sgobacktoourexample,theonewithcore.py,run.py,util,andsoon.

Inourexample,thepackageutilisourutilitylibrary.Ourcustomutilitybeltthatholdsallthosereusabletools(thatis,functions),whichweneedinourapplication.Someofthemwilldealwithdatabases(db.py),somewiththenetwork(network.py),andsomewillperformmathematicalcalculations(math.py)thatareoutsidethescopeofPython’sstandardmathlibraryandtherefore,wehadtocodethemforourselves.

Wewillseeindetailhowtoimportfunctionsandusethemintheirdedicatedchapter.Let’snowtalkaboutanotherveryimportantconcept:Python’sexecutionmodel.

Python’sexecutionmodelInthisparagraph,Iwouldliketointroduceyoutoafewveryimportantconcepts,suchasscope,names,andnamespaces.YoucanreadallaboutPython’sexecutionmodelintheofficialLanguagereference,ofcourse,butIwouldarguethatitisquitetechnicalandabstract,soletmegiveyoualessformalexplanationfirst.

NamesandnamespacesSayyouarelookingforabook,soyougotothelibraryandasksomeoneforthebookyouwanttofetch.Theytellyousomethinglike“secondfloor,sectionX,rowthree”.Soyougoupthestairs,lookforsectionX,andsoon.

Itwouldbeverydifferenttoenteralibrarywhereallthebooksarepiledtogetherinrandomorderinonebigroom.Nofloors,nosections,norows,noorder.Fetchingabookwouldbeextremelyhard.

Whenwewritecodewehavethesameissue:wehavetotryandorganizeitsothatitwillbeeasyforsomeonewhohasnopriorknowledgeaboutittofindwhatthey’relookingfor.Whensoftwareisstructuredcorrectly,italsopromotescodereuse.Ontheotherhand,disorganizedsoftwareismorelikelytoexposescatteredpiecesofduplicatedlogic.

Firstofall,let’sstartwiththebook.WerefertoabookbyitstitleandinPythonlingo,thatwouldbeaname.Pythonnamesaretheclosestabstractiontowhatotherlanguagescallvariables.Namesbasicallyrefertoobjectsandareintroducedbynamebindingoperations.Let’smakeaquickexample(noticethatanythingthatfollowsa#isacomment):

>>>n=3#integernumber

>>>address="221bBakerStreet,NW16XE,London"#S.Holmes

>>>employee={

...'age':45,

...'role':'CTO',

...'SSN':'AB1234567',

...}

>>>#let'sprintthem

>>>n

3

>>>address

'221bBakerStreet,NW16XE,London'

>>>employee

{'role':'CTO','SSN':'AB1234567','age':45}

>>>#whatifItrytoprintanameIdidn'tdefine?

>>>other_name

Traceback(mostrecentcalllast):

File"<stdin>",line1,in<module>

NameError:name'other_name'isnotdefined

Wedefinedthreeobjectsintheprecedingcode(doyourememberwhatarethethreefeatureseveryPythonobjecthas?):

Anintegernumbern(type:int,value:3)Astringaddress(type:str,value:SherlockHolmes’address)Adictionaryemployee(type:dict,value:adictionarywhichholdsthreekey/valuepairs)

Don’tworry,Iknowyou’renotsupposedtoknowwhatadictionaryis.We’llseeinthenextchapterthatit’sthekingofPythondatastructures.

Tip

Haveyounoticedthatthepromptchangedfrom>>>to...whenItypedinthedefinitionofemployee?That’sbecausethedefinitionspansovermultiplelines.

So,whataren,addressandemployee?Theyarenames.Namesthatwecanusetoretrievedatawithinourcode.Theyneedtobekeptsomewheresothatwheneverweneedtoretrievethoseobjects,wecanusetheirnamestofetchthem.Weneedsomespacetoholdthem,hence:namespaces!

Anamespaceisthereforeamappingfromnamestoobjects.Examplesarethesetofbuilt-innames(containingfunctionsthatarealwaysaccessibleforfreeinanyPythonprogram),theglobalnamesinamodule,andthelocalnamesinafunction.Eventhesetofattributesofanobjectcanbeconsideredanamespace.

Thebeautyofnamespacesisthattheyallowyoutodefineandorganizeyournameswithclarity,withoutoverlappingorinterference.Forexample,thenamespaceassociatedwiththatbookwewerelookingforinthelibrarycanbeusedtoimportthebookitself,likethis:

fromlibrary.second_floor.section_x.row_threeimportbook

Westartfromthelibrarynamespace,andbymeansofthedot(.)operator,wewalkintothatnamespace.Withinthisnamespace,welookforsecond_floor,andagainwewalkintoitwiththe.operator.Wethenwalkintosection_x,andfinallywithinthelastnamespace,row_tree,wefindthenamewewerelookingfor:book.

Walkingthroughanamespacewillbeclearerwhenwe’llbedealingwithrealcodeexamples.Fornow,justkeepinmindthatnamespacesareplaceswherenamesareassociatedtoobjects.

Thereisanotherconcept,whichiscloselyrelatedtothatofanamespace,whichI’dliketobrieflytalkabout:thescope.

ScopesAccordingtoPython’sdocumentation,ascopeisatextualregionofaPythonprogram,whereanamespaceisdirectlyaccessible.Directlyaccessiblemeansthatwhenyou’relookingforanunqualifiedreferencetoaname,Pythontriestofinditinthenamespace.

Scopesaredeterminedstatically,butactuallyduringruntimetheyareuseddynamically.Thismeansthatbyinspectingthesourcecodeyoucantellwhatthescopeofanobjectis,butthisdoesn’tpreventthesoftwaretoalterthatduringruntime.TherearefourdifferentscopesthatPythonmakesaccessible(notnecessarilyallofthempresentatthesametime,ofcourse):

Thelocalscope,whichistheinnermostoneandcontainsthelocalnames.Theenclosingscope,thatis,thescopeofanyenclosingfunction.Itcontainsnon-localnamesandalsonon-globalnames.Theglobalscopecontainstheglobalnames.Thebuilt-inscopecontainsthebuilt-innames.Pythoncomeswithasetoffunctionsthatyoucanuseinaoff-the-shelffashion,suchasprint,all,abs,andsoon.Theyliveinthebuilt-inscope.

Theruleisthefollowing:whenwerefertoaname,Pythonstartslookingforitinthecurrentnamespace.Ifthenameisnotfound,Pythoncontinuesthesearchtotheenclosingscopeandthiscontinueuntilthebuilt-inscopeissearched.Ifanamehasn’tbeenfoundaftersearchingthebuilt-inscope,thenPythonraisesaNameErrorexception,whichbasicallymeansthatthenamehasn’tbeendefined(yousawthisintheprecedingexample).

Theorderinwhichthenamespacesarescannedwhenlookingforanameistherefore:local,enclosing,global,built-in(LEGB).

Thisisallverytheoretical,solet’sseeanexample.InordertoshowyouLocalandEnclosingnamespaces,Iwillhavetodefineafewfunctions.Don’tworryifyouarenotfamiliarwiththeirsyntaxforthemoment,we’llstudyfunctionsinChapter4,Functions,theBuildingBlocksofCode.Justrememberthatinthefollowingcode,whenyouseedef,itmeansI’mdefiningafunction.scopes1.py

#LocalversusGlobal

#wedefineafunction,calledlocal

deflocal():

m=7

print(m)

m=5

print(m)

#wecall,or`execute`thefunctionlocal

local()

Intheprecedingexample,wedefinethesamenamem,bothintheglobalscopeandinthelocalone(theonedefinedbythefunctionlocal).Whenweexecutethisprogramwiththefollowingcommand(haveyouactivatedyourvirtualenv?):

$pythonscopes1.py

Weseetwonumbersprintedontheconsole:5and7.

WhathappensisthatthePythoninterpreterparsesthefile,toptobottom.First,itfindsacoupleofcommentlines,whichareskipped,thenitparsesthedefinitionofthefunctionlocal.Whencalled,thisfunctiondoestwothings:itsetsupanametoanobjectrepresentingnumber7andprintsit.ThePythoninterpreterkeepsgoinganditfindsanothernamebinding.Thistimethebindinghappensintheglobalscopeandthevalueis5.Thenextlineisacalltotheprintfunction,whichisexecuted(andsowegetthefirstvalueprintedontheconsole:5).

Afterthis,thereisacalltothefunctionlocal.Atthispoint,Pythonexecutesthefunction,soatthistime,thebindingm=7happensandit’sprinted.

Oneveryimportantthingtonoticeisthatthepartofthecodethatbelongstothedefinitionofthefunctionlocalisindentedbyfourspacesontheright.Pythoninfactdefinesscopesbyindentingthecode.Youwalkintoascopebyindentingandwalkoutofitbyunindenting.Somecodersusetwospaces,othersthree,butthesuggestednumberofspacestouseisfour.It’sagoodmeasuretomaximizereadability.We’lltalkmoreaboutalltheconventionsyoushouldembracewhenwritingPythoncodelater.

Whatwouldhappenifweremovedthatm=7line?RemembertheLEGBrule.Pythonwouldstartlookingforminthelocalscope(functionlocal),and,notfindingit,itwouldgotothenextenclosingscope.Thenextoneinthiscaseistheglobalonebecausethereisnoenclosingfunctionwrappedaroundlocal.Therefore,wewouldseetwonumber5printedontheconsole.Let’sactuallyseehowthecodewouldlooklike:scopes2.py

#LocalversusGlobal

deflocal():

#mdoesn'tbelongtothescopedefinedbythelocalfunction

#soPythonwillkeeplookingintothenextenclosingscope.

#misfinallyfoundintheglobalscope

print(m,'printingfromthelocalscope')

m=5

print(m,'printingfromtheglobalscope')

local()

Runningscopes2.pywillprintthis:

(.lpvenv)fab@xps:ch1$pythonscopes2.py

5printingfromtheglobalscope

5printingfromthelocalscope

Asexpected,Pythonprintsmthefirsttime,thenwhenthefunctionlocaliscalled,misn’tfoundinitsscope,soPythonlooksforitfollowingtheLEGBchainuntilmisfoundintheglobalscope.

Let’sseeanexamplewithanextralayer,theenclosingscope:scopes3.py

#Local,EnclosingandGlobal

defenclosing_func():

m=13

deflocal():

#mdoesn'tbelongtothescopedefinedbythelocal

#functionsoPythonwillkeeplookingintothenext

#enclosingscope.Thistimemisfoundintheenclosing

#scope

print(m,'printingfromthelocalscope')

#callingthefunctionlocal

local()

m=5

print(m,'printingfromtheglobalscope')

enclosing_func()

Runningscopes3.pywillprintontheconsole:

(.lpvenv)fab@xps:ch1$pythonscopes3.py

5printingfromtheglobalscope

13printingfromthelocalscope

Asyoucansee,theprintinstructionfromthefunctionlocalisreferringtomasbefore.misstillnotdefinedwithinthefunctionitself,soPythonstartswalkingscopesfollowingtheLEGBorder.Thistimemisfoundintheenclosingscope.

Don’tworryifthisisstillnotperfectlyclearfornow.Itwillcometoyouaswegothroughtheexamplesinthebook.TheClassessectionofthePythontutorial(officialdocumentation)hasaninterestingparagraphaboutscopesandnamespaces.Makesureyoureaditatsomepointifyouwishforadeeperunderstandingofthesubject.

Beforewefinishoffthischapter,Iwouldliketotalkabitmoreaboutobjects.Afterall,basicallyeverythinginPythonisanobject,soIthinktheydeserveabitmoreattention.

TipDownloadingtheexamplecode

Youcandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.

ObjectandclassesWhenIintroducedobjectsintheAproperintroductionsection,Isaidthatweusethemtorepresentreal-lifeobjects.Forexample,wesellgoodsofanykindontheWebnowadaysandweneedtobeabletohandle,store,andrepresentthemproperly.Butobjectsareactuallysomuchmorethanthat.Mostofwhatyouwilleverdo,inPython,hastodowithmanipulatingobjects.

So,withoutgoingtoomuchintodetail(we’lldothatinChapter6,AdvancedConcepts–OOP,Decorators,andIterators),Iwanttogiveyoutheinanutshellkindofexplanationaboutclassesandobjects.

We’vealreadyseenthatobjectsarePython’sabstractionfordata.Infact,everythinginPythonisanobject.Numbers,strings(datastructuresthatholdtext),containers,collections,evenfunctions.Youcanthinkofthemasiftheywereboxeswithatleastthreefeatures:anID(unique),atype,andavalue.

Buthowdotheycometolife?Howdowecreatethem?Howtowewriteourowncustomobjects?Theanswerliesinonesimpleword:classes.

Objectsare,infact,instancesofclasses.ThebeautyofPythonisthatclassesareobjectsthemselves,butlet’snotgodownthisroad.Itleadstooneofthemostadvancedconceptsofthislanguage:metaclasses.We’lltalkverybrieflyabouttheminChapter6,AdvancedConcepts–OOP,Decorators,andIterators.Fornow,thebestwayforyoutogetthedifferencebetweenclassesandobjects,isbymeansofanexample.

Sayafriendtellsyou“Iboughtanewbike!”Youimmediatelyunderstandwhatshe’stalkingabout.Haveyouseenthebike?No.Doyouknowwhatcoloritis?Nope.Thebrand?Nope.Doyouknowanythingaboutit?Nope.Butatthesametime,youknoweverythingyouneedinordertounderstandwhatyourfriendmeantwhenshetoldyousheboughtanewbike.Youknowthatabikehastwowheelsattachedtoaframe,asaddle,pedals,handlebars,brakes,andsoon.Inotherwords,evenifyouhaven’tseenthebikeitself,youknowtheconceptofbike.Anabstractsetoffeaturesandcharacteristicsthattogetherformsomethingcalledbike.

Incomputerprogramming,thatiscalledaclass.It’sthatsimple.Classesareusedtocreateobjects.Infact,objectsaresaidtobeinstancesofclasses.

Inotherwords,weallknowwhatabikeis,weknowtheclass.ButthenIhavemyownbike,whichisaninstanceoftheclassbike.Andmybikeisanobjectwithitsowncharacteristicsandmethods.Youhaveyourownbike.Sameclass,butdifferentinstance.Everybikeevercreatedintheworldisaninstanceofthebikeclass.

Let’sseeanexample.Wewillwriteaclassthatdefinesabikeandthenwe’llcreatetwobikes,oneredandoneblue.I’llkeepthecodeverysimple,butdon’tfretifyoudon’tunderstandeverythingaboutit;allyouneedtocareaboutatthismomentistounderstandthedifferencebetweenclassandobject(orinstanceofaclass):bike.py

#let'sdefinetheclassBike

classBike:

def__init__(self,colour,frame_material):

self.colour=colour

self.frame_material=frame_material

defbrake(self):

print("Braking!")

#let'screateacoupleofinstances

red_bike=Bike('Red','Carbonfiber')

blue_bike=Bike('Blue','Steel')

#let'sinspecttheobjectswehave,instancesoftheBikeclass.

print(red_bike.colour)#prints:Red

print(red_bike.frame_material)#prints:Carbonfiber

print(blue_bike.colour)#prints:Blue

print(blue_bike.frame_material)#prints:Steel

#let'sbrake!

red_bike.brake()#prints:Braking!

TipIhopebynowIdon’tneedtotellyoutorunthefileeverytime,right?Thefilenameisindicatedinthefirstlineofthecodeblock.Justrun$pythonfilename,andyou’llbefine.

Somanyinterestingthingstonoticehere.Firstthingsfirst;thedefinitionofaclasshappenswiththeclassstatement(highlightedinthecode).Whatevercodecomesaftertheclassstatement,andisindented,iscalledthebodyoftheclass.Inourcase,thelastlinethatbelongstotheclassdefinitionistheprint("Braking!")one.

Afterhavingdefinedtheclasswe’rereadytocreateinstances.Youcanseethattheclassbodyhoststhedefinitionoftwomethods.Amethodisbasically(andsimplistically)afunctionthatbelongstoaclass.

Thefirstmethod,__init__isaninitializer.ItusessomePythonmagictosetuptheobjectswiththevalueswepasswhenwecreateit.

NoteEverymethodthathasleadingandtrailingdoubleunderscore,inPython,iscalledmagicmethod.MagicmethodsareusedbyPythonforamultitudeofdifferentpurposes,henceit’sneveragoodideatonameacustommethodusingtwoleadingandtrailingunderscores.ThisnamingconventionisbestlefttoPython.

Theothermethodwedefined,brake,isjustanexampleofanadditionalmethodthatwecouldcallifwewantedtobrakethebike.Itcontainsjustaprintstatement,ofcourse,it’sanexample.

Wecreatedtwobikesthen.Onehasredcolorandacarbonfiberframe,andtheotheronehasbluecolorandsteelframe.Wepassthosevaluesuponcreation.Aftercreation,weprintoutthecolorpropertyandframetypeoftheredbike,andtheframetypeoftheblue

onejustasanexample.Wealsocallthebrakemethodofthered_bike.

Onelastthingtonotice.YourememberItoldyouthatthesetofattributesofanobjectisconsideredtobeanamespace?Ihopeit’sclearernow,whatImeant.Youseethatbygettingtotheframe_typepropertythroughdifferentnamespaces(red_bike,blue_bike)weobtaindifferentvalues.Nooverlapping,noconfusion.

Thedot(.)operatorisofcoursethemeansweusetowalkintoanamespace,inthecaseofobjectsaswell.

GuidelinesonhowtowritegoodcodeWritinggoodcodeisnotaseasyasitseems.AsIalreadysaidbefore,goodcodeexposesalonglistofqualitiesthatisquitehardtoputtogether.Writinggoodcodeis,tosomeextent,anart.Regardlessofwhereonthepathyouwillbehappytosettle,thereissomethingthatyoucanembracewhichwillmakeyourcodeinstantlybetter:PEP8.

AccordingtoWikipedia:

“Python’sdevelopmentisconductedlargelythroughthePythonEnhancementProposal(PEP)process.ThePEPprocessistheprimarymechanismforproposingmajornewfeatures,forcollectingcommunityinputonanissue,andfordocumentingthedesigndecisionsthathavegoneintoPython.”

AmongallthePEPs,probablythemostfamousoneisPEP8.ItlaysoutasimplebuteffectivesetofguidelinestodefinePythonaestheticsothatwewritebeautifulPythoncode.Ifyoutakeonesuggestionoutofthischapter,pleaseletitbethis:useit.Embraceit.Youwillthankmelater.

Codingtodayisnolongeracheck-in/check-outbusiness.Rather,it’smoreofasocialeffort.Severaldeveloperscollaboratetoapieceofcodethroughtoolslikegitandmercurial,andtheresultiscodethatisfatheredbymanydifferenthands.

NoteGitandMercurialareprobablythemostuseddistributedrevisioncontrolsystemstoday.Theyareessentialtoolsdesignedtohelpteamsofdeveloperscollaborateonthesamesoftware.

Thesedays,morethanever,weneedtohaveaconsistentwayofwritingcode,sothatreadabilityismaximized.WhenalldevelopersofacompanyabidewithPEP8,it’snotuncommonforanyofthemlandingonapieceofcodetothinktheywroteitthemselves.Itactuallyhappenstomeallthetime(IalwaysforgetthecodeIwrite).

Thishasatremendousadvantage:whenyoureadcodethatyoucouldhavewrittenyourself,youreaditeasily.Withoutaconvention,everycoderwouldstructurethecodethewaytheylikemost,orsimplythewaytheyweretaughtorareusedto,andthiswouldmeanhavingtointerpreteverylineaccordingtosomeoneelse’sstyle.Itwouldmeanhavingtolosemuchmoretimejusttryingtounderstandit.ThankstoPEP8,wecanavoidthis.I’msuchafanofitthatIwon’tsignoffacodereviewifthecodedoesn’trespectit.Sopleasetakethetimetostudyit,it’sveryimportant.

Intheexamplesofthisbook,IwilltrytorespectitasmuchasIcan.Unfortunately,Idon’thavetheluxuryof79characters(whichisthemaximumlinelengthsuggestedbyPEP*),andIwillhavetocutdownonblanklinesandotherthings,butIpromiseyouI’lltrytolayoutmycodesothatit’sasreadableaspossible.

ThePythonculturePythonhasbeenadoptedwidelyinallcodingindustries.It’susedbymanydifferentcompaniesformanydifferentpurposes,andit’salsousedineducation(it’sanexcellentlanguageforthatpurpose,becauseofitsmanyqualitiesandthefactthatit’seasytolearn).

OneofthereasonsPythonissopopulartodayisthatthecommunityarounditisvast,vibrant,andfullofbrilliantpeople.Manyeventsareorganizedallovertheworld,mostlyeitheraroundPythonoritsmainwebframework,Django.

Pythonisopen,andveryoftensoarethemindsofthosewhoembraceit.CheckoutthecommunitypageonthePythonwebsiteformoreinformationandgetinvolved!

ThereisanotheraspecttoPythonwhichrevolvesaroundthenotionofbeingPythonic.IthastodowiththefactthatPythonallowsyoutousesomeidiomsthataren’tfoundelsewhere,atleastnotinthesameformoreasinessofuse(IfeelquiteclaustrophobicwhenIhavetocodeinalanguagewhichisnotPythonnow).

Anyway,overtheyears,thisconceptofbeingPythonichasemergedand,thewayIunderstandit,issomethingalongthelinesofdoingthingsthewaytheyaresupposedtobedoneinPython.

TohelpyouunderstandalittlebitmoreaboutPython’scultureandaboutbeingPythonic,IwillshowyoutheZenofPython.AlovelyEastereggthatisverypopular.OpenupaPythonconsoleandtypeimportthis.Whatfollowsistheresultofthisline:

>>>importthis

TheZenofPython,byTimPeters

Beautifulisbetterthanugly.

Explicitisbetterthanimplicit.

Simpleisbetterthancomplex.

Complexisbetterthancomplicated.

Flatisbetterthannested.

Sparseisbetterthandense.

Readabilitycounts.

Specialcasesaren'tspecialenoughtobreaktherules.

Althoughpracticalitybeatspurity.

Errorsshouldneverpasssilently.

Unlessexplicitlysilenced.

Inthefaceofambiguity,refusethetemptationtoguess.

Thereshouldbeone--andpreferablyonlyone--obviouswaytodoit.

Althoughthatwaymaynotbeobviousatfirstunlessyou'reDutch.

Nowisbetterthannever.

Althoughneverisoftenbetterthan*right*now.

Iftheimplementationishardtoexplain,it'sabadidea.

Iftheimplementationiseasytoexplain,itmaybeagoodidea.

Namespacesareonehonkinggreatidea—let'sdomoreofthose!

Therearetwolevelsofreadinghere.Oneistoconsideritasasetofguidelinesthathavebeenputdowninafunway.Theotheroneistokeepitinmind,andmaybereaditonceinawhile,tryingtounderstandhowitreferstosomethingdeeper.SomePython

characteristicsthatyouwillhavetounderstanddeeplyinordertowritePythonthewayit’ssupposedtobewritten.Startwiththefunlevel,andthendigdeeper.Alwaysdigdeeper.

AnoteontheIDEsJustafewwordsaboutIntegratedDevelopmentEnvironments(IDEs).Tofollowtheexamplesinthisbookyoudon’tneedone,anytexteditorwilldofine.Ifyouwanttohavemoreadvancedfeaturessuchassyntaxcoloringandautocompletion,youwillhavetofetchyourselfanIDE.YoucanfindacomprehensivelistofopensourceIDEs(justGoogle“pythonides”)onthePythonwebsite.IpersonallyuseSublimeTexteditor.It’sfreetotryoutanditcostsjustafewdollars.IhavetriedmanyIDEsinmylife,butthisistheonethatmakesmemostproductive.

Twoextremelyimportantpiecesofadvice:

WhateverIDEyouwillchosetouse,trytolearnitwellsothatyoucanexploititsstrengths,butdon’tdependonit.ExerciseyourselftoworkwithVIM(oranyothertexteditor)onceinawhile,learntobeabletodosomeworkonanyplatform,withanysetoftools.Whatevertexteditor/IDEyouwilluse,whenitcomestowritingPython,indentationisfourspaces.Don’tusetabs,don’tmixthemwithspaces.Usefourspaces,nottwo,notthree,notfive.Justusefour.Thewholeworldworkslikethat,andyoudon’twanttobecomeanoutcastbecauseyouwerefondofthethree-spacelayout.

SummaryInthischapter,westartedtoexploretheworldofprogrammingandthatofPython.We’vebarelyscratchedthesurface,justalittle,touchingconceptsthatwillbediscussedlateroninthebookingreaterdetail.

WetalkedaboutPython’smainfeatures,whoisusingitandforwhat,andwhatarethedifferentwaysinwhichwecanwriteaPythonprogram.

Inthelastpartofthechapter,weflewoverthefundamentalnotionsofnamespace,scope,class,andobject.WealsosawhowPythoncodecanbeorganizedusingmodulesandpackages.

Onapracticallevel,welearnedhowtoinstallPythononoursystem,howtomakesurewehavethetoolsweneed,pipandvirtualenv,andwealsocreatedandactivatedourfirstvirtualenvironment.Thiswillallowustoworkinaself-containedenvironmentwithouttheriskofcompromisingthePythonsysteminstallation.

Nowyou’rereadytostartthisjourneywithme.Allyouneedisenthusiasm,anactivatedvirtualenvironment,thisbook,yourfingers,andsomecoffee.

Trytofollowtheexamples,I’llkeepthemsimpleandshort.Ifyouputthemunderyourfingertips,youwillretainthemmuchbetterthanifyoujustreadthem.

Inthenextchapter,wewillexplorePython’srichsetofbuilt-indatatypes.There’smuchtocoverandmuchtolearn!

Chapter2.Built-inDataTypes “Data!Data!Data!”hecriedimpatiently.“Ican’tmakebrickswithoutclay.”

—SherlockHolmes-TheAdventureoftheCopperBeeches

Everythingyoudowithacomputerismanagingdata.Datacomesinmanydifferentshapesandflavors.It’sthemusicyoulisten,themovieyoustream,thePDFsyouopen.Eventhechapteryou’rereadingatthisverymomentisjustafile,whichisdata.

Datacanbesimple,anintegernumbertorepresentanage,orcomplex,likeanorderplacedonawebsite.Itcanbeaboutasingleobjectoraboutacollectionofthem.

Datacanevenbeaboutdata,thatis,metadata.Datathatdescribesthedesignofotherdatastructuresordatathatdescribesapplicationdataoritscontext.

InPython,objectsareabstractionfordata,andPythonhasanamazingvarietyofdatastructuresthatyoucanusetorepresentdata,orcombinethemtocreateyourowncustomdata.Beforewedelveintothespecifics,IwantyoutobeveryclearaboutobjectsinPython,solet’stalkalittlebitmoreaboutthem.

EverythingisanobjectAswealreadysaid,everythinginPythonisanobject.Butwhatreallyhappenswhenyoutypeaninstructionlikeage=42inaPythonmodule?

TipIfyougotohttp://pythontutor.com/,youcantypethatinstructionintoatextboxandgetitsvisualrepresentation.Keepthiswebsiteinmind,it’sveryusefultoconsolidateyourunderstandingofwhatgoesonbehindthescenes.

So,whathappensisthatanobjectiscreated.Itgetsanid,thetypeissettoint(integernumber),andthevalueto42.Anameageisplacedintheglobalnamespace,pointingtothatobject.Therefore,wheneverweareintheglobalnamespace,aftertheexecutionofthatline,wecanretrievethatobjectbysimplyaccessingitthroughitsname:age.

Ifyouweretomovehouse,youwouldputalltheknives,forks,andspoonsinaboxandlabelitcutlery.Canyouseeit’sexactlythesameconcept?Here’sascreenshotofhowitmaylooklike(youmayhavetotweakthesettingstogettothesameview):

So,fortherestofthischapter,wheneveryoureadsomethingsuchasname=some_value,thinkofanameplacedinthenamespacethatistiedtothescopeinwhichtheinstructionwaswritten,withanicearrowpointingtoanobjectthathasanid,atype,andavalue.Thereisalittlebitmoretosayaboutthismechanism,butit’smucheasiertotalkaboutitoveranexample,sowe’llgetbacktothislater.

Mutableorimmutable?ThatisthequestionAfirstfundamentaldistinctionthatPythonmakesondataisaboutwhetherornotthevalueofanobjectchanges.Ifthevaluecanchange,theobjectiscalledmutable,whileifthevaluecannotchange,theobjectiscalledimmutable.

Itisveryimportantthatyouunderstandthedistinctionbetweenmutableandimmutablebecauseitaffectsthecodeyouwrite,sohere’saquestion:

>>>age=42

>>>age

42

>>>age=43#A

>>>age

43

Intheprecedingcode,ontheline#A,haveIchangedthevalueofage?Well,no.Butnowit’s43(Ihearyousay…).Yes,it’s43,but42wasanintegernumber,ofthetypeint,whichisimmutable.So,whathappenedisreallythatonthefirstline,ageisanamethatissettopointtoanintobject,whosevalueis42.Whenwetypeage=43,whathappensisthatanotherobjectiscreated,ofthetypeintandvalue43(also,theidwillbedifferent),andthenameageissettopointtoit.So,wedidn’tchangethat42to43.Weactuallyjustpointedagetoadifferentlocation:thenewintobjectwhosevalueis43.Let’sseethesamecodealsoprintingtheIDs:

>>>age=42

>>>id(age)

10456352

>>>age=43

>>>id(age)

10456384

NoticethatweprinttheIDsbycallingthebuilt-inidfunction.Asyoucansee,theyaredifferent,asexpected.Bearinmindthatagepointstooneobjectatatime:42first,then43.Nevertogether.

Now,let’sseethesameexampleusingamutableobject.Forthisexample,let’sjustuseaPersonobject,thathasapropertyage:

>>>fab=Person(age=39)

>>>fab.age

39

>>>id(fab)

139632387887456

>>>fab.age=29#Iwish!

>>>id(fab)

139632387887456#stillthesameid

Inthiscase,IsetupanobjectfabwhosetypeisPerson(acustomclass).Oncreation,theobjectisgiventheageof39.I’mprintingit,alongwiththeobjectid,rightafterwards.

Noticethat,evenafterIchangeagetobe29,theIDoffabstaysthesame(whiletheIDofagehaschanged,ofcourse).CustomobjectsinPythonaremutable(unlessyoucodethemnottobe).Keepthisconceptinmind,it’sveryimportant.I’llremindyouaboutitthroughtherestofthechapter.

NumbersLet’sstartbyexploringPython’sbuilt-indatatypesfornumbers.Pythonwasdesignedbyamanwithamaster’sdegreeinmathematicsandcomputerscience,soit’sonlylogicalthatithasamazingsupportfornumbers.

Numbersareimmutableobjects.

IntegersPythonintegershaveunlimitedrange,subjectonlytotheavailablevirtualmemory.Thismeansthatitdoesn’treallymatterhowbiganumberyouwanttostore:aslongasitcanfitinyourcomputer’smemory,Pythonwilltakecareofit.Integernumberscanbepositive,negative,and0(zero).Theysupportallthebasicmathematicaloperations,asshowninthefollowingexample:

>>>a=12

>>>b=3

>>>a+b#addition

15

>>>b-a#subtraction

-9

>>>a//b#integerdivision

4

>>>a/b#truedivision

4.0

>>>a*b#multiplication

36

>>>b**a#poweroperator

531441

>>>2**1024#averybignumber,Pythonhandlesitgracefully

17976931348623159077293051907890247336179769789423065727343008115

77326758055009631327084773224075360211201138798713933576587897688

14416622492847430639474124377767893424865485276302219601246094119

45308295208500576883815068234246288147391311054082723716335051068

4586298239947245938479716304835356329624224137216

Theprecedingcodeshouldbeeasytounderstand.Justnoticeoneimportantthing:Pythonhastwodivisionoperators,oneperformstheso-calledtruedivision(/),whichreturnsthequotientoftheoperands,andtheotherone,theso-calledintegerdivision(//),whichreturnstheflooredquotientoftheoperands.Seehowthatisdifferentforpositiveandnegativenumbers:

>>>7/4#truedivision

1.75

>>>7//4#integerdivision,flooringreturns1

1

>>>-7/4#truedivisionagain,resultisoppositeofprevious

-1.75

>>>-7//4#integerdiv.,resultnottheoppositeofprevious

-2

Thisisaninterestingexample.Ifyouwereexpectinga-1onthelastline,don’tfeelbad,it’sjustthewayPythonworks.TheresultofanintegerdivisioninPythonisalwaysroundedtowardsminusinfinity.Ifinsteadofflooringyouwanttotruncateanumbertoaninteger,youcanusethebuilt-inintfunction,likeshowninthefollowingexample:

>>>int(1.75)

1

>>>int(-1.75)

-1

Noticethattruncationisdonetowards0.

Thereisalsoanoperatortocalculatetheremainderofadivision.It’scalledmodulooperator,andit’srepresentedbyapercent(%):

>>>10%3#remainderofthedivision10//3

1

>>>10%4#remainderofthedivision10//4

2

BooleansBooleanalgebraisthatsubsetofalgebrainwhichthevaluesofthevariablesarethetruthvalues:trueandfalse.InPython,TrueandFalsearetwokeywordsthatareusedtorepresenttruthvalues.Booleansareasubclassofintegers,andbehaverespectivelylike1and0.TheequivalentoftheintclassforBooleansistheboolclass,whichreturnseitherTrueorFalse.Everybuilt-inPythonobjecthasavalueintheBooleancontext,whichmeanstheybasicallyevaluatetoeitherTrueorFalsewhenfedtotheboolfunction.We’llseeallaboutthisinChapter3,IteratingandMakingDecisions.

BooleanvaluescanbecombinedinBooleanexpressionsusingthelogicaloperatorsand,or,andnot.Again,we’llseetheminfullinthenextchapter,sofornowlet’sjustseeasimpleexample:

>>>int(True)#Truebehaveslike1

1

>>>int(False)#Falsebehaveslike0

0

>>>bool(1)#1evaluatestoTrueinabooleancontext

True

>>>bool(-42)#andsodoeseverynon-zeronumber

True

>>>bool(0)#0evaluatestoFalse

False

>>>#quickpeakattheoperators(and,or,not)

>>>notTrue

False

>>>notFalse

True

>>>TrueandTrue

True

>>>FalseorTrue

True

YoucanseethatTrueandFalsearesubclassesofintegerswhenyoutrytoaddthem.Pythonupcaststhemtointegersandperformsaddition:

>>>1+True

2

>>>False+42

42

>>>7-True

6

NoteUpcastingisatypeconversionoperationthatgoesfromasubclasstoitsparent.Intheexamplepresentedhere,TrueandFalse,whichbelongtoaclassderivedfromtheintegerclass,areconvertedbacktointegerswhenneeded.ThistopicisaboutinheritanceandwillbeexplainedindetailinChapter6,AdvancedConcepts–OOP,Decorators,andIterators.

RealsRealnumbers,orfloatingpointnumbers,arerepresentedinPythonaccordingtotheIEEE754double-precisionbinaryfloating-pointformat,whichisstoredin64bitsofinformationdividedintothreesections:sign,exponent,andmantissa.

NoteQuenchyourthirstforknowledgeaboutthisformatonWikipedia:http://en.wikipedia.org/wiki/Double-precision_floating-point_format

Usuallyprogramminglanguagesgivecoderstwodifferentformats:singleanddoubleprecision.Theformertakingup32bitsofmemory,andthelatter64.Pythonsupportsonlythedoubleformat.Let’sseeasimpleexample:

>>>pi=3.1415926536#howmanydigitsofPIcanyouremember?

>>>radius=4.5

>>>area=pi*(radius**2)

>>>area

63.61725123519331

NoteInthecalculationofthearea,Iwrappedtheradius**2withinbraces.Eventhoughthatwasn’tnecessarybecausethepoweroperatorhashigherprecedencethanthemultiplicationone,Ithinktheformulareadsmoreeasilylikethat.

Thesys.float_infostructsequenceholdsinformationabouthowfloatingpointnumberswillbehaveonyoursystem.ThisiswhatIseeonmybox:

>>>importsys

>>>sys.float_info

sys.float_info(max=1.7976931348623157e+308,max_exp=1024,max_10_exp=308,

min=2.2250738585072014e-308,min_exp=-1021,min_10_exp=-307,dig=15,

mant_dig=53,epsilon=2.220446049250313e-16,radix=2,rounds=1)

Let’smakeafewconsiderationshere:wehave64bitstorepresentfloatnumbers.Thismeanswecanrepresentatmost2**64==18,446,744,073,709,551,616numberswiththatamountofbits.Takealookatthemaxandepsilonvalueforthefloatnumbers,andyou’llrealizeit’simpossibletorepresentthemall.Thereisjustnotenoughspacesotheyareapproximatedtotheclosestrepresentablenumber.Youprobablythinkthatonlyextremelybigorextremelysmallnumberssufferfromthisissue.Well,thinkagain:

>>>3*0.1–0.3#thisshouldbe0!!!

5.551115123125783e-17

Whatdoesthistellyou?Ittellsyouthatdoubleprecisionnumberssufferfromapproximationissuesevenwhenitcomestosimplenumberslike0.1or0.3.Whyisthisimportant?Itcanbeabigproblemifyou’rehandlingprices,orfinancialcalculations,oranykindofdatathatneedsnottobeapproximated.Don’tworry,PythongivesyoutheDecimaltype,whichdoesn’tsufferfromtheseissues,we’llseetheminabit.

ComplexnumbersPythongivesyoucomplexnumberssupportoutofthebox.Ifyoudon’tknowwhatcomplexnumbersare,youcanlookthemupontheWeb.Theyarenumbersthatcanbeexpressedintheforma+ibwhereaandbarerealnumbers,andi(orjifyou’reanengineer)istheimaginaryunit,thatis,thesquarerootof-1.aandbarecalledrespectivelytherealandimaginarypartofthenumber.

It’sactuallyunlikelyyou’llbeusingthem,unlessyou’recodingsomethingscientific.Let’sseeasmallexample:

>>>c=3.14+2.73j

>>>c.real#realpart

3.14

>>>c.imag#imaginarypart

2.73

>>>c.conjugate()#conjugateofA+BjisA-Bj

(3.14-2.73j)

>>>c*2#multiplicationisallowed

(6.28+5.46j)

>>>c**2#poweroperationaswell

(2.4067000000000007+17.1444j)

>>>d=1+1j#additionandsubtractionaswell

>>>c-d

(2.14+1.73j)

FractionsanddecimalsLet’sfinishthetourofthenumberdepartmentwithalookatfractionsanddecimals.Fractionsholdarationalnumeratoranddenominatorintheirlowestforms.Let’sseeaquickexample:

>>>fromfractionsimportFraction

>>>Fraction(10,6)#madhatter?

Fraction(5,3)#noticeit'sbeenreducedtolowestterms

>>>Fraction(1,3)+Fraction(2,3)#1/3+2/3=3/3=1/1

Fraction(1,1)

>>>f=Fraction(10,6)

>>>f.numerator

5

>>>f.denominator

3

Althoughtheycanbeveryusefulattimes,it’snotthatcommontospotthemincommercialsoftware.Mucheasierinstead,istoseedecimalnumbersbeingusedinallthosecontextswhereprecisioniseverything,forexample,scientificandfinancialcalculations.

NoteIt’simportanttorememberthatarbitraryprecisiondecimalnumberscomeatapriceinperformance,ofcourse.Theamountofdatatobestoredforeachnumberisfargreaterthanitisforfractionsorfloatsaswellasthewaytheyarehandled,whichrequiresthePythoninterpretermuchmoreworkbehindthescenes.Anotherinterestingthingtoknowisthatyoucangetandsettheprecisionbyaccessingdecimal.getcontext().prec.

Let’sseeaquickexamplewithDecimalnumbers:

>>>fromdecimalimportDecimalasD#renameforbrevity

>>>D(3.14)#pi,fromfloat,soapproximationissues

Decimal('3.140000000000000124344978758017532527446746826171875')

>>>D('3.14')#pi,fromastring,sonoapproximationissues

Decimal('3.14')

>>>D(0.1)*D(3)-D(0.3)#fromfloat,westillhavetheissue

Decimal('2.775557561565156540423631668E-17')

>>>D('0.1')*D(3)-D('0.3')#fromstring,allperfect

Decimal('0.0')

NoticethatwhenweconstructaDecimalnumberfromafloat,ittakesonalltheapproximationissuesthefloatmaycomefrom.Ontheotherhand,whentheDecimalhasnoapproximationissues,forexample,whenwefeedanintorastringrepresentationtotheconstructor,thenthecalculationhasnoquirkybehavior.Whenitcomestomoney,usedecimals.

Thisconcludesourintroductiontobuilt-innumerictypes,let’snowseesequences.

ImmutablesequencesLet’sstartwithimmutablesequences:strings,tuples,andbytes.

StringsandbytesTextualdatainPythonishandledwithstrobjects,morecommonlyknownasstrings.Theyareimmutablesequencesofunicodecodepoints.Unicodecodepointscanrepresentacharacter,butcanalsohaveothermeanings,suchasformattingdataforexample.Python,unlikeotherlanguages,doesn’thaveachartype,soasinglecharacterisrenderedsimplybyastringoflength1.Unicodeisanexcellentwaytohandledata,andshouldbeusedfortheinternalsofanyapplication.Whenitcomestostoretextualdatathough,orsenditonthenetwork,youmaywanttoencodeit,usinganappropriateencodingforthemediumyou’reusing.StringliteralsarewritteninPythonusingsingle,doubleortriplequotes(bothsingleordouble).Ifbuiltwithtriplequotes,astringcanspanonmultiplelines.Anexamplewillclarifythepicture:

>>>#4waystomakeastring

>>>str1='Thisisastring.Webuiltitwithsinglequotes.'

>>>str2="Thisisalsoastring,butbuiltwithdoublequotes."

>>>str3='''Thisisbuiltusingtriplequotes,

...soitcanspanmultiplelines.'''

>>>str4="""Thistoo

...isamultilineone

...builtwithtripledouble-quotes."""

>>>str4#A

'Thistoo\nisamultilineone\nbuiltwithtripledouble-quotes.'

>>>print(str4)#B

Thistoo

isamultilineone

builtwithtripledouble-quotes.

In#Aand#B,weprintstr4,firstimplicitly,thenexplicitlyusingtheprintfunction.Aniceexercisewouldbetofindoutwhytheyaredifferent.Areyouuptothechallenge?(hint,lookupthestrfunction)

Strings,likeanysequence,havealength.Youcangetthisbycallingthelenfunction:

>>>len(str1)

49

EncodinganddecodingstringsUsingtheencode/decodemethods,wecanencodeunicodestringsanddecodebytesobjects.Utf-8isavariablelengthcharacterencoding,capableofencodingallpossibleunicodecodepoints.ItisthedominantencodingfortheWeb(andnotonly).Noticealsothatbyaddingaliteralbinfrontofastringdeclaration,we’recreatingabytesobject.

>>>s="Thisisüŋíc0de"#unicodestring:codepoints

>>>type(s)

<class'str'>

>>>encoded_s=s.encode('utf-8')#utf-8encodedversionofs

>>>encoded_s

b'Thisis\xc3\xbc\xc5\x8b\xc3\xadc0de'#result:bytesobject

>>>type(encoded_s)#anotherwaytoverifyit

<class'bytes'>

>>>encoded_s.decode('utf-8')#let'sreverttotheoriginal

'Thisisüŋíc0de'

>>>bytes_obj=b"Abytesobject"#abytesobject

>>>type(bytes_obj)

<class'bytes'>

IndexingandslicingstringsWhenmanipulatingsequences,it’sverycommontohavetoaccessthematonepreciseposition(indexing),ortogetasubsequenceoutofthem(slicing).Whendealingwithimmutablesequences,bothoperationsareread-only.

Whileindexingcomesinoneform,azero-basedaccesstoanypositionwithinthesequence,slicingcomesindifferentforms.Whenyougetasliceofasequence,youcanspecifythestartandstoppositions,andthestep.Theyareseparatedwithacolon(:)likethis:my_sequence[start:stop:step].Alltheargumentsareoptional,startisinclusive,stopisexclusive.It’smucheasiertoshowanexample,ratherthanexplainthemfurtherinwords:

>>>s="Thetroubleisyouthinkyouhavetime."

>>>s[0]#indexingatposition0,whichisthefirstchar

'T'

>>>s[5]#indexingatposition5,whichisthesixthchar

'r'

>>>s[:4]#slicing,wespecifyonlythestopposition

'The'

>>>s[4:]#slicing,wespecifyonlythestartposition

'troubleisyouthinkyouhavetime.'

>>>s[2:14]#slicing,bothstartandstoppositions

'etroubleis'

>>>s[2:14:3]#slicing,start,stopandstep(every3chars)

'erb'

>>>s[:]#quickwayofmakingacopy

'Thetroubleisyouthinkyouhavetime.'

Ofallthelines,thelastoneisprobablythemostinteresting.Ifyoudon’tspecifyaparameter,Pythonwillfillinthedefaultforyou.Inthiscase,startwillbethestartofthestring,stopwillbetheendofthesting,andstepwillbethedefault1.Thisisaneasyandquickwayofobtainingacopyofthestrings(samevalue,butdifferentobject).Canyoufindawaytogetthereversedcopyofastringusingslicing?(don’tlookitup,finditforyourself)

TuplesThelastimmutablesequencetypewe’regoingtoseeisthetuple.AtupleisasequenceofarbitraryPythonobjects.Inatuple,itemsareseparatedbycommas.TheyareusedeverywhereinPython,becausetheyallowforpatternsthatarehardtoreproduceinotherlanguages.Sometimestuplesareusedimplicitly,forexampletosetupmultiplevariablesononeline,ortoallowafunctiontoreturnmultipledifferentobjects(usuallyafunctionreturnsoneobjectonly,inmanyotherlanguages),andeveninthePythonconsole,youcanusetuplesimplicitlytoprintmultipleelementswithonesingleinstruction.We’llseeexamplesforallthesecases:

>>>t=()#emptytuple

>>>type(t)

<class'tuple'>

>>>one_element_tuple=(42,)#youneedthecomma!

>>>three_elements_tuple=(1,3,5)

>>>a,b,c=1,2,3#tupleformultipleassignment

>>>a,b,c#implicittupletoprintwithoneinstruction

(1,2,3)

>>>3inthree_elements_tuple#membershiptest

True

Noticethatthemembershipoperatorincanalsobeusedwithlists,strings,dictionaries,andingeneralwithcollectionandsequenceobjects.

NoteNoticethattocreateatuplewithoneitem,weneedtoputthatcommaaftertheitem.Thereasonisthatwithoutthecommathatitemisjustitselfwrappedinbraces,kindofinaredundantmathematicalexpression.Noticealsothatonassignment,bracesareoptionalsomy_tuple=1,2,3isthesameasmy_tuple=(1,2,3).

Onethingthattupleassignmentallowsustodo,isone-lineswaps,withnoneedforathirdtemporaryvariable.Let’sseefirstamoretraditionalwayofdoingit:

>>>a,b=1,2

>>>c=a#weneedthreelinesandatemporaryvarc

>>>a=b

>>>b=c

>>>a,b#aandbhavebeenswapped

(2,1)

Andnowlet’sseehowwewoulddoitinPython:

>>>a,b=b,a#thisisthePythonicwaytodoit

>>>a,b

(1,2)

TakealookatthelinethatshowsyouthePythonicwayofswappingtwovalues:doyourememberwhatIwroteinChapter1,IntroductionandFirstSteps–TakeaDeepBreath.APythonprogramistypicallyone-fifthtoone-thirdthesizeofequivalentJavaorC++code,andfeatureslikeone-lineswapscontributetothis.Pythoniselegant,whereelegance

inthiscontextmeansalsoeconomy.

Becausetheyareimmutable,tuplescanbeusedaskeysfordictionaries(we’llseethisshortly).Thedictobjectsneedkeystobeimmutablebecauseiftheycouldchange,thenthevaluetheyreferencewouldn’tbefoundanymore(becausethepathtoitdependsonthekey).Ifyouareintodatastructures,youknowhowniceafeaturethisoneistohave.Tome,tuplesarePython’sbuilt-indatathatmostcloselyrepresentamathematicalvector.Thisdoesn’tmeanthatthiswasthereasonforwhichtheywerecreatedthough.Tuplesusuallycontainanheterogeneoussequenceofelements,whileontheotherhandlistsaremostofthetimeshomogeneous.Moreover,tuplesarenormallyaccessedviaunpackingorindexing,whilelistsareusuallyiteratedover.

MutablesequencesMutablesequencesdifferfromtheirimmutablesistersinthattheycanbechangedaftercreation.TherearetwomutablesequencetypesinPython:listsandbytearrays.IsaidbeforethatthedictionaryisthekingofdatastructuresinPython.Iguessthismakesthelistitsrightfulqueen.

ListsPythonlistsaremutablesequences.Theyareverysimilartotuples,buttheydon’thavetherestrictionsduetoimmutability.Listsarecommonlyusedtostorecollectionsofhomogeneousobjects,butthereisnothingpreventingyoutostoreheterogeneouscollectionsaswell.Listscanbecreatedinmanydifferentways,let’sseeanexample:

>>>[]#emptylist

[]

>>>list()#sameas[]

[]

>>>[1,2,3]#aswithtuples,itemsarecommaseparated

[1,2,3]

>>>[x+5forxin[2,3,4]]#Pythonismagic

[7,8,9]

>>>list((1,3,5,7,9))#listfromatuple

[1,3,5,7,9]

>>>list('hello')#listfromastring

['h','e','l','l','o']

Inthepreviousexample,Ishowedyouhowtocreatealistusingdifferenttechniques.IwouldlikeyoutotakeagoodlookatthelinethatsaysPythonismagic,whichIamnotexpectingyoutofullyunderstandatthispoint(unlessyoucheatedandyou’renotanovice!).Thatiscalledalistcomprehension,averypowerfulfunctionalfeatureofPython,whichwe’llseeindetailinChapter5,SavingTimeandMemory.Ijustwantedtomakeyourmouthwateratthispoint.

Creatinglistsisgood,buttherealfuncomeswhenweusethem,solet’sseethemainmethodstheygiftuswith:

>>>a=[1,2,1,3]

>>>a.append(13)#wecanappendanythingattheend

>>>a

[1,2,1,3,13]

>>>a.count(1)#howmany`1`arethereinthelist?

2

>>>a.extend([5,7])#extendthelistbyanother(orsequence)

>>>a

[1,2,1,3,13,5,7]

>>>a.index(13)#positionof`13`inthelist(0-basedindexing)

4

>>>a.insert(0,17)#insert`17`atposition0

>>>a

[17,1,2,1,3,13,5,7]

>>>a.pop()#pop(removeandreturn)lastelement

7

>>>a.pop(3)#popelementatposition3

1

>>>a

[17,1,2,3,13,5]

>>>a.remove(17)#remove`17`fromthelist

>>>a

[1,2,3,13,5]

>>>a.reverse()#reversetheorderoftheelementsinthelist

>>>a

[5,13,3,2,1]

>>>a.sort()#sortthelist

>>>a

[1,2,3,5,13]

>>>a.clear()#removeallelementsfromthelist

>>>a

[]

Theprecedingcodegivesyouaroundupoflist’smainmethods.Iwanttoshowyouhowpowerfultheyare,usingextendasanexample.Youcanextendlistsusinganysequencetype:

>>>a=list('hello')#makesalistfromastring

>>>a

['h','e','l','l','o']

>>>a.append(100)#append100,heterogeneoustype

>>>a

['h','e','l','l','o',100]

>>>a.extend((1,2,3))#extendusingtuple

>>>a

['h','e','l','l','o',100,1,2,3]

>>>a.extend('...')#extendusingstring

>>>a

['h','e','l','l','o',100,1,2,3,'.','.','.']

Now,let’sseewhatarethemostcommonoperationsyoucandowithlists:

>>>a=[1,3,5,7]

>>>min(a)#minimumvalueinthelist

1

>>>max(a)#maximumvalueinthelist

7

>>>sum(a)#sumofallvaluesinthelist

16

>>>len(a)#numberofelementsinthelist

4

>>>b=[6,7,8]

>>>a+b#`+`withlistmeansconcatenation

[1,3,5,7,6,7,8]

>>>a*2#`*`hasalsoaspecialmeaning

[1,3,5,7,1,3,5,7]

Thelasttwolinesintheprecedingcodearequiteinterestingbecausetheyintroduceustoaconceptcalledoperatoroverloading.Inshort,itmeansthatoperatorssuchas+,-.*,%,andsoon,mayrepresentdifferentoperationsaccordingtothecontexttheyareusedin.Itdoesn’tmakeanysensetosumtwolists,right?Therefore,the+signisusedtoconcatenatethem.Hence,the*signisusedtoconcatenatethelisttoitselfaccordingtotherightoperand.Now,let’stakeastepfurtherdowntherabbitholeandseesomethingalittlemoreinteresting.IwanttoshowyouhowpowerfulthesortmethodcanbeandhoweasyitisinPythontoachieveresultsthatrequireagreatdealofeffortinotherlanguages:

>>>fromoperatorimportitemgetter

>>>a=[(5,3),(1,3),(1,2),(2,-1),(4,9)]

>>>sorted(a)

[(1,2),(1,3),(2,-1),(4,9),(5,3)]

>>>sorted(a,key=itemgetter(0))

[(1,3),(1,2),(2,-1),(4,9),(5,3)]

>>>sorted(a,key=itemgetter(0,1))

[(1,2),(1,3),(2,-1),(4,9),(5,3)]

>>>sorted(a,key=itemgetter(1))

[(2,-1),(1,2),(5,3),(1,3),(4,9)]

>>>sorted(a,key=itemgetter(1),reverse=True)

[(4,9),(5,3),(1,3),(1,2),(2,-1)]

Theprecedingcodedeservesalittleexplanation.Firstofall,aisalistoftuples.Thismeanseachelementinaisatuple(a2-tuple,tobepicky).Whenwecallsorted(some_list),wegetasortedversionofsome_list.Inthiscase,thesortingona2-tupleworksbysortingthemonthefirstiteminthetuple,andonthesecondwhenthefirstoneisthesame.Youcanseethisbehaviorintheresultofsorted(a),whichyields[(1,2),(1,3),...].Pythonalsogivesustheabilitytocontrolonwhichelement(s)ofthetuplethesortingmustberunagainst.Noticethatwhenweinstructthesortedfunctiontoworkonthefirstelementofeachtuple(bykey=itemgetter(0)),theresultisdifferent:[(1,3),(1,2),...].Thesortingisdoneonlyonthefirstelementofeachtuple(whichistheoneatposition0).Ifwewanttoreplicatethedefaultbehaviorofasimplesorted(a)call,weneedtousekey=itemgetter(0,1),whichtellsPythontosortfirstontheelementsatposition0withinthetuples,andthenonthoseatposition1.Comparetheresultsandyou’llseetheymatch.

Forcompleteness,Iincludedanexampleofsortingonlyontheelementsatposition1,andthesamebutinreverseorder.IfyouhaveeverseensortinginJava,Iexpectyoutobeonyourkneescryingwithjoyatthisverymoment.

ThePythonsortingalgorithmisverypowerful,anditwaswrittenbyTimPeters(we’vealreadyseenthisname,canyourecallwhen?).ItisaptlynamedTimsort,anditisablendbetweenmergeandinsertionsortandhasbettertimeperformancesthanmostotheralgorithmsusedformainstreamprogramminglanguages.Timsortisastablesortingalgorithm,whichmeansthatwhenmultiplerecordshavethesamekey,theiroriginalorderispreserved.We’veseenthisintheresultofsorted(a,key=itemgetter(0))whichhasyielded[(1,3),(1,2),...]inwhichtheorderofthosetwotupleshasbeenpreservedbecausetheyhavethesamevalueatposition0.

BytearraysToconcludeouroverviewofmutablesequencetypes,let’sspendacoupleofminutesonthebytearraytype.Basically,theyrepresentthemutableversionofbytesobjects.Theyexposemostoftheusualmethodsofmutablesequencesaswellasmostofthemethodsofthebytestype.Itemsareintegersintherange[0,256).

NoteWhenitcomestointervals,I’mgoingtousethestandardnotationforopen/closedranges.Asquarebracketononeendmeansthatthevalueisincluded,whilearoundbracemeansit’sexcluded.Thegranularityisusuallyinferredbythetypeoftheedgeelementsso,forexample,theinterval[3,7]meansallintegersbetween3and7,inclusive.Ontheotherhand,(3,7)meansallintegersbetween3and7exclusive(hence4,5,and6).Itemsinabytearraytypeareintegersbetween0and256,0isincluded,256isnot.Onereasonintervalsareoftenexpressedlikethisistoeasecoding.Ifwebreakarange[a,b)intoNconsecutiveranges,wecaneasilyrepresenttheoriginaloneasaconcatenationlikethis:

Themiddlepoints(ki)beingexcludedononeend,andincludedontheotherend,allowforeasyconcatenationandsplittingwhenintervalsarehandledinthecode.

Let’sseeaquickexamplewiththetypebytearray:

>>>bytearray()#emptybytearrayobject

bytearray(b'')

>>>bytearray(10)#zero-filledinstancewithgivenlength

bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')

>>>bytearray(range(5))#bytearrayfromiterableofintegers

bytearray(b'\x00\x01\x02\x03\x04')

>>>name=bytearray(b'Lina')#A-bytearrayfrombytes

>>>name.replace(b'L',b'l')

bytearray(b'lina')

>>>name.endswith(b'na')

True

>>>name.upper()

bytearray(b'LINA')

>>>name.count(b'L')

1

Asyoucanseeintheprecedingcode,thereareafewwaystocreateabytearrayobject.Theycanbeusefulinmanysituations,forexample,whenreceivingdatathroughasocket,theyeliminatetheneedtoconcatenatedatawhilepolling,hencetheyproveveryhandy.Ontheline#A,Icreatedthenamebytearrayfromthestringb'Lina'toshowyouhowthebytearrayobjectexposesmethodsfrombothsequencesandstrings,whichisextremelyhandy.Ifyouthinkaboutit,theycanbeconsideredasmutablestrings.

SettypesPythonalsoprovidestwosettypes,setandfrozenset.Thesettypeismutable,whilefrozensetisimmutable.Theyareunorderedcollectionsofimmutableobjects.

Hashabilityisacharacteristicthatallowsanobjecttobeusedasasetmemberaswellasakeyforadictionary,aswe’llseeverysoon.

NoteAnobjectishashableifithasahashvaluewhichneverchangesduringitslifetime.

Objectsthatcompareequallymusthavethesamehashvalue.Setsareverycommonlyusedtotestformembership,solet’sintroducetheinoperatorinthefollowingexample:

>>>small_primes=set()#emptyset

>>>small_primes.add(2)#addingoneelementatatime

>>>small_primes.add(3)

>>>small_primes.add(5)

>>>small_primes

{2,3,5}

>>>small_primes.add(1)#LookwhatI'vedone,1isnotaprime!

>>>small_primes

{1,2,3,5}

>>>small_primes.remove(1)#solet'sremoveit

>>>3insmall_primes#membershiptest

True

>>>4insmall_primes

False

>>>4notinsmall_primes#negatedmembershiptest

True

>>>small_primes.add(3)#tryingtoadd3again

>>>small_primes

{2,3,5}#nochange,duplicationisnotallowed

>>>bigger_primes=set([5,7,11,13])#fastercreation

>>>small_primes|bigger_primes#unionoperator`|`

{2,3,5,7,11,13}

>>>small_primes&bigger_primes#intersectionoperator`&`

{5}

>>>small_primes-bigger_primes#differenceoperator`-`

{2,3}

Intheprecedingcode,youcanseetwodifferentwaystocreateaset.Onecreatesanemptysetandthenaddselementsoneatatime.Theothercreatesthesetusingalistofnumbersasargumenttotheconstructor,whichdoesalltheworkforus.Ofcourse,youcancreateasetfromalistortuple(oranyiterable)andthenyoucanaddandremovemembersfromthesetasyouplease.

Anotherwayofcreatingasetisbysimplyusingthecurlybracesnotation,likethis:

>>>small_primes={2,3,5,5,3}

>>>small_primes

{2,3,5}

NoticeIaddedsomeduplicationtoemphasizethattheresultsetwon’thaveany.

NoteWe’llseeiterableobjectsanditerationinthenextchapter.Fornow,justknowthatiterableobjectsareobjectsyoucaniterateoninadirection.

Let’sseeanexampleabouttheimmutablecounterpartofthesettype:frozenset.

>>>small_primes=frozenset([2,3,5,7])

>>>bigger_primes=frozenset([5,7,11])

>>>small_primes.add(11)#wecannotaddtoafrozenset

Traceback(mostrecentcalllast):

File"<stdin>",line1,in<module>

AttributeError:'frozenset'objecthasnoattribute'add'

>>>small_primes.remove(2)#neitherwecanremove

Traceback(mostrecentcalllast):

File"<stdin>",line1,in<module>

AttributeError:'frozenset'objecthasnoattribute'remove'

>>>small_primes&bigger_primes#intersect,union,etc.allowed

frozenset({5,7})

Asyoucansee,frozensetobjectsarequitelimitedinrespectoftheirmutablecounterpart.Theystillproveveryeffectiveformembershiptest,union,intersectionanddifferenceoperations,andforperformancereasons.

Mappingtypes–dictionariesOfallthebuilt-inPythondatatypes,thedictionaryisprobablythemostinterestingone.It’stheonlystandardmappingtype,anditisthebackboneofeveryPythonobject.

Adictionarymapskeystovalues.Keysneedtobehashableobjects,whilevaluescanbeofanyarbitrarytype.Dictionariesaremutableobjects.

Therearequiteafewdifferentwaystocreateadictionary,soletmegiveyouasimpleexampleofhowtocreateadictionaryequalto{'A':1,'Z':-1}infivedifferentways:

>>>a=dict(A=1,Z=-1)

>>>b={'A':1,'Z':-1}

>>>c=dict(zip(['A','Z'],[1,-1]))

>>>d=dict([('A',1),('Z',-1)])

>>>e=dict({'Z':-1,'A':1})

>>>a==b==c==d==e#aretheyallthesame?

True#indeed!

Haveyounoticedthosedoubleequals?Assignmentisdonewithoneequal,whiletocheckwhetheranobjectisthesameasanotherone(or5inonego,inthiscase),weusedoubleequals.Thereisalsoanotherwaytocompareobjects,whichinvolvestheisoperator,andcheckswhetherthetwoobjectsarethesame(iftheyhavethesameID,notjustthevalue),butunlessyouhaveagoodreasontouseit,youshouldusethedoubleequalinstead.Intheprecedingcode,Ialsousedonenicefunction:zip.Itisnamedafterthereal-lifezip,whichgluestogethertwothingstakingoneelementfromeachatatime.Letmeshowyouanexample:

>>>list(zip(['h','e','l','l','o'],[1,2,3,4,5]))

[('h',1),('e',2),('l',3),('l',4),('o',5)]

>>>list(zip('hello',range(1,6)))#equivalent,morePythonic

[('h',1),('e',2),('l',3),('l',4),('o',5)]

Intheprecedingexample,Ihavecreatedthesamelistintwodifferentways,onemoreexplicit,andtheotheralittlebitmorePythonic.ForgetforamomentthatIhadtowrapthelistconstructoraroundthezipcall(thereasonisbecausezipreturnsaniterator,notalist),andconcentrateontheresult.Seehowziphascoupledthefirstelementsofitstwoargumentstogether,thenthesecondones,thenthethirdones,andsoonandsoforth?Takealookatyourpants(oratyourpurseifyou’realady)andyou’llseethesamebehaviorinyouractualzip.Butlet’sgobacktodictionariesandseehowmanywonderfulmethodstheyexposeforallowingustomanipulatethemaswewant.Let’sstartwiththebasicoperations:

>>>d={}

>>>d['a']=1#let'ssetacoupleof(key,value)pairs

>>>d['b']=2

>>>len(d)#howmanypairs?

2

>>>d['a']#whatisthevalueof'a'?

1

>>>d#howdoes`d`looknow?

{'a':1,'b':2}

>>>deld['a']#let'sremove`a`

>>>d

{'b':2}

>>>d['c']=3#let'sadd'c':3

>>>'c'ind#membershipischeckedagainstthekeys

True

>>>3ind#notthevalues

False

>>>'e'ind

False

>>>d.clear()#let'scleaneverythingfromthisdictionary

>>>d

{}

Noticehowaccessingkeysofadictionary,regardlessofthetypeofoperationwe’reperforming,isdonethroughsquarebrackets.Doyourememberstrings,list,andtuples?Wewereaccessingelementsatsomepositionthroughsquarebracketsaswell.YetanotherexampleofPython’sconsistency.

Let’sseenowthreespecialobjectscalleddictionaryviews:keys,values,anditems.Theseobjectsprovideadynamicviewofthedictionaryentriesandtheychangewhenthedictionarychanges.keys()returnsallthekeysinthedictionary,values()returnsallthevaluesinthedictionary,anditems()returnsallthe(key,value)pairsinthedictionary.

NoteIt’sveryimportanttoknowthat,evenifadictionaryisnotintrinsicallyordered,accordingtothePythondocumentation:“Keysandvaluesareiteratedoverinanarbitraryorderwhichisnon-random,variesacrossPythonimplementations,anddependsonthedictionary’shistoryofinsertionsanddeletions.Ifkeys,valuesanditemsviewsareiteratedoverwithnointerveningmodificationstothedictionary,theorderofitemswilldirectlycorrespond.”

Enoughwiththischatter,let’sputallthisdownintocode:

>>>d=dict(zip('hello',range(5)))

>>>d

{'e':1,'h':0,'o':4,'l':3}

>>>d.keys()

dict_keys(['e','h','o','l'])

>>>d.values()

dict_values([1,0,4,3])

>>>d.items()

dict_items([('e',1),('h',0),('o',4),('l',3)])

>>>3ind.values()

True

>>>('o',4)ind.items()

True

Afewthingstonoticeintheprecedingcode.First,noticehowwe’recreatingadictionarybyiteratingoverthezippedversionofthestring'hello'andthelist[0,1,2,3,4].Thestring'hello'hastwo'l'charactersinside,andtheyarepairedupwiththevalues2

and3bythezipfunction.Noticehowinthedictionary,thesecondoccurrenceofthe'l'key(theonewithvalue3),overwritesthefirstone(theonewithvalue2).Anotherthingtonoticeisthatwhenaskingforanyview,theoriginalorderislost,butisconsistentwithintheviews,asexpected.Noticealsothatyoumayhavedifferentresultswhenyoutrythiscodeonyourmachine.Pythondoesn’tguaranteethat,itonlyguaranteestheconsistencyoftheorderinwhichtheviewsarepresented.

We’llseehowtheseviewsarefundamentaltoolswhenwetalkaboutiteratingovercollections.Let’stakealooknowatsomeothermethodsexposedbyPython’sdictionaries,there’splentyofthemandtheyareveryuseful:

>>>d

{'e':1,'h':0,'o':4,'l':3}

>>>d.popitem()#removesarandomitem

('e',1)

>>>d

{'h':0,'o':4,'l':3}

>>>d.pop('l')#removeitemwithkey`l`

3

>>>d.pop('not-a-key')#removeakeynotindictionary:KeyError

Traceback(mostrecentcalllast):

File"<stdin>",line1,in<module>

KeyError:'not-a-key'

>>>d.pop('not-a-key','default-value')#withadefaultvalue?

'default-value'#wegetthedefaultvalue

>>>d.update({'another':'value'})#wecanupdatedictthisway

>>>d.update(a=13)#orthisway(likeafunctioncall)

>>>d

{'a':13,'another':'value','h':0,'o':4}

>>>d.get('a')#sameasd['a']butifkeyismissingnoKeyError

13

>>>d.get('a',177)#defaultvalueusedifkeyismissing

13

>>>d.get('b',177)#likeinthiscase

177

>>>d.get('b')#keyisnotthere,soNoneisreturned

Allthesemethodsarequitesimpletounderstand,butit’sworthtalkingaboutthatNone,foramoment.EveryfunctioninPythonreturnsNone,unlessthereturnstatementisexplicitlyused,butwe’llseethiswhenweexplorefunctions.Noneisfrequentlyusedtorepresenttheabsenceofavalue,aswhendefaultargumentsarenotpassedtoafunction.SomeinexperiencedcoderssometimeswritecodethatreturnseitherFalseorNone.BothFalseandNoneevaluatetoFalsesoitmayseemthereisnotmuchdifferencebetweenthem.Butactually,Iwouldarguethereisquiteanimportantdifference:Falsemeansthatwehaveinformation,andtheinformationwehaveisFalse.Nonemeansnoinformation.Andnoinformationisverydifferentfromaninformation,whichisFalse.Inlayman’sterms,ifyouaskyourmechanic“ismycarready?”thereisabigdifferencebetweentheanswer“No,it’snot”(False)and“Ihavenoidea”(None).

OnelastmethodIreallylikeofdictionariesissetdefault.Itbehaveslikeget,butalsosetsthekeywiththegivenvalueifitisnotthere.Let’sseeandexample:

>>>d={}

>>>d.setdefault('a',1)#'a'ismissing,wegetdefaultvalue

1

>>>d

{'a':1}#also,thekey/valuepair('a',1)hasnowbeenadded

>>>d.setdefault('a',5)#let'strytooverridethevalue

1

>>>d

{'a':1}#didn'twork,asexpected

So,we’renowattheendofthistour.Testyourknowledgeaboutdictionariestryingtoforeseehowdlookslikeafterthisline.

>>>d={}

>>>d.setdefault('a',{}).setdefault('b',[]).append(1)

It’snotthatcomplicated,butdon’tworryifyoudon’tgetitimmediately.Ijustwantedtospuryoutoexperimentwithdictionaries.

Thisconcludesourtourofbuilt-indatatypes.BeforeImakesomeconsiderationsaboutwhatwe’veseeninthischapter,Iwanttobrieflytakeapeekatthecollectionsmodule.

ThecollectionsmoduleWhenPythongeneralpurposebuilt-incontainers(tuple,list,set,anddict)aren’tenough,wecanfindspecializedcontainerdatatypesinthecollectionsmodule.Theyare:

Datatype Description

namedtuple() Afactoryfunctionforcreatingtuplesubclasseswithnamedfields

deque Alist-likecontainerwithfastappendsandpopsoneitherend

ChainMap Adict-likeclassforcreatingasingleviewofmultiplemappings

Counter Adictsubclassforcountinghashableobjects

OrderedDict Adictsubclassthatrememberstheorderentrieswereadded

defaultdict Adictsubclassthatcallsafactoryfunctiontosupplymissingvalues

UserDict Awrapperarounddictionaryobjectsforeasierdictsubclassing

UserList Awrapperaroundlistobjectsforeasierlistsubclassing

UserString Awrapperaroundstringobjectsforeasierstringsubclassing

Wedon’thavetheroomtocoverallofthem,butyoucanfindplentyofexamplesintheofficialdocumentation,sohereI’lljustgiveasmallexampletoshowyounamedtuple,defaultdict,andChainMap.

NamedtuplesAnamedtupleisatuple-likeobjectthathasfieldsaccessiblebyattributelookupaswellasbeingindexableanditerable(it’sactuallyasubclassoftuple).Thisissortofacompromisebetweenafull-fledgedobjectandatuple,anditcanbeusefulinthosecaseswhereyoudon’tneedthefullpowerofacustomobject,butyouwantyourcodetobemorereadablebyavoidingweirdindexing.Anotherusecaseiswhenthereisachancethatitemsinthetupleneedtochangetheirpositionafterrefactoring,forcingthecodertorefactoralsoallthelogicinvolved,whichcanbeverytricky.Asusual,anexampleisbetterthanathousandwords(orwasitapicture?).Saywearehandlingdataabouttheleftandrighteyeofapatient.Wesaveonevalueforthelefteye(position0)andonefortherighteye(position1)inaregulartuple.Here’showthatmightbe:

>>>vision=(9.5,8.8)

>>>vision

(9.5,8.8)

>>>vision[0]#lefteye(implicitpositionalreference)

9.5

>>>vision[1]#righteye(implicitpositionalreference)

8.8

Nowlet’spretendwehandlevisionobjectallthetime,andatsomepointthedesignerdecidestoenhancethembyaddinginformationforthecombinedvision,sothatavisionobjectstoresdatainthisformat:(lefteye,combined,righteye).

Doyouseethetroublewe’reinnow?Wemayhavealotofcodethatdependsonvision[0]beingthelefteyeinformation(whichstillis)andvision[1]beingtherighteyeinformation(whichisnolongerthecase).Wehavetorefactorourcodewhereverwehandletheseobjects,changingvision[1]tovision[2],anditcanbepainful.Wecouldhaveprobablyapproachedthisabitbetterfromthebeginning,byusinganamedtuple.LetmeshowyouwhatImean:

>>>fromcollectionsimportnamedtuple

>>>Vision=namedtuple('Vision',['left','right'])

>>>vision=Vision(9.5,8.8)

>>>vision[0]

9.5

>>>vision.left#sameasvision[0],butexplicit

9.5

>>>vision.right#sameasvision[1],butexplicit

8.8

Ifwithinourcodewerefertoleftandrighteyeusingvision.leftandvision.right,allweneedtodotofixthenewdesignissueistochangeourfactoryandthewaywecreateinstances.Therestofthecodewon’tneedtochange.

>>>Vision=namedtuple('Vision',['left','combined','right'])

>>>vision=Vision(9.5,9.2,8.8)

>>>vision.left#stillperfect

9.5

>>>vision.right#stillperfect(thoughnowisvision[2])

8.8

>>>vision.combined#thenewvision[1]

9.2

Youcanseehowconvenientitistorefertothosevaluesbynameratherthanbyposition.Afterall,awisemanoncewrote“Explicitisbetterthanimplicit”(canyourecallwhere?Thinkzenifyoudon’t…).Thisexamplemaybealittleextreme,ofcourseit’snotlikelythatourcodedesignerwillgoforachangelikethis,butyou’dbeamazedtoseehowfrequentlyissuessimilartothisonehappeninaprofessionalenvironment,andhowpainfulitistorefactorthem.

DefaultdictThedefaultdictdatatypeisoneofmyfavorites.Itallowsyoutoavoidcheckingifakeyisinadictionarybysimplyinsertingitforyouonyourfirstaccessattempt,withadefaultvaluewhosetypeyoupassoncreation.Insomecases,thistoolcanbeveryhandyandshortenyourcodealittle.Let’sseeaquickexample:sayweareupdatingthevalueofage,byaddingoneyear.Ifageisnotthere,weassumeitwas0andweupdateitto1.

>>>d={}

>>>d['age']=d.get('age',0)+1#agenotthere,weget0+1

>>>d

{'age':1}

>>>d={'age':39}

>>>d['age']=d.get('age',0)+1#disthere,weget40

>>>d

{'age':40}

Nowlet’sseehowitwouldworkwithadefaultdictdatatype.Thesecondlineisactuallytheshortversionofa4-lineslongifclausethatwewouldhavetowriteifdictionariesdidn’thavethegetmethod.We’llseeallaboutifclausesinChapter3,IteratingandMakingDecisions.

>>>fromcollectionsimportdefaultdict

>>>dd=defaultdict(int)#intisthedefaulttype(0thevalue)

>>>dd['age']+=1#shortfordd['age']=dd['age']+1

>>>dd

defaultdict(<class'int'>,{'age':1})#1,asexpected

>>>dd['age']=39

>>>dd['age']+=1

>>>dd

defaultdict(<class'int'>,{'age':40})#40,asexpected

Noticehowwejustneedtoinstructthedefaultdictfactorythatwewantanintnumbertobeusedincasethekeyismissing(we’llget0,whichisthedefaultfortheinttype).Also,noticethateventhoughinthisexamplethereisnogainonthenumberoflines,thereisdefinitelyagaininreadability,whichisveryimportant.Youcanalsouseadifferenttechniquetoinstantiateadefaultdictdatatype,whichinvolvescreatingafactoryobject.Fordiggingdeeper,pleaserefertotheofficialdocumentation.

ChainMapTheChainMapisanextremelynicedatatypewhichwasintroducedinPython3.3.ItbehaveslikeanormaldictionarybutaccordingtothePythondocumentation:isprovidedforquicklylinkinganumberofmappingssotheycanbetreatedasasingleunit.Thisisusuallymuchfasterthancreatingonedictionaryandrunningmultipleupdatecallsonit.ChainMapcanbeusedtosimulatenestedscopesandisusefulintemplating.Theunderlyingmappingsarestoredinalist.Thatlistispublicandcanbeaccessedorupdatedusingthemapsattribute.Lookupssearchtheunderlyingmappingssuccessivelyuntilakeyisfound.Incontrast,writes,updates,anddeletionsonlyoperateonthefirstmapping.

Averycommonusecaseisprovidingdefaults,solet’sseeanexample:

>>>fromcollectionsimportChainMap

>>>default_connection={'host':'localhost','port':4567}

>>>connection={'port':5678}

>>>conn=ChainMap(connection,default_connection)#mapcreation

>>>conn['port']#portisfoundinthefirstdictionary

5678

>>>conn['host']#hostisfetchedfromtheseconddictionary

'localhost'

>>>conn.maps#wecanseethemappingobjects

[{'port':5678},{'host':'localhost','port':4567}]

>>>conn['host']='packtpub.com'#let'saddhost

>>>conn.maps

[{'host':'packtpub.com','port':5678},

{'host':'localhost','port':4567}]

>>>delconn['port']#let'sremovetheportinformation

>>>conn.maps

[{'host':'packtpub.com'},

{'host':'localhost','port':4567}]

>>>conn['port']#nowportisfetchedfromtheseconddictionary

4567

>>>dict(conn)#easytomergeandconverttoregulardictionary

{'host':'packtpub.com','port':4567}

IjustlovehowPythonmakesyourlifeeasy.YouworkonaChainMapobject,configurethefirstmappingasyouwant,andwhenyouneedacompletedictionarywithallthedefaultsaswellasthecustomizeditems,youjustfeedtheChainMapobjecttoadictconstructor.Ifyouhavenevercodedinotherlanguages,suchasJavaorC++,youprobablywon’tbeabletofullyappreciatehowpreciousthisis,howPythonmakesyourlifesomucheasier.Ido,IfeelclaustrophobiceverytimeIhavetocodeinsomeotherlanguage.

FinalconsiderationsThat’sit.NowyouhaveseenaverygoodportionofthedatastructuresthatyouwilluseinPython.IencourageyoutotakeadiveintothePythondocumentationandexperimentfurtherwitheachandeverydatatypewe’veseeninthischapter.It’sworthit,believeme.Everythingyou’llwritewillbeabouthandlingdata,somakesureyourknowledgeaboutitisrocksolid.

Beforeweleapintothenextchapter,I’dliketomakesomefinalconsiderationsaboutdifferentaspectsthattomymindareimportantandnottobeneglected.

SmallvaluescachingWhenwediscussedobjectsatthebeginningofthischapter,wesawthatwhenweassignedanametoanobject,Pythoncreatestheobject,setsitsvalue,andthenpointsthenametoit.Wecanassigndifferentnamestothesamevalueandweexpectdifferentobjectstobecreated,likethis:

>>>a=1000000

>>>b=1000000

>>>id(a)==id(b)

False

Intheprecedingexample,aandbareassignedtotwointobjects,whichhavethesamevaluebuttheyarenotthesameobject,asyoucansee,theiridisnotthesame.Solet’sdoitagain:

>>>a=5

>>>b=5

>>>id(a)==id(b)

True

Ohoh!IsPythonbroken?Whyarethetwoobjectsthesamenow?Wedidn’tdoa=b=5,wesetthemupseparately.Well,theanswerisperformances.Pythoncachesshortstringsandsmallnumbers,toavoidhavingmanycopiesofthemcloggingupthesystemmemory.Everythingishandledproperlyunderthehoodsoyoudon’tneedtoworryabit,butmakesurethatyourememberthisbehaviorshouldyourcodeeverneedtofiddlewithIDs.

HowtochoosedatastructuresAswe’veseen,Pythonprovidesyouwithseveralbuilt-indatatypesandsometimes,ifyou’renotthatexperienced,choosingtheonethatservesyoubestcanbetricky,especiallywhenitcomestocollections.Forexample,sayyouhavemanydictionariestostore,eachofwhichrepresentsacustomer.Withineachcustomerdictionarythere’san'id':'code'uniqueidentificationcode.Inwhatkindofcollectionwouldyouplacethem?Well,unlessIknowmoreaboutthesecustomers,it’sveryhardtoanswer.WhatkindofaccesswillIneed?WhatsortofoperationswillIhavetoperformoneachofthem,andhowmanytimes?Willthecollectionchangeovertime?WillIneedtomodifythecustomerdictionariesinanyway?WhatisgoingtobethemostfrequentoperationIwillhavetoperformonthecollection?

Ifyoucananswertheprecedingquestions,thenyouwillknowwhattochoose.Ifthecollectionnevershrinksorgrows(inotherwords,itwon’tneedtoadd/deleteanycustomerobjectaftercreation)orshuffles,thentuplesareapossiblechoice.Otherwiselistsareagoodcandidate.Everycustomerdictionaryhasauniqueidentifierthough,soevenadictionarycouldwork.Letmedrafttheseoptionsforyou:

#examplecustomerobjects

customer1={'id':'abc123','full_name':'MasterYoda'}

customer2={'id':'def456','full_name':'Obi-WanKenobi'}

customer3={'id':'ghi789','full_name':'AnakinSkywalker'}

#collecttheminatuple

customers=(customer1,customer2,customer3)

#orcollecttheminalist

customers=[customer1,customer2,customer3]

#ormaybewithinadictionary,theyhaveauniqueidafterall

customers={

'abc123':customer1,

'def456':customer2,

'ghi789':customer3,

}

Somecustomerswehavethere,right?Iprobablywouldn’tgowiththetupleoption,unlessIwantedtohighlightthatthecollectionisnotgoingtochange.I’dsayusuallyalistisbetter,itallowsformoreflexibility.

Anotherfactortokeepinmindisthattuplesandlistsareorderedcollections,whileifyouuseadictionaryorasetyoulosetheordering,soyouneedtoknowiforderingisimportantinyourapplication.

Whataboutperformances?Forexampleinalist,operationssuchasinsertionandmembershipcantakeO(n),whiletheyareO(1)foradictionary.It’snotalwayspossibletousedictionariesthough,ifwedon’thavetheguaranteethatwecanuniquelyidentifyeachitemofthecollectionbymeansofoneofitsproperties,andthatthepropertyinquestionishashable(soitcanbeakeyindict).

NoteIfyou’rewonderingwhatO(n)andO(1)mean,pleaseGoogle“bigOnotation”andgeta

gistofitfromanywhere.Inthiscontext,let’sjustsaythatifperforminganoperationOp

onadatastructuretakesO(f(n)),itwouldmeanthatOptakesatmostatime tocomplete,wherecissomepositiveconstant,nisthesizeoftheinput,andfissomefunction.So,thinkofO(…)asanupperboundfortherunningtimeofanoperation(itcanbeusedalsotosizeothermeasurablequantities,ofcourse).

Anotherwayofunderstandingifyouhavechosentherightdatastructureisbylookingatthecodeyouhavetowriteinordertomanipulateit.Ifeverythingcomeseasilyandflowsnaturally,thenyouprobablyhavechosencorrectly,butifyoufindyourselfthinkingyourcodeisgettingunnecessarilycomplicated,thenyouprobablyshouldtryanddecidewhetheryouneedtoreconsideryourchoices.It’squitehardtogiveadvicewithoutapracticalcasethough,sowhenyouchooseadatastructureforyourdata,trytokeepeaseofuseandperformanceinmindandgiveprecedencetowhatmattersmostinthecontextyouare.

AboutindexingandslicingAtthebeginningofthischapter,wesawslicingappliedonstrings.Slicingingeneralappliestoasequence,sotuples,lists,strings,etc.Withlists,slicingcanalsobeusedforassignment.I’vealmostneverseenthisusedinprofessionalcode,butstill,youknowyoucan.Couldyouslicedictionariesorsets?Ihearyouscream“Ofcoursenot!Theyarenotordered!“.Excellent,Iseewe’reonthesamepagehere,solet’stalkaboutindexing.

ThereisonecharacteristicaboutPythonindexingIhaven’tmentionedbefore.I’llshowyoubyexample.Howdoyouaddressthelastelementofacollection?Let’ssee:

>>>a=list(range(10))#`a`has10elements.Lastoneis9.

>>>a

[0,1,2,3,4,5,6,7,8,9]

>>>len(a)#itslengthis10elements

10

>>>a[len(a)-1]#positionoflastoneislen(a)-1

9

>>>a[-1]#butwedon'tneedlen(a)!Pythonrocks!

9

>>>a[-2]#equivalenttolen(a)-2

8

>>>a[-3]#equivalenttolen(a)-3

7

Ifthelistahas10elements,becauseofthe0-indexpositioningsystemofPython,thefirstoneisatposition0andthelastoneisatposition9.Intheprecedingexample,theelementsareconvenientlyplacedinapositionequaltotheirvalue:0isatposition0,1atposition1,andsoon.

So,inordertofetchthelastelement,weneedtoknowthelengthofthewholelist(ortuple,orstring,andsoon)andthensubtract1.Hence:len(a)–1.ThisissocommonanoperationthatPythonprovidesyouwithawaytoretrieveelementsusingnegativeindexing.Thisprovesveryusefulwhenyoudosomeseriousdatamanipulation.Here’sanicediagramabouthowindexingworksonthestring"HelloThere":

Tryingtoaddressindexesgreaterthan9orsmallerthan-10willraiseanIndexError,asexpected.

AboutthenamesYoumayhavenoticedthat,inordertokeeptheexampleasshortaspossible,Ihavecalledmanyobjectsusingsimpleletters,likea,b,c,d,andsoon.Thisisperfectlyokwhenyoudebugontheconsoleorwhenyoushowthata+b==7,butit’sbadpracticewhenitcomestoprofessionalcoding(oranytypeofcoding,forallthatmatter).IhopeyouwillindulgemeifIsometimesdoit,thereasonistopresentthecodeinamorecompactway.

Inarealenvironmentthough,whenyouchoosenamesforyourdata,youshouldchoosethemcarefullyandtheyshouldreflectwhatthedataisabout.So,ifyouhaveacollectionofCustomerobjects,customersisaperfectlygoodnameforit.Wouldcustomers_list,customers_tuple,orcustomers_collectionworkaswell?Thinkaboutitforasecond.Isitgoodtotiethenameofthecollectiontothedatatype?Idon’tthinkso,atleastinmostcases.SoI’dsayifyouhaveanexcellentreasontodosogoahead,otherwisedon’t.Thereasonis,oncethatcustomers_tuplestartsbeingusedindifferentplacesofyourcode,andyourealizeyouactuallywanttousealistinsteadofatuple,you’reupforsomefunrefactoring(alsoknownaswastedtime).Namesfordatashouldbenouns,andnamesforfunctionsshouldbeverbs.Namesshouldbeasexpressiveaspossible.Pythonisactuallyaverygoodexamplewhenitcomestonames.Mostofthetimeyoucanjustguesswhatafunctioniscalledifyouknowwhatitdoes.Crazy,huh?

Chapter2,MeaningfulNamesofCleanCode,RobertC.Martin,PrenticeHallisentirelydedicatedtonames.It’sanamazingbookthathelpedmeimprovemycodingstyleinmanydifferentways,amustreadifyouwanttotakeyourcodingtothenextlevel.

SummaryInthischapter,we’veexploredthebuilt-indatatypesofPython.We’veseenhowmanytheyareandhowmuchcanbeachievedbyjustusingthemindifferentcombinations.

We’veseennumbertypes,sequences,sets,mappings,collections,we’veseenthateverythingisanobject,we’velearnedthedifferencebetweenmutableandimmutable,andwe’vealsolearnedaboutslicingandindexing(and,proudly,negativeindexingaswell).

We’vepresentedsimpleexamples,butthere’smuchmorethatyoucanlearnaboutthissubject,sostickyournoseintotheofficialdocumentationandexplore.

Mostofall,Iencourageyoutotryoutalltheexercisesbyyourself,getyourfingersusingthatcode,buildsomemusclememory,andexperiment,experiment,experiment.Learnwhathappenswhenyoudividebyzero,whenyoucombinedifferentnumbertypesintoasingleexpression,whenyoumanagestrings.Playwithalldatatypes.Exercisethem,breakthem,discoveralltheirmethods,enjoythemandlearnthemwell,damnwell.

Ifyourfoundationisnotrocksolid,howgoodcanyourcodebe?Anddataisthefoundationforeverything.Datashapeswhatdancesaroundit.

Themoreyouprogresswiththebook,themoreit’slikelythatyouwillfindsomediscrepanciesormaybeasmalltypohereandthereinmycode(oryours).Youwillgetanerrormessage,somethingwillbreak.That’swonderful!Whenyoucode,thingsbreakallthetime,youdebugandfixallthetime,soconsidererrorsasusefulexercisestolearnsomethingnewaboutthelanguageyou’reusing,andnotasfailuresorproblems.Errorswillkeepcomingupuntilyourverylastlineofcode,that’sforsure,soyoumayaswellstartmakingyourpeacewiththemnow.

Thenextchapterisaboutiteratingandmakingdecisions.We’llseehowtoactuallyputthosecollectionsinuse,andtakedecisionsbasedonthedatawe’representedwith.We’llstarttogoalittlefasternowthatyourknowledgeisbuildingup,somakesureyou’recomfortablewiththecontentsofthischapterbeforeyoumovetothenextone.Oncemore,havefun,explore,breakthings.It’saverygoodwaytolearn.

Chapter3.IteratingandMakingDecisions “Insanity:doingthesamethingoverandoveragainandexpectingdifferentresults.”

—AlbertEinstein

Inthepreviouschapter,we’veseenPythonbuilt-indatatypes.Nowthatyou’refamiliarwithdatainitsmanyformsandshapes,it’stimetostartlookingathowaprogramcanuseit.

AccordingtoWikipedia:

Incomputerscience,controlflow(oralternatively,flowofcontrol)referstothespecificationoftheorderinwhichtheindividualstatements,instructionsorfunctioncallsofanimperativeprogramareexecutedorevaluated.

Inordertocontroltheflowofaprogram,wehavetwomainweapons:conditionalprogramming(alsoknownasbranching)andlooping.Wecanusetheminmanydifferentcombinationsandvariations,butinthischapter,insteadofgoingthroughallpossiblevariousformsofthosetwoconstructsina“documentation”fashion,I’drathergiveyouthebasicsandthenI’llwriteacoupleofsmallscriptswithyou.Inthefirstone,we’llseehowtocreatearudimentaryprimenumbergenerator,whileinthesecondone,we’llseehowtoapplydiscountstocustomersbasedoncoupons.Thiswayyoushouldgetabetterfeelingabouthowconditionalprogrammingandloopingcanbeused.

ConditionalprogrammingConditionalprogramming,orbranching,issomethingyoudoeveryday,everymoment.It’saboutevaluatingconditions:ifthelightisgreen,thenIcancross,ifit’sraining,thenI’mtakingtheumbrella,andifI’mlateforwork,thenI’llcallmymanager.

Themaintoolistheifstatement,whichcomesindifferentformsandcolors,butbasicallywhatitdoesisevaluateanexpressionand,basedontheresult,choosewhichpartofthecodetoexecute.Asusual,let’sseeanexample:conditional.1.py

late=True

iflate:

print('Ineedtocallmymanager!')

Thisispossiblythesimplestexample:whenfedtotheifstatement,lateactsasaconditionalexpression,whichisevaluatedinaBooleancontext(exactlylikeifwewerecallingbool(late)).IftheresultoftheevaluationisTrue,thenweenterthebodyofcodeimmediatelyaftertheifstatement.Noticethattheprintinstructionisindented:thismeansitbelongstoascopedefinedbytheifclause.Executionofthiscodeyields:

$pythonconditional.1.py

Ineedtocallmymanager!

SincelateisTrue,theprintstatementwasexecuted.Let’sexpandonthisexample:conditional.2.py

late=False

iflate:

print('Ineedtocallmymanager!')#1

else:

print('noneedtocallmymanager…')#2

ThistimeIsetlate=False,sowhenIexecutethecode,theresultisdifferent:

$pythonconditional.2.py

noneedtocallmymanager…

Dependingontheresultofevaluatingthelateexpression,wecaneitherenterblock#1orblock#2,butnotboth.Block#1isexecutedwhenlateevaluatestoTrue,whileblock#2isexecutedwhenlateevaluatestoFalse.TryassigningFalse/Truevaluestothelatename,andseehowtheoutputforthiscodechangesaccordingly.

Theprecedingexamplealsointroducestheelseclause,whichbecomesveryhandywhenwewanttoprovideanalternativesetofinstructionstobeexecutedwhenanexpressionevaluatestoFalsewithinanifclause.Theelseclauseisoptional,asit’sevidentbycomparingtheprecedingtwoexamples.

Aspecializedelse:elifSometimesallyouneedistodosomethingifaconditionismet(simpleifclause).Othertimesyouneedtoprovideanalternative,incasetheconditionisFalse(if/elseclause),buttherearesituationswhereyoumayhavemorethantwopathstochoosefrom,so,sincecallingthemanager(ornotcallingthem)iskindofabinarytypeofexample(eitheryoucalloryoudon’t),let’schangethetypeofexampleandkeepexpanding.Thistimewedecidetaxpercentages.Ifmyincomeislessthen10k,Iwon’tpayanytaxes.Ifitisbetween10kand30k,I’llpay20%taxes.Ifitisbetween30kand100k,I’llpay35%taxes,andover100k,I’ll(gladly)pay45%taxes.Let’sputthisalldownintobeautifulPythoncode:taxes.py

income=15000

ifincome<10000:

tax_coefficient=0.0#1

elifincome<30000:

tax_coefficient=0.2#2

elifincome<100000:

tax_coefficient=0.35#3

else:

tax_coefficient=0.45#4

print('Iwillpay:',income*tax_coefficient,'intaxes')

Executingtheprecedingcodeyields:

$pythontaxes.py

Iwillpay:3000.0intaxes

Let’sgothroughtheexamplelinebyline:westartbysettinguptheincomevalue.Intheexample,myincomeis15k.Weentertheifclause.Noticethatthistimewealsointroducedtheelifclause,whichisacontractionforelse-if,andit’sdifferentfromabareelseclauseinthatitalsohasitsowncondition.So,theifexpressionincome<10000,evaluatestoFalse,thereforeblock#1isnotexecuted.Thecontrolpassestothenextconditionevaluator:elifincome<30000.ThisoneevaluatestoTrue,thereforeblock#2isexecuted,andbecauseofthis,Pythonthenresumesexecutionafterthewholeif/elif/elif/elseclause(whichwecanjustcallifclausefromnowon).Thereisonlyoneinstructionaftertheifclause,theprintcall,whichtellsusIwillpay3kintaxesthisyear(15k*20%).Noticethattheorderismandatory:ifcomesfirst,then(optionally)asmanyelifasyouneed,andthen(optionally)anelseclause.

Interesting,right?Nomatterhowmanylinesofcodeyoumayhavewithineachblock,whenoneoftheconditionsevaluatestoTrue,theassociatedblockisexecutedandthenexecutionresumesafterthewholeclause.IfnoneoftheconditionsevaluatestoTrue(forexample,income=200000),thenthebodyoftheelseclausewouldbeexecuted(block#4).Thisexampleexpandsourunderstandingofthebehavioroftheelseclause.Itsblockofcodeisexecutedwhennoneoftheprecedingif/elif/…/elifexpressionshasevaluatedtoTrue.

Trytomodifythevalueofincomeuntilyoucancomfortablyexecuteallblocksatyourwill(oneperexecution,ofcourse).Andthentrytheboundaries.Thisiscrucial,wheneveryouhaveconditionsexpressedasequalitiesorinequalities(==,!=,<,>,<=,>=),thosenumbersrepresentboundaries.Itisessentialtotestboundariesthoroughly.ShouldIallowyoutodriveat18or17?AmIcheckingyouragewithage<18,orage<=18?Youcan’timaginehowmanytimesIhadtofixsubtlebugsthatstemmedfromusingthewrongoperator,sogoaheadandexperimentwiththeprecedingcode.Changesome<to<=andsetincometobeoneoftheboundaryvalues(10k,30k,100k)aswellasanyvalueinbetween.Seehowtheresultchanges,getagoodunderstandingofitbeforeproceeding.

Beforewemovetothenexttopic,let’sseeanotherexamplethatshowsushowtonestifclauses.Sayyourprogramencountersanerror.Ifthealertsystemistheconsole,weprinttheerror.Ifthealertsystemisane-mail,wesenditaccordingtotheseverityoftheerror.Ifthealertsystemisanythingotherthanconsoleore-mail,wedon’tknowwhattodo,thereforewedonothing.Let’sputthisintocode:errorsalert.py

alert_system='console'#othervaluecanbe'email'

error_severity='critical'#othervalues:'medium'or'low'

error_message='OMG!Somethingterriblehappened!'

ifalert_system=='console':

print(error_message)#1

elifalert_system=='email':

iferror_severity=='critical':

send_email('admin@example.com',error_message)#2

eliferror_severity=='medium':

send_email('support.1@example.com',error_message)#3

else:

send_email('support.2@example.com',error_message)#4

Theprecedingexampleisquiteinteresting,initssilliness.Itshowsustwonestedifclauses(outerandinner).Italsoshowsustheouterifclausedoesn’thaveanyelse,whiletheinneronedoes.Noticehowindentationiswhatallowsustonestoneclausewithinanotherone.

Ifalert_system=='console',body#1isexecuted,andnothingelsehappens.Ontheotherhand,ifalert_system=='email',thenweenterintoanotherifclause,whichwecalledinner.Intheinnerifclause,accordingtoerror_severity,wesendane-mailtoeitheranadmin,first-levelsupport,orsecond-levelsupport(blocks#2,#3,and#4).Thesend_emailfunctionisnotdefinedinthisexample,thereforetryingtorunitwouldgiveyouanerror.Inthesourcecodeofthebook,whichyoucandownloadfromthewebsite,Iincludedatricktoredirectthatcalltoaregularprintfunction,justsoyoucanexperimentontheconsolewithoutactuallysendingane-mail.Trychangingthevaluesandseehowitallworks.

TheternaryoperatorOnelastthingIwouldliketoshowyoubeforemovingontothenextsubject,istheternaryoperatoror,inlayman’sterms,theshortversionofanif/elseclause.Whenthevalueofanameistobeassignedaccordingtosomecondition,sometimesit’seasierandmorereadabletousetheternaryoperatorinsteadofaproperifclause.Inthefollowingexample,thetwocodeblocksdoexactlythesamething:ternary.py

order_total=247#GBP

#classicif/elseform

iforder_total>100:

discount=25#GBP

else:

discount=0#GBP

print(order_total,discount)

#ternaryoperator

discount=25iforder_total>100else0

print(order_total,discount)

Forsimplecaseslikethis,Ifinditverynicetobeabletoexpressthatlogicinonelineinsteadoffour.Remember,asacoder,youspendmuchmoretimereadingcodethenwritingit,soPythonconcisenessisinvaluable.

Areyouclearonhowtheternaryoperatorworks?Basicallyisname=somethingifconditionelsesomething-else.SonameisassignedsomethingifconditionevaluatestoTrue,andsomething-elseifconditionevaluatestoFalse.

Nowthatyouknoweverythingaboutcontrollingthepathofthecode,let’smoveontothenextsubject:looping.

LoopingIfyouhaveanyexperiencewithloopinginotherprogramminglanguages,youwillfindPython’swayofloopingabitdifferent.Firstofall,whatislooping?Loopingmeansbeingabletorepeattheexecutionofacodeblockmorethanonce,accordingtotheloopparameterswe’regiven.Therearedifferentloopingconstructs,whichservedifferentpurposes,andPythonhasdistilledallofthemdowntojusttwo,whichyoucanusetoachieveeverythingyouneed.Thesearetheforandwhilestatements.

Whileit’sdefinitelypossibletodoeverythingyouneedusingeitherofthem,theyservedifferentpurposesandthereforethey’reusuallyusedindifferentcontexts.We’llexplorethisdifferencethoroughlythroughthischapter.

TheforloopTheforloopisusedwhenloopingoverasequence,likealist,tuple,oracollectionofobjects.Let’sstartwithasimpleexamplethatismorelikeC++style,andthenlet’sgraduallyseehowtoachievethesameresultsinPython(you’lllovePython’ssyntax).simple.for.py

fornumberin[0,1,2,3,4]:

print(number)

Thissimplesnippetofcode,whenexecuted,printsallnumbersfrom0to4.Theforloopisfedthelist[0,1,2,3,4]andateachiteration,numberisgivenavaluefromthesequence(whichisiteratedsequentially,inorder),thenthebodyoftheloopisexecuted(theprintline).numberchangesateveryiteration,accordingtowhichvalueiscomingnextfromthesequence.Whenthesequenceisexhausted,theforloopterminates,andtheexecutionofthecoderesumesnormallywiththecodeaftertheloop.

IteratingoverarangeSometimesweneedtoiterateoverarangeofnumbers,anditwouldbequiteunpleasanttohavetodosobyhardcodingthelistsomewhere.Insuchcases,therangefunctioncomestotherescue.Let’sseetheequivalentoftheprevioussnippetofcode:simple.for.py

fornumberinrange(5):

print(number)

TherangefunctionisusedextensivelyinPythonprogramswhenitcomestocreatingsequences:youcancallitbypassingonevalue,whichactsasstop(countingfrom0),oryoucanpasstwovalues(startandstop),oreventhree(start,stop,andstep).Checkoutthefollowingexample:

>>>list(range(10))#onevalue:from0tovalue(excluded)

[0,1,2,3,4,5,6,7,8,9]

>>>list(range(3,8))#twovalues:fromstarttostop(excluded)

[3,4,5,6,7]

>>>list(range(-10,10,4))#threevalues:stepisadded

[-10,-6,-2,2,6]

Forthemoment,ignorethatweneedtowraprange(...)withinalist.Therangeobjectisalittlebitspecial,butinthiscasewe’rejustinterestedinunderstandingwhatarethevaluesitwillreturntous.Youseethatthedealisthesamewithslicing:startisincluded,stopexcluded,andoptionallyyoucanaddastepparameter,whichbydefaultis1.

Trymodifyingtheparametersoftherange()callinoursimple.for.pycodeandseewhatitprints,getcomfortablewithit.

IteratingoverasequenceNowwehaveallthetoolstoiterateoverasequence,solet’sbuildonthatexample:simple.for.2.py

surnames=['Rivest','Shamir','Adleman']

forpositioninrange(len(surnames)):

print(position,surnames[position])

Theprecedingcodeaddsalittlebitofcomplexitytothegame.Executionwillshowthisresult:

$pythonsimple.for.2.py

0Rivest

1Shamir

2Adleman

Let’susetheinside-outtechniquetobreakitdown,ok?Westartfromtheinnermostpartofwhatwe’retryingtounderstand,andweexpandoutwards.So,len(surnames)isthelengthofthesurnameslist:3.Therefore,range(len(surnames))isactuallytransformedintorange(3).Thisgivesustherange[0,3),whichisbasicallyasequence(0,1,2).Thismeansthattheforloopwillrunthreeiterations.Inthefirstone,positionwilltakevalue0,whileinthesecondone,itwilltakevalue1,andfinallyvalue2inthethirdandlastiteration.Whatis(0,1,2),ifnotthepossibleindexingpositionsforthesurnameslist?Atposition0wefind'Rivest',atposition1,'Shamir',andatposition2,'Adleman'.Ifyouarecuriousaboutwhatthesethreemencreatedtogether,changeprint(position,surnames[position])toprint(surnames[position][0],end='')addafinalprint()outsideoftheloop,andrunthecodeagain.

Now,thisstyleofloopingisactuallymuchclosertolanguageslikeJavaorC++.InPythonit’squiteraretoseecodelikethis.Youcanjustiterateoveranysequenceorcollection,sothereisnoneedtogetthelistofpositionsandretrieveelementsoutofasequenceateachiteration.It’sexpensive,needlesslyexpensive.Let’schangetheexampleintoamorePythonicform:simple.for.3.py

surnames=['Rivest','Shamir','Adleman']

forsurnameinsurnames:

print(surname)

Nowthat’ssomething!It’spracticallyEnglish.Theforloopcaniterateoverthesurnameslist,anditgivesbackeachelementinorderateachinteraction.Runningthiscodewillprintthethreesurnames,oneatatime.It’smucheasiertoread,right?

Whatifyouwantedtoprintthepositionaswellthough?Orwhatifyouactuallyneededitforanyreason?Shouldyougobacktotherange(len(...))form?No.Youcanusetheenumeratebuilt-infunction,likethis:simple.for.4.py

surnames=['Rivest','Shamir','Adleman']

forposition,surnameinenumerate(surnames):

print(position,surname)

Thiscodeisveryinterestingaswell.Noticethatenumerategivesbacka2-tuple(position,surname)ateachiteration,butstill,it’smuchmorereadable(andmoreefficient)thantherange(len(...))example.Youcancallenumeratewithastart

parameter,likeenumerate(iterable,start),anditwillstartfromstart,ratherthan0.JustanotherlittlethingthatshowsyouhowmuchthoughthasbeengivenindesigningPythonsothatitmakesyourlifeeasy.

Usingaforloopitispossibletoiterateoverlists,tuples,andingeneralanythingthatinPythoniscallediterable.Thisisaveryimportantconcept,solet’stalkaboutitabitmore.

IteratorsanditerablesAccordingtothePythondocumentation,aniterableis:

“Anobjectcapableofreturningitsmembersoneatatime.Examplesofiterablesincludeallsequencetypes(suchaslist,str,andtuple)andsomenon-sequencetypeslikedict,fileobjects,andobjectsofanyclassesyoudefinewithan__iter__()or__getitem__()method.Iterablescanbeusedinaforloopandinmanyotherplaceswhereasequenceisneeded(zip(),map(),…).Whenaniterableobjectispassedasanargumenttothebuilt-infunctioniter(),itreturnsaniteratorfortheobject.Thisiteratorisgoodforonepassoverthesetofvalues.Whenusingiterables,itisusuallynotnecessarytocalliter()ordealwithiteratorobjectsyourself.Theforstatementdoesthatautomaticallyforyou,creatingatemporaryunnamedvariabletoholdtheiteratorforthedurationoftheloop.”

Simplyput,whathappenswhenyouwriteforkinsequence:...body…,isthattheforloopaskssequenceforthenextelement,itgetssomethingback,itcallsthatsomethingk,andthenexecutesitsbody.Then,onceagain,theforloopaskssequenceagainforthenextelement,itcallsitkagain,andexecutesthebodyagain,andsoonandsoforth,untilthesequenceisexhausted.Emptysequenceswillresultinzeroexecutionsofthebody.

Somedatastructures,wheniteratedover,producetheirelementsinorder,likelists,tuples,andstrings,whilesomeothersdon’t,likesetsanddictionaries.

Pythongivesustheabilitytoiterateoveriterables,usingatypeofobjectcallediterator.Accordingtotheofficialdocumentation,aniteratoris:

“Anobjectrepresentingastreamofdata.Repeatedcallstotheiterator’s__next__()method(orpassingittothebuilt-infunctionnext())returnsuccessiveitemsinthestream.WhennomoredataareavailableaStopIterationexceptionisraisedinstead.Atthispoint,theiteratorobjectisexhaustedandanyfurthercallstoits__next__()methodjustraiseStopIterationagain.Iteratorsarerequiredtohavean__iter__()methodthatreturnstheiteratorobjectitselfsoeveryiteratorisalsoiterableandmaybeusedinmostplaceswhereotheriterablesareaccepted.Onenotableexceptioniscodewhichattemptsmultipleiterationpasses.Acontainerobject(suchasalist)producesafreshnewiteratoreachtimeyoupassittotheiter()functionoruseitinaforloop.Attemptingthiswithaniteratorwilljustreturnthesameexhaustediteratorobjectusedinthepreviousiterationpass,makingitappearlikeanemptycontainer.”

Don’tworryifyoudon’tfullyunderstandalltheprecedinglegalese,youwillinduetime.Iputithereasahandyreferenceforthefuture.

Inpractice,thewholeiterable/iteratormechanismissomewhathiddenbehindthecode.Unlessyouneedtocodeyourowniterableoriteratorforsomereason,youwon’thavetoworryaboutthistoomuch.Butit’sveryimportanttounderstandhowPythonhandlesthis

keyaspectofcontrolflowbecauseitwillshapethewayyouwillwriteyourcode.

IteratingovermultiplesequencesLet’sseeanotherexampleofhowtoiterateovertwosequencesofthesamelength,inordertoworkontheirrespectiveelementsinpairs.Saywehavealistofpeopleandalistofnumbersrepresentingtheageofthepeopleinthefirstlist.Wewanttoprintapairperson/ageononelineforallofthem.Let’sstartwithanexampleandlet’srefineitgradually.multiple.sequences.py

people=['Jonas','Julio','Mike','Mez']

ages=[25,30,31,39]

forpositioninrange(len(people)):

person=people[position]

age=ages[position]

print(person,age)

Bynow,thiscodeshouldbeprettystraightforwardforyoutounderstand.Weneedtoiterateoverthelistofpositions(0,1,2,3)becausewewanttoretrieveelementsfromtwodifferentlists.Executingitwegetthefollowing:

$pythonmultiple.sequences.py

Jonas25

Julio30

Mike31

Mez39

ThiscodeisbothinefficientandnotPythonic.Inefficientbecauseretrievinganelementgiventhepositioncanbeanexpensiveoperation,andwe’redoingitfromscratchateachiteration.Themailmandoesn’tgobacktothebeginningoftheroadeachtimehedeliversaletter,right?Hemovesfromhousetohouse.Fromonetothenextone.Let’strytomakeitbetterusingenumerate:multiple.sequences.enumerate.py

people=['Jonas','Julio','Mike','Mez']

ages=[25,30,31,39]

forposition,personinenumerate(people):

age=ages[position]

print(person,age)

Better,butstillnotperfect.Andstillabitugly.We’reiteratingproperlyonpeople,butwe’restillfetchingageusingpositionalindexing,whichwewanttoloseaswell.Well,noworries,Pythongivesyouthezipfunction,remember?Let’suseit!multiple.sequences.zip.py

people=['Jonas','Julio','Mike','Mez']

ages=[25,30,31,39]

forperson,ageinzip(people,ages):

print(person,age)

Ah!Somuchbetter!Onceagain,comparetheprecedingcodewiththefirstexampleandadmirePython’selegance.ThereasonIwantedtoshowthisexampleistwofold.Onthe

onehand,IwantedtogiveyouanideaofhowshorterthecodeinPythoncanbecomparedtootherlanguageswherethesyntaxdoesn’tallowyoutoiterateoversequencesorcollectionsaseasily.Andontheotherhand,andmuchmoreimportantly,noticethatwhentheforloopaskszip(sequenceA,sequenceB)forthenextelement,itgetsbackatuple,notjustasingleobject.Itgetsbackatuplewithasmanyelementsasthenumberofsequenceswefeedtothezipfunction.Let’sexpandalittleonthepreviousexampleintwoways:usingexplicitandimplicitassignment:multiple.sequences.explicit.py

people=['Jonas','Julio','Mike','Mez']

ages=[25,30,31,39]

nationalities=['Belgium','Spain','England','Bangladesh']

forperson,age,nationalityinzip(people,ages,nationalities):

print(person,age,nationality)

Intheprecedingcode,weaddedthenationalitieslist.Nowthatwefeedthreesequencestothezipfunction,theforloopgetsbacka3-tupleateachiteration.Noticethatthepositionoftheelementsinthetuplerespectsthepositionofthesequencesinthezipcall.Executingthecodewillyieldthefollowingresult:

$pythonmultiple.sequences.explicit.py

Jonas25Belgium

Julio30Spain

Mike31England

Mez39Bangladesh

Sometimes,forreasonsthatmaynotbeclearinasimpleexampleliketheprecedingone,youmaywanttoexplodethetuplewithinthebodyoftheforloop.Ifthatisyourdesire,it’sperfectlypossibletodoso.multiple.sequences.implicit.py

people=['Jonas','Julio','Mike','Mez']

ages=[25,30,31,39]

nationalities=['Belgium','Spain','England','Bangladesh']

fordatainzip(people,ages,nationalities):

person,age,nationality=data

print(person,age,nationality)

It’sbasicallydoingwhattheforloopdoesautomaticallyforyou,butinsomecasesyoumaywanttodoityourself.Here,the3-tupledatathatcomesfromzip(...),isexplodedwithinthebodyoftheforloopintothreevariables:person,age,andnationality.

ThewhileloopIntheprecedingpages,wesawtheforloopinaction.It’sincrediblyusefulwhenyouneedtoloopoverasequenceoracollection.Thekeypointtokeepinmind,whenyouneedtobeabletodiscriminatewhichloopingconstructtouse,isthattheforlooprockswhenyouhavetoiterateoverafiniteamountofelements.Itcanbeahugeamount,butstill,somethingthatatsomepointends.

Thereareothercasesthough,whenyoujustneedtoloopuntilsomeconditionissatisfied,orevenloopindefinitelyuntiltheapplicationisstopped.Caseswherewedon’treallyhavesomethingtoiterateon,andthereforetheforloopwouldbeapoorchoice.Butfearnot,forthesecasesPythonprovidesuswiththewhileloop.

Thewhileloopissimilartotheforloop,inthattheybothloopandateachiterationtheyexecuteabodyofinstructions.Whatisdifferentbetweenthemisthatthewhileloopdoesn’tloopoverasequence(itcan,butyouhavetomanuallywritethelogicanditwouldn’tmakeanysense,youwouldjustwanttouseaforloop),rather,itloopsaslongasacertainconditionissatisfied.Whentheconditionisnolongersatisfied,theloopends.

Asusual,let’sseeanexamplewhichwillclarifyeverythingforus.Wewanttoprintthebinaryrepresentationofapositivenumber.Inordertodoso,werepeatedlydividethenumberbytwo,collectingtheremainder,andthenproducetheinverseofthelistofremainders.Letmegiveyouasmallexampleusingnumber6,whichis110inbinary.

6/2=3(remainder:0)

3/2=1(remainder:1)

1/2=0(remainder:1)

Listofremainders:0,1,1.

Inverseis1,1,0,whichisalsothebinaryrepresentationof6:110

Let’swritesomecodetocalculatethebinaryrepresentationfornumber39:1001112.

binary.py

n=39

remainders=[]

whilen>0:

remainder=n%2#remainderofdivisionby2

remainders.append(remainder)#wekeeptrackofremainders

n//=2#wedividenby2

#reassignthelisttoitsreversedcopyandprintit

remainders=remainders[::-1]

print(remainders)

Intheprecedingcode,Ihighlightedtwothings:n>0,whichistheconditiontokeeplooping,andremainders[::-1]whichisaniceandeasywaytogetthereversedversionofalist(missingstartandendparameters,step=-1,producesthesamelist,fromendtostart,inreverseorder).Wecanmakethecodealittleshorter(andmorePythonic),byusingthedivmodfunction,whichiscalledwithanumberandadivisor,andreturnsatuplewiththeresultoftheintegerdivisionanditsremainder.Forexample,divmod(13,5)

wouldreturn(2,3),andindeed5*2+3=13.binary.2.py

n=39

remainders=[]

whilen>0:

n,remainder=divmod(n,2)

remainders.append(remainder)

#reassignthelisttoitsreversedcopyandprintit

remainders=remainders[::-1]

print(remainders)

Intheprecedingcode,wehavereassignedntotheresultofthedivisionby2,andtheremainder,inonesingleline.

Noticethattheconditioninawhileloopisaconditiontocontinuelooping.IfitevaluatestoTrue,thenthebodyisexecutedandthenanotherevaluationfollows,andsoon,untiltheconditionevaluatestoFalse.Whenthathappens,theloopisexitedimmediatelywithoutexecutingitsbody.

NoteIftheconditionneverevaluatestoFalse,theloopbecomesasocalledinfiniteloop.Infiniteloopsareusedforexamplewhenpollingfromnetworkdevices:youaskthesocketifthereisanydata,youdosomethingwithitifthereisany,thenyousleepforasmallamountoftime,andthenyouaskthesocketagain,overandoveragain,withouteverstopping.

Havingtheabilitytoloopoveracondition,ortoloopindefinitely,isthereasonwhytheforloopaloneisnotenough,andthereforePythonprovidesthewhileloop.

TipBytheway,ifyouneedthebinaryrepresentationofanumber,checkoutthebinfunction.

Justforfun,let’sadaptoneoftheexamples(multiple.sequences.py)usingthewhilelogic.multiple.sequences.while.py

people=['Jonas','Julio','Mike','Mez']

ages=[25,30,31,39]

position=0

whileposition<len(people):

person=people[position]

age=ages[position]

print(person,age)

position+=1

Intheprecedingcode,Ihavehighlightedtheinitialization,condition,andupdateofthevariableposition,whichmakesitpossibletosimulatetheequivalentforloopcodebyhandlingtheiterationvariablemanually.Everythingthatcanbedonewithaforloopcanalsobedonewithawhileloop,eventhoughyoucanseethere’sabitofboilerplateyou

havetogothroughinordertoachievethesameresult.Theoppositeisalsotrue,butsimulatinganeverendingwhileloopusingaforlooprequiressomerealtrickery,sowhywouldyoudothat?Usetherighttoolforthejob,and99.9%ofthetimesyou’llbefine.

So,torecap,useaforloopwhenyouneedtoiterateoverone(oracombinationof)iterable,andawhileloopwhenyouneedtoloopaccordingtoaconditionbeingsatisfiedornot.Ifyoukeepinmindthedifferencebetweenthetwopurposes,youwillneverchoosethewrongloopingconstruct.

Let’snowseehowtoalterthenormalflowofaloop.

ThebreakandcontinuestatementsAccordingtothetaskathand,sometimesyouwillneedtoaltertheregularflowofaloop.Youcaneitherskipasingleiteration(asmanytimesyouwant),oryoucanbreakoutoftheloopentirely.Acommonusecaseforskippingiterationsisforexamplewhenyou’reiteratingoveralistofitemsandyouneedtoworkoneachofthemonlyifsomeconditionisverified.Ontheotherhand,ifyou’reiteratingoveracollectionofitems,andyouhavefoundoneofthemthatsatisfiessomeneedyouhave,youmaydecidenottocontinuetheloopentirelyandthereforebreakoutofit.Therearecountlesspossiblescenarios,soit’sbettertoseeacoupleofexamples.

Let’ssayyouwanttoapplya20%discounttoallproductsinabasketlistforthosewhichhaveanexpirationdateoftoday.Thewayyouachievethisistousethecontinuestatement,whichtellstheloopingconstruct(fororwhile)toimmediatelystopexecutionofthebodyandgotothenextiteration,ifany.Thisexamplewilltakeusalittledeeperdowntherabbitwhole,sobereadytojump.discount.py

fromdatetimeimportdate,timedelta

today=date.today()

tomorrow=today+timedelta(days=1)#today+1dayistomorrow

products=[

{'sku':'1','expiration_date':today,'price':100.0},

{'sku':'2','expiration_date':tomorrow,'price':50},

{'sku':'3','expiration_date':today,'price':20},

]

forproductinproducts:

ifproduct['expiration_date']!=today:

continue

product['price']*=0.8#equivalenttoapplying20%discount

print(

'Priceforsku',product['sku'],

'isnow',product['price'])

Youseewestartbyimportingthedateandtimedeltaobjects,thenwesetupourproducts.Thosewithsku1and3haveanexpirationdateoftoday,whichmeanswewanttoapply20%discountonthem.Weloopovereachproductandweinspecttheexpirationdate.Ifitisnot(inequalityoperator,!=)today,wedon’twanttoexecutetherestofthebodysuite,sowecontinue.

Noticethatisnotimportantwhereinthebodysuiteyouplacethecontinuestatement(youcanevenuseitmorethanonce).Whenyoureachit,executionstopsandgoesbacktothenextiteration.Ifwerunthediscount.pymodule,thisistheoutput:

$pythondiscount.py

Priceforsku1isnow80.0

Priceforsku3isnow16.0

Whichshowsyouthatthelasttwolinesofthebodyhaven’tbeenexecutedforskunumber2.

Let’snowseeanexampleofbreakingoutofaloop.SaywewanttotellifatleastanyoftheelementsinalistevaluatestoTruewhenfedtotheboolfunction.Giventhatweneedtoknowifthereisatleastone,whenwefinditwedon’tneedtokeepscanningthelistanyfurther.InPythoncode,thistranslatestousingthebreakstatement.Let’swritethisdownintocode:any.py

items=[0,None,0.0,True,0,7]#Trueand7evaluatetoTrue

found=False#thisiscalled"flag"

foriteminitems:

print('scanningitem',item)

ifitem:

found=True#weupdatetheflag

break

iffound:#weinspecttheflag

print('AtleastoneitemevaluatestoTrue')

else:

print('AllitemsevaluatetoFalse')

Theprecedingcodeissuchacommonpatterninprogramming,youwillseeitalot.Whenyouinspectitemsthisway,basicallywhatyoudoistosetupaflagvariable,thenstarttheinspection.Ifyoufindoneelementthatmatchesyourcriteria(inthisexample,thatevaluatestoTrue),thenyouupdatetheflagandstopiterating.Afteriteration,youinspecttheflagandtakeactionaccordingly.Executionyields:

$pythonany.py

scanningitem0

scanningitemNone

scanningitem0.0

scanningitemTrue

AtleastoneitemevaluatestoTrue

SeehowexecutionstoppedafterTruewasfound?

Thebreakstatementactsexactlylikethecontinueone,inthatitstopsexecutingthebodyoftheloopimmediately,butalso,preventsanyotheriterationtorun,effectivelybreakingoutoftheloop.

Thecontinueandbreakstatementscanbeusedtogetherwithnolimitationintheirnumber,bothintheforandwhileloopingconstructs.

TipBytheway,thereisnoneedtowritecodetodetectifthereisatleastoneelementinasequencethatevaluatestoTrue.Justcheckouttheanybuilt-infunction.

AspecialelseclauseOneofthefeaturesI’veseenonlyinthePythonlanguageistheabilitytohaveelseclausesafterwhileandforloops.It’sveryrarelyused,butit’sdefinitelynicetohave.Inshort,youcanhaveanelsesuiteafterafororwhileloop.Iftheloopendsnormally,becauseofexhaustionoftheiterator(forloop)orbecausetheconditionisfinallynotmet(whileloop),thentheelsesuite(ifpresent)isexecuted.Incaseexecutionisinterruptedbyabreakstatement,theelseclauseisnotexecuted.Let’stakeanexampleofaforloopthatiteratesoveragroupofitems,lookingforonethatwouldmatchsomecondition.Incasewedon’tfindatleastonethatsatisfiesthecondition,wewanttoraiseanexception.Thismeanswewanttoarresttheregularexecutionoftheprogramandsignalthattherewasanerror,orexception,thatwecannotdealwith.ExceptionswillbethesubjectofChapter7,Testing,Profiling,andDealingwithExceptions,sodon’tworryifyoudon’tfullyunderstandthemnow.Justbearinmindthattheywillaltertheregularflowofthecode.Letmenowshowyoutwoexamplesthatdoexactlythesamething,butoneofthemisusingthespecialfor…elsesyntax.Saythatwewanttofindamongacollectionofpeopleonethatcoulddriveacar.for.no.else.py

classDriverException(Exception):

pass

people=[('James',17),('Kirk',9),('Lars',13),('Robert',8)]

driver=None

forperson,ageinpeople:

ifage>=18:

driver=(person,age)

break

ifdriverisNone:

raiseDriverException('Drivernotfound.')

Noticetheflagpatternagain.WesetdrivertobeNone,thenifwefindoneweupdatethedriverflag,andthen,attheendoftheloop,weinspectittoseeifonewasfound.Ikindofhavethefeelingthatthosekidswoulddriveaverymetalliccar,butanyway,noticethatifadriverisnotfound,aDriverExceptionisraised,signalingtheprogramthatexecutioncannotcontinue(we’relackingthedriver).

Thesamefunctionalitycanberewrittenabitmoreelegantlyusingthefollowingcode:for.else.py

classDriverException(Exception):

pass

people=[('James',17),('Kirk',9),('Lars',13),('Robert',8)]

forperson,ageinpeople:

ifage>=18:

driver=(person,age)

break

else:

raiseDriverException('Drivernotfound.')

Noticethatwearen’tforcedtousetheflagpatternanymore.Theexceptionisraisedaspartoftheforlooplogic,whichmakesgoodsensebecausetheforloopischeckingonsomecondition.Allweneedistosetupadriverobjectincasewefindone,becausetherestofthecodeisgoingtousethatinformationsomewhere.Noticethecodeisshorterandmoreelegant,becausethelogicisnowcorrectlygroupedtogetherwhereitbelongs.

PuttingthisalltogetherNowthatyouhaveseenallthereistoseeaboutconditionalsandloops,it’stimetospicethingsupalittle,andseethosetwoexamplesIanticipatedatthebeginningofthischapter.We’llmixandmatchhere,soyoucanseehowonecanusealltheseconceptstogether.Let’sstartbywritingsomecodetogeneratealistofprimenumbersuptosomelimit.PleasebearinmindthatI’mgoingtowriteaveryinefficientandrudimentaryalgorithmtodetectprimes.Theimportantthingforyouistoconcentrateonthosebitsinthecodethatbelongtothischapter’ssubject.

Example1–aprimegeneratorAccordingtoWikipedia:

“Aprimenumber(oraprime)isanaturalnumbergreaterthan1thathasnopositivedivisorsotherthan1anditself.Anaturalnumbergreaterthan1thatisnotaprimenumberiscalledacompositenumber.”

Basedonthisdefinition,ifweconsiderthefirst10naturalnumbers,wecanseethat2,3,5,and7areprimes,while1,4,6,8,9,10arenot.InordertohaveacomputertellyouifanumberNisprime,youcandividethatnumberbyallnaturalnumbersintherange[2,N).Ifanyofthosedivisionsyieldszeroasaremainder,thenthenumberisnotaprime.Enoughchatter,let’sgetdowntobusiness.I’llwritetwoversionsofthis,thesecondofwhichwillexploitthefor…elsesyntax.primes.py

primes=[]#thiswillcontaintheprimesintheend

upto=100#thelimit,inclusive

forninrange(2,upto+1):

is_prime=True#flag,newateachiterationofouterfor

fordivisorinrange(2,n):

ifn%divisor==0:

is_prime=False

break

ifis_prime:#checkonflag

primes.append(n)

print(primes)

Lotsofthingstonoticeintheprecedingcode.Firstofallwesetupanemptylistprimes,whichwillcontaintheprimesattheend.Thelimitis100,andyoucanseeit’sinclusiveinthewaywecallrange()intheouterloop.Ifwewroterange(2,upto)thatwouldbe[2,upto),right?Thereforerange(2,upto+1)givesus[2,upto+1)==[2,upto].

So,twoforloops.Intheouteroneweloopoverthecandidateprimes,thatis,allnaturalnumbersfrom2toupto.Insideeachiterationofthisouterloopwesetupaflag(whichissettoTrueateachiteration),andthenstartdividingthecurrentnbyallnumbersfrom2ton–1.Ifwefindaproperdivisorforn,itmeansniscomposite,andthereforewesettheflagtoFalseandbreaktheloop.Noticethatwhenwebreaktheinnerone,theouteronekeepsongoingnormally.Thereasonwhywebreakafterhavingfoundaproperdivisorfornisthatwedon’tneedanyfurtherinformationtobeabletotellthatnisnotaprime.

Whenwecheckontheis_primeflag,ifitisstillTrue,itmeanswecouldn’tfindanynumberin[2,n)thatisaproperdivisorforn,thereforenisaprime.Weappendntotheprimeslist,andhop!Anotheriteration,untilnequals100.

Runningthiscodeyields:

$pythonprimes.py

[2,3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59,61,67,

71,73,79,83,89,97]

Beforeweproceed,onequestion:ofalliterationsoftheouterloop,oneofthemisdifferentthanalltheothers.Couldyoutellwhichone,andwhy?Thinkaboutitforasecond,gobacktothecodeandtrytofigureitoutforyourself,andthenkeepreadingon.

Didyoufigureitout?Ifnot,don’tfeelbad,it’sperfectlynormal.Iaskedyoutodoitasasmallexercisebecauseit’swhatcodersdoallthetime.Theskilltounderstandwhatthecodedoesbysimplylookingatitissomethingyoubuildovertime.It’sveryimportant,sotrytoexerciseitwheneveryoucan.I’lltellyoutheanswernow:theiterationthatbehavesdifferentlyfromallothersisthefirstone.Thereasonisbecauseinthefirstiteration,nis2.Thereforetheinnermostforloopwon’tevenrun,becauseit’saforloopwhichiteratesoverrange(2,2),andwhatisthatifnot[2,2)?Tryitoutforyourself,writeasimpleforloopwiththatiterable,putaprintinthebodysuite,andseeifanythinghappens(itwon’t…).

Now,fromanalgorithmicpointofviewthiscodeisinefficientsolet’satleastmakeitmorebeautiful:primes.else.py

primes=[]

upto=100

forninrange(2,upto+1):

fordivisorinrange(2,n):

ifn%divisor==0:

break

else:

primes.append(n)

print(primes)

Muchnicer,right?Theis_primeflagiscompletelygone,andweappendntotheprimeslistwhenweknowtheinnerforloophasn’tencounteredanybreakstatements.Seehowthecodelookscleanerandreadsbetter?

Example2–applyingdiscountsInthisexample,IwanttoshowyouatechniqueIlikealot.Inmanyprogramminglanguages,otherthantheif/elif/elseconstructs,inwhateverformorsyntaxtheymaycome,youcanfindanotherstatement,usuallycalledswitch/case,thatinPythonismissing.Itistheequivalentofacascadeofif/elif/…/elif/elseclauses,withasyntaxsimilartothis(warning!JavaScriptcode!):switch.js

switch(day_number){

case1:

case2:

case3:

case4:

case5:

day="Weekday";

break;

case6:

day="Saturday";

break;

case0:

day="Sunday";

break;

default:

day="";

alert(day_number+'isnotavaliddaynumber.')

}

Intheprecedingcode,weswitchonavariablecalledday_number.Thismeanswegetitsvalueandthenwedecidewhatcaseitfitsin(ifany).From1to5thereisacascade,whichmeansnomatterthenumber,[1,5]allgodowntothebitoflogicthatsetsdayas"Weekday".Thenwehavesinglecasesfor0and6andadefaultcasetopreventerrors,whichalertsthesystemthatday_numberisnotavaliddaynumber,thatis,notin[0,6].Pythonisperfectlycapableofrealizingsuchlogicusingif/elif/elsestatements:switch.py

if1<=day_number<=5:

day='Weekday'

elifday_number==6:

day='Saturday'

elifday_number==0:

day='Sunday'

else:

day=''

raiseValueError(

str(day_number)+'isnotavaliddaynumber.')

Intheprecedingcode,wereproducethesamelogicoftheJavaScriptsnippet,inPython,usingif/elif/elsestatements.IraisedValueErrorexceptionjustasanexampleattheend,ifday_numberisnotin[0,6].Thisisonepossiblewayoftranslatingtheswitch/caselogic,butthereisalsoanotherone,sometimescalleddispatching,whichIwillshowyou

inthelastversionofthenextexample.

TipBytheway,didyounoticethefirstlineoftheprevioussnippet?HaveyounoticedthatPythoncanmakedouble(actually,evenmultiple)comparisons?It’sjustwonderful!

Let’sstartthenewexamplebysimplywritingsomecodethatassignsadiscounttocustomersbasedontheircouponvalue.I’llkeepthelogicdowntoaminimumhere,rememberthatallwereallycareaboutisconditionalsandloops.coupons.py

customers=[

dict(id=1,total=200,coupon_code='F20'),#F20:fixed,£20

dict(id=2,total=150,coupon_code='P30'),#P30:percent,30%

dict(id=3,total=100,coupon_code='P50'),#P50:percent,50%

dict(id=4,total=110,coupon_code='F15'),#F15:fixed,£15

]

forcustomerincustomers:

code=customer['coupon_code']

ifcode=='F20':

customer['discount']=20.0

elifcode=='F15':

customer['discount']=15.0

elifcode=='P30':

customer['discount']=customer['total']*0.3

elifcode=='P50':

customer['discount']=customer['total']*0.5

else:

customer['discount']=0.0

forcustomerincustomers:

print(customer['id'],customer['total'],customer['discount'])

Westartbysettingupsomecustomers.Theyhaveanordertotal,acouponcode,andanid.Imadeupfourdifferenttypesofcoupon,twoarefixedandtwoarepercentagebased.Youcanseethatintheif/elif/elsecascadeIapplythediscountaccordingly,andIsetitasa'discount'keyinthecustomerdict.

AttheendIjustprintoutpartofthedatatoseeifmycodeisworkingproperly.

$pythoncoupons.py

120020.0

215045.0

310050.0

411015.0

Thiscodeissimpletounderstand,butallthoseclausesarekindofclutteringthelogic.It’snoteasytoseewhat’sgoingonatafirstglance,andIdon’tlikeit.Incaseslikethis,youcanexploitadictionarytoyouradvantage,likethis:coupons.dict.py

customers=[

dict(id=1,total=200,coupon_code='F20'),#F20:fixed,£20

dict(id=2,total=150,coupon_code='P30'),#P30:percent,30%

dict(id=3,total=100,coupon_code='P50'),#P50:percent,50%

dict(id=4,total=110,coupon_code='F15'),#F15:fixed,£15

]

discounts={

'F20':(0.0,20.0),#eachvalueis(percent,fixed)

'P30':(0.3,0.0),

'P50':(0.5,0.0),

'F15':(0.0,15.0),

}

forcustomerincustomers:

code=customer['coupon_code']

percent,fixed=discounts.get(code,(0.0,0.0))

customer['discount']=percent*customer['total']+fixed

forcustomerincustomers:

print(customer['id'],customer['total'],customer['discount'])

Runningtheprecedingcodeyieldsexactlythesameresultwehadfromthesnippetbeforeit.Wesparedtwolines,butmoreimportantly,wegainedalotinreadability,asthebodyoftheforloopnowisjustthreelineslong,andveryeasytounderstand.Theconcepthereistouseadictionaryasdispatcher.Inotherwords,wetrytofetchsomethingfromthedictionarybasedonacode(ourcoupon_code),andbyusingdict.get(key,default),wemakesurewealsocaterforwhenthecodeisnotinthedictionaryandweneedadefaultvalue.

NoticethatIhadtoapplysomeverysimplelinearalgebrainordertocalculatethediscountproperly.Eachdiscounthasapercentageandfixedpartinthedictionary,representedbya2-tuple.Byapplyingpercent*total+fixed,wegetthecorrectdiscount.Whenpercentis0,theformulajustgivesthefixedamount,anditgivespercent*totalwhenfixedis0.Simplebuteffective.

Thistechniqueisimportantbecauseitisalsousedinothercontexts,withfunctions,whereitactuallybecomesmuchmorepowerfulthanwhatwe’veseenintheprecedingsnippet.Ifit’snotcompletelycleartoyouhowitworks,Isuggestyoutotakeyourtimeandexperimentwithit.Changevaluesandaddprintstatementstoseewhat’sgoingonwhiletheprogramisrunning.

AquickpeekattheitertoolsmoduleAchapteraboutiterables,iterators,conditionallogic,andloopingwouldn’tbecompletewithoutspendingafewwordsabouttheitertoolsmodule.Ifyouareintoiterating,thisisakindofheaven.

AccordingtothePythonofficialdocumentation,theitertoolsmoduleis:

“AmodulewhichimplementsanumberofiteratorbuildingblocksinspiredbyconstructsfromAPL,Haskell,andSML.EachhasbeenrecastinaformsuitableforPython.Themodulestandardizesacoresetoffast,memoryefficienttoolsthatareusefulbythemselvesorincombination.Together,theyforman“iteratoralgebra”makingitpossibletoconstructspecializedtoolssuccinctlyandefficientlyinpurePython.”

BynomeansdoIhavetheroomheretoshowyouallthegoodiesyoucanfindinthismodule,soIencourageyoutogoandcheckitoutforyourself,Ipromiseyou’llenjoyit.

Inanutshell,itprovidesyouwiththreebroadcategoriesofiterators.Iwillgiveyouaverysmallexampleofoneiteratortakenfromeachoneofthem,justtomakeyourmouthwateralittle.

InfiniteiteratorsInfiniteiteratorsallowyoutoworkwithaforloopinadifferentfashion,likeifitwasawhileloop.infinite.py

fromitertoolsimportcount

fornincount(5,3):

ifn>20:

break

print(n,end=',')#insteadofnewline,commaandspace

Runningthecodegivesthis:

$pythoninfinite.py

5,8,11,14,17,20,

Thecountfactoryclassmakesaniteratorthatjustgoesonandoncounting.Itstartsfrom5andkeepsadding3toit.Weneedtomanuallybreakitifwedon’twanttogetstuckinaninfiniteloop.

IteratorsterminatingontheshortestinputsequenceThiscategoryisveryinteresting.Itallowsyoutocreateaniteratorbasedonmultipleiterators,combiningtheirvaluesaccordingtosomelogic.Thekeypointhereisthatamongthoseiterators,incaseanyofthemareshorterthantherest,theresultingiteratorwon’tbreak,itwillsimplystopassoonastheshortestiteratorisexhausted.Thisisverytheoretical,Iknow,soletmegiveyouanexampleusingcompress.ThisiteratorgivesyoubackthedataaccordingtoacorrespondingiteminaselectorbeingTrueorFalse:

compress('ABC',(1,0,1))wouldgiveback'A'and'C',becausetheycorrespondtothe1's.Let’sseeasimpleexample:compress.py

fromitertoolsimportcompress

data=range(10)

even_selector=[1,0]*10

odd_selector=[0,1]*10

even_numbers=list(compress(data,even_selector))

odd_numbers=list(compress(data,odd_selector))

print(odd_selector)

print(list(data))

print(even_numbers)

print(odd_numbers)

Noticethatodd_selectorandeven_selectorare20elementslong,whiledataisjust10elementslong.compresswillstopassoonasdatahasyieldeditslastelement.Runningthiscodeproducesthefollowing:

$pythoncompress.py

[0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1]

[0,1,2,3,4,5,6,7,8,9]

[0,2,4,6,8]

[1,3,5,7,9]

It’saveryfastandnicewayofselectingelementsoutofaniterable.Thecodeisverysimple,justnoticethatinsteadofusingaforlooptoiterateovereachvaluethatisgivenbackbythecompresscalls,weusedlist(),whichdoesthesame,butinsteadofexecutingabodyofinstructions,putsallthevaluesintoalistandreturnsit.

CombinatoricgeneratorsLastbutnotleast,combinatoricgenerators.Thesearereallyfun,ifyouareintothiskindofthing.Let’sjustseeasimpleexampleonpermutations.

AccordingtoWolframMathworld:

“Apermutation,alsocalledan“arrangementnumber”or“order”,isarearrangementoftheelementsofanorderedlistSintoaone-to-onecorrespondencewithSitself.”

Forexample,thepermutationsofABCare6:ABC,ACB,BAC,BCA,CAB,andCBA.

IfasethasNelements,thenthenumberofpermutationsofthemisN!(Nfactorial).ForthestringABCthepermutationsare3!=3*2*1=6.Let’sdoitinPython:permutations.py

fromitertoolsimportpermutations

print(list(permutations('ABC')))

Thisveryshortsnippetofcodeproducesthefollowingresult:

$pythonpermutations.py

[('A','B','C'),('A','C','B'),('B','A','C'),('B','C','A'),('C',

'A','B'),('C','B','A')]

Beverycarefulwhenyouplaywithpermutation.Theirnumbergrowsataratethatisproportionaltothefactorialofthenumberoftheelementsyou’repermuting,andthatnumbercangetreallybig,reallyfast.

SummaryInthischapter,we’vetakenanotherstepforwardtoexpandourcodingvocabulary.We’veseenhowtodrivetheexecutionofthecodebyevaluatingconditions,andwe’veseenhowtoloopanditerateoversequencesandcollectionsofobjects.Thisgivesusthepowertocontrolwhathappenswhenourcodeisrun,whichmeanswearegettinganideaonhowtoshapeitsothatitdoeswhatwewantanditreactstodatathatchangesdynamically.

We’vealsoseenhowtocombineeverythingtogetherinacoupleofsimpleexamples,andintheendwehavetakenabrieflookattheitertoolsmodule,whichisfullofinterestingiteratorswhichcanenrichourabilitieswithPythonevenmore.

Nowit’stimetoswitchgears,totakeanotherstepforwardandtalkaboutfunctions.Thenextchapterisallaboutthembecausetheyareextremelyimportant.Makesureyou’recomfortablewithwhathasbeendoneuptonow:Iwanttoprovideyouwithinterestingexamples,soI’llhavetogoalittlefaster.Ready?Turnthepage.

Chapter4.Functions,theBuildingBlocksofCode “Tocreatearchitectureistoputinorder.Putwhatinorder?Functionandobjects.”

—LeCorbusier

Inthischapter,we’regoingtoexplorefunctions.WealreadysaidthateverythingisanobjectinPython,andfunctionsarenoexceptiontothis.But,whatexactlyisafunction?Afunctionisasequenceofinstructionsthatperformatask,bundledasaunit.Thisunitcanthenbeimportedandusedwhereverit’sneeded.Therearemanyadvantagestousingfunctionsinyourcode,aswe’llseeshortly.

Ibelievethesaying,apictureisworthonethousandwords,isparticularlytruewhenexplainingfunctionstosomeonewhoisnewtothisconcept,sopleasetakealookatthefollowingimage:

Asyoucansee,afunctionisablockofinstructions,packagedasawhole,likeabox.Functionscanacceptinputargumentsandproduceoutputvalues.Bothoftheseareoptional,aswe’llseeintheexamplesinthischapter.

AfunctioninPythonisdefinedbyusingthedefkeyword,afterwhichthenameofthefunctionfollows,terminatedbyapairofbraces(whichmayormaynotcontaininputparameters)and,finally,acolon(:)signalstheendofthefunctiondefinitionline.Immediatelyafterwards,indentedbyfourspaces,wefindthebodyofthefunction,whichisthesetofinstructionsthatthefunctionwillexecutewhencalled.

NoteNotethattheindentationbyfourspacesisnotmandatory,butitistheamountofspacessuggestedbyPEP8,and,inpractice,itisthemostwidelyusedspacingmeasure.

Afunctionmayormaynotreturnoutput.Ifafunctionwantstoreturnoutput,itdoessobyusingthereturnkeyword,followedbythedesiredoutput.Ifyouhaveaneagleeye,youmayhavenoticedthelittle*afterOptionalintheoutputsectionoftheprecedingpicture.ThisisbecauseafunctionalwaysreturnssomethinginPython,evenifyoudon’texplicitlyusethereturnclause.Ifthefunctionhasnoreturnstatementinitsbody,it’sreturnvalue

isNone.Thereasonsbehindthisdesignchoiceareoutofthescopeofanintroductorychapter,soallyouneedtoknowisthatthisbehaviorwillmakeyourlifeeasier,asalways,thankyouPython.

Whyusefunctions?Functionsareamongthemostimportantconceptsandconstructsofanylanguage,soletmegiveyouafewreasonswhyweneedthem:

Theyreducecodeduplicationinaprogram.Byhavingaspecifictasktakencareofbyaniceblockofpackagedcodethatwecanimportandcallwheneverwewant,wedon’tneedtoduplicateitsimplementation.Theyhelpinsplittingacomplextaskorprocedureintosmallerblocks,eachofwhichbecomesafunction.Theyhidetheimplementationdetailsfromtheirusers.Theyimprovetraceability.Theyimprovereadability.

Let’slookatafewexamplestogetabetterunderstandingofeachpoint.

ReducecodeduplicationImaginethatyouarewritingapieceofscientificsoftware,andyouneedtocalculateprimesuptoalimit,aswedidinthepreviouschapter.Youwriteseveralalgorithmsandprimenumbers,beingthebasisofmanydifferenttypesofcalculations,keepcreepingintoyourcode.Well,youhaveanicealgorithmtocalculatethem,soyoucopyandpasteittowhereveryouneed.Oneday,though,yourfriendMisterSmartygivesyouabetteralgorithmtocalculateprimenumbers,andthiswillsaveyoualotoftime.Atthispoint,youneedtogooveryourwholecodebaseandreplacetheoldcodewiththenewcode.

Thisisactuallyaverybadwaytogoaboutit.It’serror-prone,youneverknowwhatlinesyouarechoppingoutorleavingtherebymistakewhenyoucutandpastecodeinothercode,andyoumayalsoriskmissingoneoftheplaceswhereprimecalculationwasdone,leavingyoursoftwarewithdifferentversions.Canyouimagineifyoudiscoveredthattheoldwaywasbuggy?Youwouldhaveanundetectedbuginyourcode,andbugslikethisarequitehardtospot,especiallyinbigcodebases.

So,whatshouldyoudo?Simple!Youwriteafunction,get_prime_numbers(upto),anduseitanywhereyouneedalistofprimes.WhenMisterSmartycomestoyouandgivesyouthenewcode,allyouhavetodoisreplacethebodyofthatfunctionwiththenewimplementation,andyou’redone!Therestofthesoftwarewillautomaticallyadapt,sinceit’sjustcallingthefunction.

Yourcodewillbeshorter,itwillnotsufferfrominconsistenciesbetweenoldandnewwaysofperformingatask,orundetectedbugsduetocopyandpastefailuresoroversights.Usefunctions,andyou’llonlygainfromit,Ipromise.

SplittingacomplextaskFunctionsareveryusefulalsotosplitalongorcomplextaskintosmallerpieces.Theendresultisthatthecodebenefitsfromitinseveralways,forexample,readability,testability,andreuse.Togiveyouasimpleexample,imaginethatyou’repreparingareport.Yourcodeneedstofetchdatafromadatasource,parseit,filterit,polishit,andthenawholeseriesofalgorithmsneedstoberunagainstit,inordertoproducetheresultswhichwillfeedtheReportclass.It’snotuncommontoreadprocedureslikethisthatarejustonebigfunctiondo_report(data_source).Therearetensorhundredsoflinesofcodewhichendwithreturnreport.

Situationslikethisarecommonincodeproducedbyscientists.Theyhavebrilliantmindsandtheycareaboutthecorrectnessoftheendresultbut,unfortunately,sometimestheyhavenotraininginprogrammingtheory.Itisnottheirfault,onecannotknoweverything.Now,pictureinyourheadsomethinglikeafewhundredlinesofcode.It’sveryhardtofollowthrough,tofindtheplaceswherethingsarechangingcontext(likefinishingonetaskandstartingthenextone).Doyouhavethepictureinyourmind?Good.Don’tdoit!Instead,lookatthiscode:data.science.example.py

defdo_report(data_source):

#fetchandpreparedata

data=fetch_data(data_source)

parsed_data=parse_data(data)

filtered_data=filter_data(parsed_data)

polished_data=polish_data(filtered_data)

#runalgorithmsondata

final_data=analyse(polished_data)

#createandreturnreport

report=Report(final_data)

returnreport

Thepreviousexampleisfictitious,ofcourse,butcanyouseehoweasyitwouldbetogothroughthecode?Iftheendresultlookswrong,itwouldbeveryeasytodebugeachofthesingledataoutputsinthedo_reportfunction.Moreover,it’seveneasiertoexcludepartoftheprocesstemporarilyfromthewholeprocedure(youjustneedtocommentoutthepartsyouneedtosuspend).Codelikethisiseasiertodealwith.

HideimplementationdetailsLet’sstaywiththeprecedingexampletotalkaboutthispointaswell.Youcanseethat,bygoingthroughthecodeofthedo_reportfunction,youcangetaprettygoodunderstandingwithoutreadingonesinglelineofimplementation.Thisisbecausefunctionshidetheimplementationdetails.Thisfeaturemeansthat,ifyoudon’tneedtodelveintodetails,youarenotforcedto,inthewayyouwouldifdo_reportwasjustonebigfatfunction.Inordertounderstandwhatwasgoingon,youwouldhavetoreadtheimplementationdetails.Youdon’tneedtowithfunctions.Thisreducesthetimeyouspendreadingthecodeandsince,inaprofessionalenvironment,readingcodetakesmuchmoretimethanactuallywritingit,it’sveryimportanttoreduceitasmuchaswecan.

ImprovereadabilityCoderssometimesdon’tseethepointinwritingafunctionwithabodyofoneortwolinesofcode,solet’slookatanexamplethatshowsyouwhyyoushoulddoit.

Imaginethatyouneedtomultiplytwomatrices:

Wouldyouprefertohavetoreadthiscode:matrix.multiplication.nofunc.py

a=[[1,2],[3,4]]

b=[[5,1],[2,1]]

c=[[sum(i*jfori,jinzip(r,c))forcinzip(*b)]

forrina]

Orwouldyoupreferthisone:matrix.multiplication.func.py

#thisfunctioncouldalsobedefinedinanothermodule

defmatrix_mul(a,b):

return[[sum(i*jfori,jinzip(r,c))forcinzip(*b)]

forrina]

a=[[1,2],[3,4]]

b=[[5,1],[2,1]]

c=matrix_mul(a,b)

It’smucheasiertounderstandthatcistheresultofthemultiplicationbetweenaandbinthesecondexample.It’smucheasiertoreadthroughthecodeand,ifyoudon’tneedtomodifythatpart,youdon’tevenneedtogointotheimplementationdetails.

Therefore,readabilityisimprovedherewhile,inthefirstsnippet,youwouldhavetospendtimetryingtounderstandwhatthatcomplicatedlistcomprehensionwasdoing.

NoteDon’tworryifyoudon’tunderstandlistcomprehensions,we’llstudytheminthenextchapter.

ImprovetraceabilityImaginethatyouhavewrittenane-commercewebsite.Youhavedisplayedtheproductpricesalloverthepages.ImaginethatthepricesinyourdatabasearestoredwithnoVAT,butyouwanttodisplaythemonthewebsitewithVATat20%.Here’safewwaysofcalculatingtheVAT-inclusivepricefromtheVAT-exclusiveprice.vat.py

price=100#GBP,noVAT

final_price1=price*1.2

final_price2=price+price/5.0

final_price3=price*(100+20)/100.0

final_price4=price+price*0.2

AllthesefourdifferentwaysofcalculatingaVAT-inclusivepriceareperfectlyacceptable,andIpromiseyouIhavefoundthemallinmycolleagues’code,overtheyears.Now,imaginethatyouhavestartedsellingyourproductsindifferentcountriesandsomeofthemhavedifferentVATratessoyouneedtorefactoryourcode(throughoutthewebsite)inordertomakethatVATcalculationdynamic.

HowdoyoutracealltheplacesinwhichyouareperformingaVATcalculation?CodingtodayisacollaborativetaskandyoucannotbesuretheVAThasbeencalculatedusingonlyoneofthoseforms.It’sgoingtobehell,believeme.

So,let’swriteafunctionthattakestheinputvalues,vatandprice(VAT-exclusive),andreturnsaVAT-inclusiveprice.vat.function.py

defcalculate_price_with_vat(price,vat):

returnprice*(100+vat)/100

NowyoucanimportthatfunctionandapplyitinanyplaceofyourwebsitewhereyouneedtocalculateaVAT-inclusivepriceandwhenyouneedtotracethosecalls,youcansearchforcalculate_price_with_vat.

NoteNotethat,intheprecedingexample,priceisassumedtobeVAT-exclusive,andvathasapercentagevalue(forexample,19,20,23,andsoon).

ScopesandnameresolutionDoyourememberwhenwetalkedaboutscopesandnamespacesinthefirstchapter?We’regoingtoexpandonthatconceptnow.Finally,wecantalkaboutfunctionsandthiswillmakeeverythingeasiertounderstand.Let’sstartwithaverysimpleexample.scoping.level.1.py

defmy_function():

test=1#thisisdefinedinthelocalscopeofthefunction

print('my_function:',test)

test=0#thisisdefinedintheglobalscope

my_function()

print('global:',test)

Ihavedefinedthenametestintwodifferentplacesinthepreviousexample.Itisactuallyintwodifferentscopes.Oneistheglobalscope(test=0),andtheotheristhelocalscopeofthefunctionmy_function(test=1).Ifyouexecutethecode,you’llseethis:

$pythonscoping.level.1.py

my_function:1

global:0

It’sclearthattest=1shadowstheassignmenttest=0inmy_function.Intheglobalcontext,testisstill0,asyoucanseefromtheoutputoftheprogrambutwedefinethenametestagaininthefunctionbody,andwesetittopointtoanintegerofvalue1.Boththetwotestnamesthereforeexist,oneintheglobalscope,pointingtoanintobjectwithvalue0,theotherinthemy_functionscope,pointingtoanintobjectwithvalue1.Let’scommentoutthelinewithtest=1.Pythongoesandsearchesforthenametestinthenextenclosingnamespace(recalltheLEGBrule:Local,Enclosing,Global,Built-indescribedinChapter1,IntroductionandFirstSteps–TakeaDeepBreath)and,inthiscase,wewillseethevalue0printedtwice.Tryitinyourcode.

Now,let’sraisethestakeshereandlevelup:scoping.level.2.py

defouter():

test=1#outerscope

definner():

test=2#innerscope

print('inner:',test)

inner()

print('outer:',test)

test=0#globalscope

outer()

print('global:',test)

Intheprecedingcode,wehavetwolevelsofshadowing.Onelevelisinthefunctionouter,andtheotheroneisinthefunctioninner.Itisfarfromrocketscience,butitcanbetricky.Ifwerunthecode,weget:

$pythonscoping.level.2.py

inner:2

outer:1

global:0

Trycommentingoutthelinetest=1.Whatdoyouthinktheresultwillbe?Well,whenreachingthelineprint('outer:',test),Pythonwillhavetolookfortestinthenextenclosingscope,thereforeitwillfindandprint0,insteadof1.Makesureyoucommentouttest=2aswell,toseeifyouunderstandwhathappens,andiftheLEGBruleisclear,beforeproceeding.

AnotherthingtonoteisthatPythongivesyoutheabilitytodefineafunctioninanotherfunction.Theinnerfunction’snameisdefinedwithinthenamespaceoftheouterfunction,exactlyaswouldhappenwithanyothername.

TheglobalandnonlocalstatementsGoingbacktotheprecedingexample,wecanalterwhathappenstotheshadowingofthetestnamebyusingoneofthesetwospecialstatements:globalandnonlocal.Asyoucanseefromthepreviousexample,whenwedefinetest=2inthefunctioninner,weoverwritetestneitherinthefunctionouter,norintheglobalscope.Wecangetreadaccesstothosenamesifweusetheminanestedscopethatdoesn’tdefinethem,butwecannotmodifythembecause,whenwewriteanassignmentinstruction,we’reactuallydefininganewnameinthecurrentscope.

Howdowechangethisbehavior?Well,wecanusethenonlocalstatement.Accordingtotheofficialdocumentation:

“Thenonlocalstatementcausesthelistedidentifierstorefertopreviouslyboundvariablesinthenearestenclosingscopeexcludingglobals.”

Let’sintroduceitinthefunctioninner,andseewhathappens:scoping.level.2.nonlocal.py

defouter():

test=1#outerscope

definner():

nonlocaltest

test=2#nearestenclosingscope

print('inner:',test)

inner()

print('outer:',test)

test=0#globalscope

outer()

print('global:',test)

NoticehowinthebodyofthefunctioninnerIhavedeclaredthetestnametobenonlocal.Runningthiscodeproducesthefollowingresult:

$pythonscoping.level.2.nonlocal.py

inner:2

outer:2

global:0

Wow,lookatthatresult!Itmeansthat,bydeclaringtesttobenonlocalinthefunctioninner,weactuallygettobindthenametesttothatdeclaredinthefunctionouter.Ifweremovedthenonlocaltestlinefromthefunctioninnerandtriedthesametrickinthefunctionouter,wewouldgetaSyntaxError,becausethenonlocalstatementworksonenclosingscopesexcludingtheglobalone.

Isthereawaytogettothattest=0intheglobalnamespacethen?Ofcourse,wejustneedtousetheglobalstatement.Let’stryit.scoping.level.2.global.py

defouter():

test=1#outerscope

definner():

globaltest

test=2#globalscope

print('inner:',test)

inner()

print('outer:',test)

test=0#globalscope

outer()

print('global:',test)

Notethatwehavenowdeclaredthenametesttobeglobal,whichwillbasicallybindittotheonewedefinedintheglobalnamespace(test=0).Runthecodeandyoushouldgetthefollowing:

$pythonscoping.level.2.global.py

inner:2

outer:1

global:2

Thisshowsthatthenameaffectedbytheassignmenttest=2isnowtheglobalone.Thistrickwouldalsoworkintheouterfunctionbecause,inthiscase,we’rereferringtotheglobalscope.Tryitforyourselfandseewhatchanges,getcomfortablewithscopesandnameresolution,it’sveryimportant.

InputparametersAtthebeginningofthischapter,wesawthatafunctioncantakeinputparameters.Beforewedelveintoallpossibletypeofparameters,let’smakesureyouhaveaclearunderstandingofwhatpassingaparametertoafunctionmeans.Therearethreekeypointstokeepinmind:

ArgumentpassingisnothingmorethanassigninganobjecttoalocalvariablenameAssigninganobjecttoanargumentnameinsideafunctiondoesn’taffectthecallerChangingamutableobjectargumentinafunctionaffectsthecaller

Let’slookatanexampleforeachofthesepoints.

ArgumentpassingTakealookatthefollowingcode.Wedeclareanamexintheglobalscope,thenwedeclareafunctionfunc(y)andwecallit,passingx.Ihighlightedthecallinthecode.key.points.argument.passing.py

x=3

deffunc(y):

print(y)

func(x)#prints:3

Whenfunciscalledwithx,whathappensisthatwithinitslocalscope,anameyiscreated,andit’spointedtothesameobjectxispointingto.Thisisbetterclarifiedbythefollowingpicture:

Therightpartoftheprecedingpicturedepictsthestateoftheprogramwhenexecutionhasreachedtheend,afterfunchasreturned(None).TakealookattheFramescolumn,andnotethatwehavetwonames,xandfunc,intheglobalnamespace(Globalframe),pointingtoanint(withavalueofthree)andtoafunctionobject,respectively.Rightbelowit,intherectangletitledfunc,wecanseethefunction’slocalnamespace,inwhichonlyonenamehasbeendefined:y.Becausewehavecalledfuncwithx(line5intheleftpartofthepicture),yispointingtothesameobjectthatxispointingto.Thisiswhathappensunderthehoodwhenanargumentispassedtoafunction.Ifwehadusedthenamexinsteadofyinthefunctiondefinition,thingswouldhavebeenexactlythesame(onlymaybeabitconfusingatfirst),therewouldbealocalxinthefunction,andaglobalxoutside,aswesawintheScopesandnameresolutionsection.

So,inanutshell,whatreallyhappensisthatthefunctioncreatesinitslocalscopethenamesdefinedasargumentsand,whenwecallit,webasicallytellPythonwhichobjectsthosenamesmustbepointedtowards.

Assignmenttoargumentnamesdon’taffectthecallerThisissomethingthatcanbetrickytounderstandatfirst,solet’slookatanexample.key.points.assignment.py

x=3

deffunc(x):

x=7#definingalocalx,notchangingtheglobalone

func(x)

print(x)#prints:3

Intheprecedingcode,whenthelinex=7isexecuted,whathappensisthatwithinthelocalscopeofthefunctionfunc,thenamexispointedtoanintegerwithvalue7,leavingtheglobalxunaltered.

ChangingamutableaffectsthecallerThisisthefinalpoint,andit’sveryimportantbecausePythonapparentlybehavesdifferentlywithmutables(justapparentlythough).Let’slookatanexample:key.points.mutable.py

x=[1,2,3]

deffunc(x):

x[1]=42#thisaffectsthecaller!

func(x)

print(x)#prints:[1,42,3]

Wow,weactuallychangedtheoriginalobject!Ifyouthinkaboutit,thereisnothingweirdinthisbehavior.Thenamexinthefunctionissettopointtothecallerobjectbythefunctioncallandwithinthebodyofthefunction,we’renotchangingx,inthatwe’renotchangingitsreference,or,inotherwords,wearenotchangingtheobjectxispointingto.Whatwe’redoingisaccessingthatobject’selementatposition1,andchangingitsvalue.

Rememberpoint#2:“Assigninganobjecttoanargumentnamewithinafunctiondoesn’taffectthecaller“.Ifthatiscleartoyou,thefollowingcodeshouldnotbesurprising.key.points.mutable.assignment.py

x=[1,2,3]

deffunc(x):

x[1]=42#thischangesthecaller!

x='somethingelse'#thispointsxtoanewstringobject

func(x)

print(x)#stillprints:[1,42,3]

TakealookatthetwolinesIhavehighlighted.Atfirst,wejustaccessthecallerobjectagain,atposition1,andchangeitsvaluetonumber42.Then,wereassignxtopointtothestring'somethingelse'.Thisleavesthecallerunaltered,accordingtopoint#2,and,infact,theoutputisthesameasthatoftheprevioussnippet.

Takeyourtimetoplayaroundwiththisconceptandexperimentwithprintsandcallstotheidfunctionuntileverythingisclearinyourmind.ThisisoneofthekeyaspectsofPythonanditmustbeveryclear,otherwiseyouriskintroducingsubtlebugsintoyourcode.

Nowthatwehaveagoodunderstandingofinputparametersandhowtheybehave,let’sseehowwecanspecifythem.

HowtospecifyinputparametersTherearefivedifferentwaysofspecifyinginputparameters.Let’slookatthemonebyone.

PositionalargumentsPositionalargumentsarereadfromlefttorightandtheyarethemostcommontypeofarguments.arguments.positional.py

deffunc(a,b,c):

print(a,b,c)

func(1,2,3)#prints:123

Thereisnotmuchelsetosay.Theycanbeasnumerousasyouwantandtheyareassignedbyposition.Inthefunctioncall,1comesfirst,2comessecondand3comesthird,thereforetheyareassignedtoa,bandcrespectively.

KeywordargumentsanddefaultvaluesKeywordargumentsareassignedbykeywordusingthename=valuesyntax.arguments.keyword.py

deffunc(a,b,c):

print(a,b,c)

func(a=1,c=2,b=3)#prints:132

Keywordargumentsactwhencallingthefunctioninsteadofrespectingtheleft-to-rightpositionalassignment,k.Keywordargumentsarematchedbyname,evenwhentheydon’trespectthedefinition’soriginalposition(we’llseethatthereisalimitationtothisbehaviorlater,whenwemixandmatchdifferenttypesofarguments).

Thecounterpartofkeywordarguments,onthedefinitionside,isdefaultvalues.Thesyntaxisthesame,name=value,andallowsustonothavetoprovideanargumentifwearehappywiththegivendefault.arguments.default.py

deffunc(a,b=4,c=88):

print(a,b,c)

func(1)#prints:1488

func(b=5,a=7,c=9)#prints:759

func(42,c=9)#prints:4249

Thearetwothingstonotice,whichareveryimportant.Firstofall,youcannotspecifyadefaultargumentontheleftofapositionalone.Second,notehowintheexamples,whenanargumentispassedwithoutusingtheargument_name=valuesyntax,itmustbethefirstoneinthelist,,anditisalwaysassignedtoa.Tryandscramblethoseargumentsandseewhathappens.Pythonerrormessagesareverygoodattellingyouwhat’swrong.So,forexample,ifyoutriedsomethinglikethis:

func(b=1,c=2,42)#positionalargumentafterkeywordone

Youwouldgetthefollowingerror:

SyntaxError:non-keywordargafterkeywordarg

Thisinformsyouthatyou’vecalledthefunctionincorrectly.

VariablepositionalargumentsSometimesyoumaywanttopassavariablenumberofpositionalargumentstoafunctionandPythonprovidesyouwiththeabilitytodoit.Let’slookataverycommonusecase,theminimumfunction.Thisisafunctionthatcalculatestheminimumofitsinputvalues.arguments.variable.positional.py

defminimum(*n):

#print(n)#nisatuple

ifn:#explainedafterthecode

mn=n[0]

forvalueinn[1:]:

ifvalue<mn:

mn=value

print(mn)

minimum(1,3,-7,9)#n=(1,3,-7,9)-prints:-7

minimum()#n=()-prints:nothing

Asyoucansee,whenwespecifyaparameterprependinga*toitsname,wearetellingPythonthatthatparameterwillbecollectingavariablenumberofpositionalarguments,accordingtohowthefunctioniscalled.Withinthefunction,nisatuple.Uncommenttheprint(n)toseeforyourselfandplayaroundwithitforabit.

NoteHaveyounoticedhowwecheckedifnwasn’temptywithasimpleifn:?ThisisduetothefactthatcollectionobjectsevaluatetoTruewhennon-empty,andotherwiseFalseinPython.Thisistruefortuples,sets,lists,dictionaries,andsoon.

Oneotherthingtonoteisthatwemaywanttothrowanerrorwhenwecallthefunctionwithnoarguments,insteadofsilentlydoingnothing.Inthiscontext,we’renotconcernedaboutmakingthisfunctionrobust,butinunderstandingvariablepositionalarguments.

Let’smakeanotherexampletoshowyoutwothingsthat,inmyexperience,areconfusingtothosewhoarenewtothis.arguments.variable.positional.unpacking.py

deffunc(*args):

print(args)

values=(1,3,-7,9)

func(values)#equivalentto:func((1,3,-7,9))

func(*values)#equivalentto:func(1,3,-7,9)

Takeagoodlookatthelasttwolinesoftheprecedingexample.Inthefirstone,wecall

funcwithoneargument,afourelementstuple.Inthesecondexample,byusingthe*syntax,we’redoingsomethingcalledunpacking,whichmeansthatthefourelementstupleisunpacked,andthefunctioniscalledwithfourarguments:1,3,-7,9.

ThisbehaviorispartofthemagicPythondoestoallowyoutodoamazingthingswhencallingfunctionsdynamically.

VariablekeywordargumentsVariablekeywordargumentsareverysimilartovariablepositionalarguments.Theonlydifferenceisthesyntax(**insteadof*)andthattheyarecollectedinadictionary.Collectionandunpackingworkinthesameway,solet’slookatanexample:arguments.variable.keyword.py

deffunc(**kwargs):

print(kwargs)

#Allcallsequivalent.Theyprint:{'a':1,'b':42}

func(a=1,b=42)

func(**{'a':1,'b':42})

func(**dict(a=1,b=42))

Allthecallsareequivalentintheprecedingexample.Youcanseethataddinga**infrontoftheparameternameinthefunctiondefinitiontellsPythontousethatnametocollectavariablenumberofkeywordparameters.Ontheotherhand,whenwecallthefunction,wecaneitherpassname=valueargumentsexplicitly,orunpackadictionaryusingthesame**syntax.

Thereasonwhybeingabletopassavariablenumberofkeywordparametersissoimportantmaynotbeevidentatthemoment,so,howaboutamorerealisticexample?Let’sdefineafunctionthatconnectstoadatabase.Wewanttoconnecttoadefaultdatabasebysimplycallingthisfunctionwithnoparameters.Wealsowanttoconnecttoanyotherdatabasebypassingthefunctiontheappropriatearguments.Beforeyoureadon,spendacoupleofminutesfiguringoutasolutionbyyourself.arguments.variable.db.py

defconnect(**options):

conn_params={

'host':options.get('host','127.0.0.1'),

'port':options.get('port',5432),

'user':options.get('user',''),

'pwd':options.get('pwd',''),

}

print(conn_params)

#wethenconnecttothedb(commentedout)

#db.connect(**conn_params)

connect()

connect(host='127.0.0.42',port=5433)

connect(port=5431,user='fab',pwd='gandalf')

Noteinthefunctionwecanprepareadictionaryofconnectionparameters(conn_params)inthefunctionusingdefaultvaluesasfallback,allowingthemtobeoverwritteniftheyare

providedinthefunctioncall.Therearebetterwaystodothiswithfewerlinesofcodebutwe’renotconcernedwiththatnow.Runningtheprecedingcodeyieldsthefollowingresult:

$pythonarguments.variable.db.py

{'host':'127.0.0.1','pwd':'','user':'','port':5432}

{'host':'127.0.0.42','pwd':'','user':'','port':5433}

{'host':'127.0.0.1','pwd':'gandalf','user':'fab','port':5431}

Notethecorrespondencebetweenthefunctioncallsandtheoutput.Notehowdefaultvaluesareeitherthereoroverridden,accordingtowhatwaspassedtothefunction.

Keyword-onlyargumentsPython3allowsforanewtypeofparameter:thekeyword-onlyparameter.Wearegoingtostudythemonlybrieflyastheirusecasesarenotthatfrequent.Therearetwowaysofspecifyingthem,eitherafterthevariablepositionalarguments,orafterabare*.Let’sseeanexampleofboth.arguments.keyword.only.py

defkwo(*a,c):

print(a,c)

kwo(1,2,3,c=7)#prints:(1,2,3)7

kwo(c=4)#prints:()4

#kwo(1,2)#breaks,invalidsyntax,withthefollowingerror

#TypeError:kwo()missing1requiredkeyword-onlyargument:'c'

defkwo2(a,b=42,*,c):

print(a,b,c)

kwo2(3,b=7,c=99)#prints:3799

kwo2(3,c=13)#prints:34213

#kwo2(3,23)#breaks,invalidsyntax,withthefollowingerror

#TypeError:kwo2()missing1requiredkeyword-onlyargument:'c'

Asanticipated,thefunction,kwo,takesavariablenumberofpositionalarguments(a)andakeyword-onlyfunction,c.TheresultsofthecallsarestraightforwardandyoucanuncommentthethirdcalltoseewhaterrorPythonreturns.

Thesameappliestothefunction,kwo2,whichdiffersfromkwointhatittakesapositionalargumenta,akeywordargumentb,andthenakeyword-onlyargument,c.Youcanuncommentthethirdcalltoseetheerror.

Nowthatyouknowhowtospecifydifferenttypesofinputparameters,let’sseehowyoucancombinetheminfunctiondefinitions.

CombininginputparametersYoucancombineinputparameters,aslongasyoufollowtheseorderingrules:

Whendefiningafunction,normalpositionalargumentscomefirst(name),thenanydefaultarguments(name=value),thenthevariablepositionalarguments(*name,orsimply*),thenanykeyword-onlyarguments(eithernameorname=valueformis

good),thenanyvariablekeywordarguments(**name).Ontheotherhand,whencallingafunction,argumentsmustbegiveninthefollowingorder:positionalargumentsfirst(value),thenanycombinationofkeywordarguments(name=value),variablepositionalarguments(*name),thenvariablekeywordarguments(**name).

Sincethiscanbeabittrickywhenlefthanginginthetheoreticalworld,let’slookatacoupleofquickexamples.arguments.all.py

deffunc(a,b,c=7,*args,**kwargs):

print('a,b,c:',a,b,c)

print('args:',args)

print('kwargs:',kwargs)

func(1,2,3,*(5,7,9),**{'A':'a','B':'b'})

func(1,2,3,5,7,9,A='a',B='b')#sameaspreviousone

Notetheorderoftheparametersinthefunctiondefinition,andthatthetwocallsareequivalent.Inthefirstone,we’reusingtheunpackingoperatorsforiterablesanddictionaries,whileinthesecondonewe’reusingamoreexplicitsyntax.Theexecutionofthisyields(Iprintedonlytheresultofonecall):

$pythonarguments.all.py

a,b,c:123

args:(5,7,9)

kwargs:{'A':'a','B':'b'}

Let’snowlookatanexamplewithkeyword-onlyarguments.arguments.all.kwonly.py

deffunc_with_kwonly(a,b=42,*args,c,d=256,**kwargs):

print('a,b:',a,b)

print('c,d:',c,d)

print('args:',args)

print('kwargs:',kwargs)

#bothcallsequivalent

func_with_kwonly(3,42,c=0,d=1,*(7,9,11),e='E',f='F')

func_with_kwonly(3,42,*(7,9,11),c=0,d=1,e='E',f='F')

NotethatIhavehighlightedthekeyword-onlyargumentsinthefunctiondeclaration.Theycomeafterthevariablepositionalargument*args,anditwouldbethesameiftheycamerightafterasingle*(inwhichcasetherewouldn’tbeavariablepositionalargument).Theexecutionofthisyields(Iprintedonlytheresultofonecall):

$pythonarguments.all.kwonly.py

a,b:342

c,d:01

args:(7,9,11)

kwargs:{'f':'F','e':'E'}

OneotherthingtonotearethenamesIgavetothevariablepositionalandkeyword

arguments.You’refreetochoosedifferently,butbeawarethatargsandkwargsaretheconventionalnamesgiventotheseparameters,atleastgenerically.Nowthatyouknowhowtodefineafunctioninallpossibleflavors,letmeshowyousomethingtricky:mutabledefaults.

Avoidthetrap!MutabledefaultsOnethingtobeveryawareofwithPythonisthatdefaultvaluesarecreatedatdeftime,therefore,subsequentcallstothesamefunctionwillpossiblybehavedifferentlyaccordingtothemutabilityoftheirdefaultvalues.Let’slookatanexample:arguments.defaults.mutable.py

deffunc(a=[],b={}):

print(a)

print(b)

print('#'*12)

a.append(len(a))#thiswillaffecta'sdefaultvalue

b[len(a)]=len(a)#andthiswillaffectb'sone

func()

func()

func()

Theparametersbothhavemutabledefaultvalues.Thismeansthat,ifyouaffectthoseobjects,anymodificationwillstickaroundinsubsequentfunctioncalls.Seeifyoucanunderstandtheoutputofthosecalls:

$pythonarguments.defaults.mutable.py

[]

{}

############

[0]

{1:1}

############

[0,1]

{1:1,2:2}

############

It’sinteresting,isn’tit?Whilethisbehaviormayseemveryweirdatfirst,itactuallymakessense,andit’sveryhandy,forexample,whenusingmemoizationtechniques(Googleanexampleofthat,ifyou’reinterested).

Evenmoreinterestingiswhathappenswhen,betweenthecalls,weintroduceonethatdoesn’tusedefaults,likethis:arguments.defaults.mutable.intermediate.call.py

func()

func(a=[1,2,3],b={'B':1})

func()

Whenwerunthiscode,thisistheoutput:

$pythonarguments.defaults.mutable.intermediate.call.py

[]

{}

############

[1,2,3]

{'B':1}

############

[0]

{1:1}

############

Thisoutputshowsusthatthedefaultsareretainedevenifwecallthefunctionwithothervalues.Onequestionthatcomestomindis,howdoIgetafreshemptyvalueeverytime?Well,theconventionisthefollowing:arguments.defaults.mutable.no.trap.py

deffunc(a=None):

ifaisNone:

a=[]

#dowhateveryouwantwith`a`...

Notethat,byusingtheprecedingtechnique,ifaisn’tpassedwhencallingthefunction,youalwaysgetabrandnewemptylist.

Okay,enoughwiththeinput,let’slookattheothersideofthecoin,theoutput.

ReturnvaluesReturnvaluesoffunctionsareoneofthosethingswherePythonislightyearsaheadofmostotherlanguages.Functionsareusuallyallowedtoreturnoneobject(onevalue)but,inPython,youcanreturnatuple,andthisimpliesthatyoucanreturnwhateveryouwant.Thisfeatureallowsacodertowritesoftwarethatwouldbemuchhardertowriteinanyotherlanguage,orcertainlymoretedious.We’vealreadysaidthattoreturnsomethingfromafunctionweneedtousethereturnstatement,followedbywhatwewanttoreturn.Therecanbeasmanyreturnstatementsasneededinthebodyofafunction.

Ontheotherhand,ifwithinthebodyofafunctionwedon’treturnanything,thefunctionwillreturnNone.Thisbehaviorisharmlessand,eventhoughIdon’thavetheroomheretogointodetailexplainingwhyPythonwasdesignedlikethis,letmejusttellyouthatthisfeatureallowsforseveralinterestingpatterns,andconfirmsPythonasaveryconsistentlanguage.

Isayit’sharmlessbecauseyouareneverforcedtocollecttheresultofafunctioncall.I’llshowyouwhatImeanwithanexample:return.none.py

deffunc():

pass

func()#thereturnofthiscallwon'tbecollected.It'slost.

a=func()#thereturnofthisoneinsteadiscollectedinto`a`

print(a)#prints:None

Notethatthewholebodyofthefunctioniscomprisedonlyofthepassstatement.Astheofficialdocumentationtellsus,passisanulloperation.Whenitisexecuted,nothinghappens.Itisusefulasaplaceholderwhenastatementisrequiredsyntactically,butnocodeneedstobeexecuted.Inotherlanguages,wewouldprobablyjustindicatethatwithapairofcurlybraces({}),whichdefineanemptyscopebutinPythonascopeisdefinedbyindentingcode,thereforeastatementsuchaspassisnecessary.

Noticealsothatthefirstcallofthefunctionfuncreturnsavalue(None)whichwedon’tcollect.AsIsaidbefore,collectingthereturnvalueofafunctioncallisnotmandatory.

Now,that’sgoodbutnotveryinterestingso,howaboutwewriteaninterestingfunction?RememberthatinChapter1,IntroductionandFirstSteps–TakeaDeepBreath,wetalkedaboutthefactorialofafunction.Let’swriteourownhere(forsimplicity,IwillassumethefunctionisalwayscalledcorrectlywithappropriatevaluessoIwon’tsanity-checkontheinputargument):return.single.value.py

deffactorial(n):

ifnin(0,1):

return1

result=n

forkinrange(2,n):

result*=k

returnresult

f5=factorial(5)#f5=120

Notethatwehavetwopointsofreturn.Ifniseither0or1(inPythonit’scommontousetheintypeofcheckasIdidinsteadofthemoreverboseifn==0orn==1:),wereturn1.Otherwise,weperformtherequiredcalculation,andwereturnresult.CanwewritethisfunctionalittlebitmorePythonically?Yes,butI’llletyoufigureoutthatforyourself,asanexercise.return.single.value.2.py

fromfunctoolsimportreduce

fromoperatorimportmul

deffactorial(n):

returnreduce(mul,range(1,n+1),1)

f5=factorial(5)#f5=120

Iknowwhatyou’rethinking,oneline?Pythoniselegant,andconcise!Ithinkthisfunctionisreadableevenifyouhaveneverseenreduceormul,butifyoucan’treaditorunderstandit,setasideafewminutesanddosomeresearchonthePythondocumentationuntilitsbehavioriscleartoyou.Beingabletolookupfunctionsinthedocumentationandunderstandcodewrittenbysomeoneelseisataskeverydeveloperneedstobeabletoperform,sothinkofthisasagoodexercise,andgoodluck!

TipTothisend,makesureyoulookupthehelpfunction,whichcomesinveryhandyexploringwiththeconsole.

ReturningmultiplevaluesUnlikeinmostotherlanguages,inPythonit’sveryeasytoreturnmultipleobjectsfromafunction.Thisfeatureopensupawholeworldofpossibilitiesandallowsyoutocodeinastylethatishardtoreproducewithotherlanguages.Ourthinkingislimitedbythetoolsweuse,thereforewhenPythongivesyoumorefreedomthanotherlanguages,itisactuallyboostingyourowncreativityaswell.Toreturnmultiplevaluesisveryeasy,youjustusetuples(eitherexplicitlyorimplicitly).Let’slookatasimpleexamplethatmimicsthedivmodbuilt-infunction:return.multiple.py

defmoddiv(a,b):

returna//b,a%b

print(moddiv(20,7))#prints(2,6)

Icouldhavewrappedthehighlightedpartintheprecedingcodeinbraces,makingitanexplicittuple,butthere’snoneedforthat.Theprecedingfunctionreturnsboththeresultandtheremainderofthedivision,atthesametime.

AfewusefultipsWhenwritingfunctions,it’sveryusefultofollowguidelinessothatyouwritethemwell.I’llquicklypointsomeofthemouthere:

Functionsshoulddoonething:Functionsthatdoonethingareeasytodescribeinoneshortsentence.Functionswhichdomultiplethingscanbesplitintosmallerfunctionswhichdoonething.Thesesmallerfunctionsareusuallyeasiertoreadandunderstand.Rememberthedatascienceexamplewesawafewpagesago.Functionsshouldbesmall:Thesmallertheyare,theeasieritistotestthemandtowritethemsothattheydoonething.Thefewerinputparameters,thebetter:Functionswhichtakealotofargumentsquicklybecomehardertomanage(amongotherissues).Functionsshouldbeconsistentintheirreturnvalues:ReturningFalseorNoneisnotthesamething,evenifwithinaBooleancontexttheybothevaluatetoFalse.Falsemeansthatwehaveinformation(False),whileNonemeansthatthereisnoinformation.Trywritingfunctionswhichreturninaconsistentway,nomatterwhathappensintheirbody.Functionsshouldn’thavesideeffects:Inotherwords,functionsshouldnotaffectthevaluesyoucallthemwith.Thisisprobablythehardeststatementtounderstandatthispoint,soI’llgiveyouanexampleusinglists.Inthefollowingcode,notehownumbersisnotsortedbythesortedfunction,whichactuallyreturnsasortedcopyofnumbers.Conversely,thelist.sort()methodisactingonthenumbersobjectitself,andthatisfinebecauseitisamethod(afunctionthatbelongstoanobjectandthereforehastherightstomodifyit):

>>>numbers=[4,1,7,5]

>>>sorted(numbers)#won'tsorttheoriginal`numbers`list

[1,4,5,7]

>>>numbers#let'sverify

[4,1,7,5]#good,untouched

>>>numbers.sort()#thiswillactonthelist

>>>numbers

[1,4,5,7]

Followtheseguidelinesandyou’llwritebetterfunctions,whichwillserveyouwell.

NoteChapter3,FunctionsinCleanCodebyRobertC.Martin,PrenticeHallisdedicatedtofunctionsandit’sprobablythebestsetofguidelinesI’veeverreadonthesubject.

RecursivefunctionsWhenafunctioncallsitselftoproducearesult,itissaidtoberecursive.Sometimesrecursivefunctionsareveryusefulinthattheymakeiteasiertowritecode.Somealgorithmsareveryeasytowriteusingtherecursiveparadigm,whileothersarenot.Thereisnorecursivefunctionthatcannotberewritteninaniterativefashion,soit’susuallyuptotheprogrammertochoosethebestapproachforthecaseathand.

Arecursivefunctionusuallyhasasetofbasecasesforwhichthereturnvaluedoesn’tdependonasubsequentcalltothefunctionitselfandasetofrecursivecases,forwhichthereturnvalueiscalculatedwithoneormorecallstothefunctionitself.

Asanexample,wecanconsiderthe(hopefullyfamiliarbynow)factorialfunctionN!.ThebasecaseiswhenNiseither0or1.Thefunctionreturns1withnoneedforfurthercalculation.Ontheotherhand,inthegeneralcase,N!returnstheproduct1*2*…*(N-1)*N.Ifyouthinkaboutit,N!canberewrittenlikethis:N!=(N-1)!*N.Asapracticalexample,consider5!=1*2*3*4*5=(1*2*3*4)*5=4!*5.

Let’swritethisdownincode:recursive.factorial.py

deffactorial(n):

ifnin(0,1):#basecase

return1

returnfactorial(n-1)*n#recursivecase

NoteWhenwritingrecursivefunctions,alwaysconsiderhowmanynestedcallsyoumake,thereisalimit.Forfurtherinformationonthis,checkoutsys.getrecursionlimit()andsys.setrecursionlimit().

Recursivefunctionsareusedalotwhenwritingalgorithmsandtheycanbereallyfuntowrite.Asagoodexercise,trytosolveacoupleofsimpleproblemsusingbotharecursiveandaniterativeapproach.

AnonymousfunctionsOnelasttypeoffunctionsthatIwanttotalkaboutareanonymousfunctions.Thesefunctions,whicharecalledlambdasinPython,areusuallyusedwhenafully-fledgedfunctionwithitsownnamewouldbeoverkill,andallwewantisaquick,simpleone-linerthatdoesthejob.

ImaginethatyouwantalistofallthenumbersuptoNwhicharemultiplesoffive.Imaginethatyouwanttofilterthoseoutusingthefilterfunction,whichtakesafunctionandaniterableandconstructsafilterobjectwhichyoucaniterateon,fromthoseelementsofiterableforwhichthefunctionreturnsTrue.Withoutusingananonymousfunction,youwoulddosomethinglikethis:filter.regular.py

defis_multiple_of_five(n):

returnnotn%5

defget_multiples_of_five(n):

returnlist(filter(is_multiple_of_five,range(n)))

print(get_multiples_of_five(50))

Ihavehighlightedthemainlogicofget_multiples_of_five.Notehowthefilterusesis_multiple_of_fivetofilterthefirstnnaturalnumbers.Thisseemsabitexcessive,thetaskissimpleandwedon’tneedtokeeptheis_multiple_of_fivefunctionaroundforanythingelse.Let’srewriteitusingalambdafunction:filter.lambda.py

defget_multiples_of_five(n):

returnlist(filter(lambdak:notk%5,range(n)))

print(get_multiples_of_five(50))

Thelogicisexactlythesamebutthefilteringfunctionisnowalambda.Definingalambdaisveryeasyandfollowsthisform:func_name=lambda[parameter_list]:expression.Afunctionobjectisreturned,whichisequivalenttothis:deffunc_name([parameter_list]):returnexpression.

NoteNotethatoptionalparametersareindicatedfollowingthecommonsyntaxofwrappingtheminsquarebrackets.

Let’slookatanothercoupleofexamplesofequivalentfunctionsdefinedinthetwoforms:lambda.explained.py

#example1:adder

defadder(a,b):

returna+b

#isequivalentto:

adder_lambda=lambdaa,b:a+b

#example2:touppercase

defto_upper(s):

returns.upper()

#isequivalentto:

to_upper_lambda=lambdas:s.upper()

Theprecedingexamplesareverysimple.Thefirstoneaddstwonumbers,andthesecondoneproducestheuppercaseversionofastring.NotethatIassignedwhatisreturnedbythelambdaexpressionstoaname(adder_lambda,to_upper_lambda),butthereisnoneedforthatwhenyouuselambdasinthewaywedidinthefilterexamplebefore.

FunctionattributesEveryfunctionisafully-fledgedobjectand,assuch,theyhavemanyattributes.Someofthemarespecialandcanbeusedinanintrospectivewaytoinspectthefunctionobjectatruntime.Thefollowingscriptisanexamplethatshowsallofthemandhowtodisplaytheirvalueforanexamplefunction:func.attributes.py

defmultiplication(a,b=1):

"""Returnamultipliedbyb."""

returna*b

special_attributes=[

"__doc__","__name__","__qualname__","__module__",

"__defaults__","__code__","__globals__","__dict__",

"__closure__","__annotations__","__kwdefaults__",

]

forattributeinspecial_attributes:

print(attribute,'->',getattr(multiplication,attribute))

Iusedthebuilt-ingetattrfunctiontogetthevalueofthoseattributes.getattr(obj,attribute)isequivalenttoobj.attributeandcomesinhandywhenweneedtogetanattributeatruntimeusingitsstringname.Runningthisscriptyields:

$pythonfunc.attributes.py

__doc__->Returnamultipliedbyb.

__name__->multiplication

__qualname__->multiplication

__module__->__main__

__defaults__->(1,)

__code__-><codeobjectmultiplicationat0x7ff529e79300,file

"ch4/func.attributes.py",line1>

__globals__->{...omitted…}

__dict__->{}

__closure__->None

__annotations__->{}

__kwdefaults__->None

Ihaveomittedthevalueofthe__globals__attribute,itwastoobig.AnexplanationofthemeaningofthisattributecanbefoundinthetypessectionofthePythonDataModeldocumentationpage.

Built-infunctionsPythoncomeswithalotofbuilt-infunctions.Theyareavailableanywhereandyoucangetalistofthembyinspectingthebuiltinmodulewithdir(__builtin__),orbygoingtotheofficialPythondocumentation.Unfortunately,Idon’thavetheroomtogothroughallofthemhere.Someofthemwe’vealreadyseen,suchasany,bin,bool,divmod,filter,float,getattr,id,int,len,list,min,print,set,tuple,type,andzip,buttherearemanymore,whichyoushouldreadatleastonce.

Getfamiliarwiththem,experiment,writeasmallpieceofcodeforeachofthem,makesureyouhavethematthetipofyourfingerssothatyoucanusethemwhenyouneedthem.

OnefinalexampleBeforewefinishoffthischapter,howaboutafinalexample?Iwasthinkingwecouldwriteafunctiontogeneratealistofprimenumbersuptoalimit.We’vealreadyseenthecodeforthissolet’smakeitafunctionand,tokeepitinteresting,let’soptimizeitabit.

Itturnsoutthatyoudon’tneedtodivideitbyallnumbersfrom2toN-1todecideifa

numberNisprime.Youcanstopat .Moreover,youdon’tneedtotestthedivisionfor

allnumbersfrom2to ,youcanjustusetheprimesinthatrange.I’llleaveittoyoutofigureoutwhythisworks,ifyou’reinterested.Let’sseehowthecodechanges:primes.py

frommathimportsqrt,ceil

defget_primes(n):

"""Calculatealistofprimesupton(included)."""

primelist=[]

forcandidateinrange(2,n+1):

is_prime=True

root=int(ceil(sqrt(candidate)))#divisionlimit

forprimeinprimelist:#wetryonlytheprimes

ifprime>root:#noneedtocheckanyfurther

break

ifcandidate%prime==0:

is_prime=False

break

ifis_prime:

primelist.append(candidate)

returnprimelist

Thecodeisthesameasinthepreviouschapter.Wehavechangedthedivisionalgorithmsothatweonlytestdivisibilityusingthepreviouslycalculatedprimesandwestoppedoncethetestingdivisorwasgreaterthantherootofthecandidate.Weusedtheresultlistprimelisttogettheprimesforthedivision.Wecalculatedtherootvalueusingafancyformula,theintegervalueoftheceilingoftherootofthecandidate.Whileasimpleint(k**0.5)+1wouldhaveservedourpurposeaswell,theformulaIchoseiscleanerandrequiresmetouseacoupleofimports,whichIwantedtoshowyou.Checkoutthefunctionsinthemathmodule,theyareveryinteresting!

DocumentingyourcodeI’mabigfanofcodethatdoesn’tneeddocumentation.Whenyouprogramcorrectly,choosetherightnamesandtakecareofthedetails,yourcodeshouldcomeoutasself-explanatoryanddocumentationshouldnotbeneeded.Sometimesacommentisveryusefulthough,andsoissomedocumentation.YoucanfindtheguidelinesfordocumentingPythoninPEP257–Docstringconventions,butI’llshowyouthebasicshere.

Pythonisdocumentedwithstrings,whichareaptlycalleddocstrings.Anyobjectcanbedocumented,andyoucanuseeitherone-lineormulti-linedocstrings.One-linersareverysimple.Theyshouldnotprovideanothersignatureforthefunction,butclearlystateitspurpose.docstrings.py

defsquare(n):

"""Returnthesquareofanumbern."""

returnn**2

defget_username(userid):

"""Returntheusernameofausergiventheirid."""

returndb.get(user_id=userid).username

Usingtripledouble-quotedstringsallowsyoutoexpandeasilylateron.Usesentencesthatendinaperiod,anddon’tleaveblanklinesbeforeorafter.

Multi-linecommentsarestructuredinasimilarway.Thereshouldbeaone-linerthatbrieflygivesyouthegistofwhattheobjectisabout,andthenamoreverbosedescription.Asanexample,Ihavedocumentedafictitiousconnectfunction,usingtheSphinxnotation,inthefollowingexample.

NoteSphinxisprobablythemostwidelyusedtoolforcreatingPythondocumentation.Infact,theofficialPythondocumentationwaswrittenwithit.It’sdefinitelyworthspendingsometimecheckingitout.docstrings.py

defconnect(host,port,user,password):

"""Connecttoadatabase.

ConnecttoaPostgreSQLdatabasedirectly,usingthegiven

parameters.

:paramhost:ThehostIP.

:paramport:Thedesiredport.

:paramuser:Theconnectionusername.

:parampassword:Theconnectionpassword.

:return:Theconnectionobject.

"""

#bodyofthefunctionhere…

returnconnection

ImportingobjectsNowthatyouknowalotaboutfunctions,let’sseehowtousethem.Thewholepointofwritingfunctionsistobeabletolaterreusethem,andthisinPythontranslatestoimportingthemintothenamespaceinwhichyouneedthem.Therearemanydifferentwaystoimportobjectsintoanamespace,butthemostcommononesarejusttwo:importmodule_nameandfrommodule_nameimportfunction_name.Ofcourse,thesearequitesimplisticexamples,butbearwithmeforthetimebeing.

Theformimportmodule_namefindsthemodulemodule_nameanddefinesanameforitinthelocalnamespacewheretheimportstatementisexecuted.

Theformfrommodule_nameimportidentifierisalittlebitmorecomplicatedthanthat,butbasicallydoesthesamething.Itfindsmodule_nameandsearchesforanattribute(orasubmodule)andstoresareferencetoidentifierinthelocalnamespace.

Bothformshavetheoptiontochangethenameoftheimportedobjectusingtheasclause,likethis:

frommymoduleimportmyfuncasbetter_named_func

Justtogiveyouaflavorofwhatimportinglookslike,here’sanexamplefromatestmoduleofanumbertheorylibraryIwrotesomeyearsago(it’savailableonBitbucket):karma/test_nt.py

importunittest#importstheunittestmodule

frommathimportsqrt#importsonefunctionfrommath

fromrandomimportrandint,sample#twoimportsatonce

frommockimportpatch

fromnose.toolsimport(#multilineimport

assert_equal,

assert_list_equal,

assert_not_in,

)

fromkarmaimportnt,utils

IcommentedsomeofthemandIhopeit’seasytofollow.Whenyouhaveastructureoffilesstartingintherootofyourproject,youcanusethedotnotationtogettotheobjectyouwanttoimportintoyourcurrentnamespace,beitapackage,amodule,aclass,afunction,oranythingelse.Thefrommoduleimportsyntaxalsoallowsacatch-allclausefrommoduleimport*,whichissometimesusedtogetallthenamesfromamoduleintothecurrentnamespaceatonce,butit’sfrowneduponforseveralreasons:performances,theriskofsilentlyshadowingothernames,andsoon.YoucanreadallthatthereistoknowaboutimportsintheofficialPythondocumentationbut,beforeweleavethesubject,letmegiveyouabetterexample.

Imaginethatyouhavedefinedacoupleoffunctions:square(n)andcube(n)inamodule,funcdef.py,whichisinthelibfolder.Youwanttousetheminacoupleofmodules

whichareatthesamelevelofthelibfolder,calledfunc_import.py,andfunc_from.py.Showingthetreestructureofthatprojectproducessomethinglikethis:

├──func_from.py

├──func_import.py

├──lib

├──funcdef.py

└──__init__.py

BeforeIshowyouthecodeofeachmodule,pleaserememberthatinordertotellPythonthatitisactuallyapackage,weneedtoputa__init__.pymoduleinit.

NoteTherearetwothingstonoteaboutthe__init__.pyfile.Firstofall,itisafullyfledgedPythonmodulesoyoucanputcodeintoitasyouwouldwithanyothermodule.Second,asofPython3.3,itspresenceisnolongerrequiredtomakeafolderbeinterpretedasaPythonpackage.

Thecodeisasfollows:funcdef.py

defsquare(n):

returnn**2

defcube(n):

returnn**3

func_import.py

importlib.funcdef

print(lib.funcdef.square(10))

print(lib.funcdef.cube(10))

func_from.py

fromlib.funcdefimportsquare,cube

print(square(10))

print(cube(10))

Boththesefiles,whenexecuted,print100and1000.Youcanseehowdifferentlywethenaccessthesquareandcubefunctions,accordingtohowandwhatweimportedinthecurrentscope.

RelativeimportsTheimportswe’veseenuntilnowarecalledabsolute,thatistosaytheydefinethewholepathofthemodulethatwewanttoimport,orfromwhichwewanttoimportanobject.ThereisanotherwayofimportingobjectsintoPython,whichiscalledrelativeimport.It’shelpfulinsituationsinwhichwewanttorearrangethestructureoflargepackageswithouthavingtoeditsub-packages,orwhenwewanttomakeamoduleinsideapackageabletoimportitself.Relativeimportsaredonebyaddingasmanyleadingdotsinfrontofthemoduleasthenumberoffoldersweneedtobacktrack,inordertofindwhatwe’researchingfor.Simplyput,itissomethinglikethis:

from.mymoduleimportmyfunc

Foracompleteexplanationofrelativeimports,refertoPEP328(https://www.python.org/dev/peps/pep-0328).

Inlaterchapters,we’llcreateprojectsusingdifferentlibrariesandwe’lluseseveraldifferenttypesofimports,includingrelativeones,somakesureyoutakeabitoftimetoreadupaboutitintheofficialPythondocumentation.

SummaryInthischapter,finallyweexploredtheworldoffunctions.Theyareextremelyimportantand,fromnowon,we’llusethembasicallyeverywhere.Wetalkedaboutthemainreasonsforusingthem,themostimportantofwhicharecodereuseandimplementationhiding.

Wesawthatafunctionobjectislikeaboxthattakesoptionalinputandproducesoutput.Wecanfeedinputvaluestoafunctioninmanydifferentways,usingpositionalandkeywordarguments,andusingvariablesyntaxforbothtypes.

Nowyoushouldknowhowtowriteafunction,howtodocumentit,importitintoyourcode,andcallit.

ThenextchapterwillforcemetopushmyfootdownonthethrottleevenmoresoIsuggestyoutakeanyopportunityyougettoconsolidateandenrichtheknowledgeyou’vegathereduntilnowbyputtingyournoseintothePythonofficialdocumentation.

Readyforthecoolstuff?Let’sgo!

Chapter5.SavingTimeandMemory “It’snotthedailyincreasebutdailydecrease.Hackawayattheunessential.”

—BruceLee

IlovethisquotefromBruceLee,hewassuchawiseman!Especially,thesecondpart,hackawayattheunessential,istomewhatmakesacomputerprogramelegant.Afterall,ifthereisabetterwayofdoingthingssothatwedon’twastetimeormemory,whynot?

Sometimes,therearevalidreasonsfornotpushingourcodeuptothemaximumlimit:forexample,sometimestoachieveanegligibleimprovement,wehavetosacrificeonreadabilityormaintainability.Doesitmakeanysensetohaveawebpageservedin1secondwithunreadable,complicatedcode,whenwecanserveitin1.05secondswithreadable,cleancode?No,itmakesnosense.

Ontheotherhand,sometimesit’sperfectlylicittotryandshaveoffamillisecondfromafunction,especiallywhenthefunctionismeanttobecalledthousandsoftimes.Everymillisecondyousavetheremeansonesecondsavedperthousandofcalls,andthiscouldbemeaningfulforyourapplication.

Inlightoftheseconsiderations,thefocusofthischapterwillnotbetogiveyouthetoolstopushyourcodetotheabsolutelimitsofperformanceandoptimization“nomatterwhat”,butrather,togiveyouthetoolstowriteefficient,elegantcodethatreadswell,runsfast,anddoesn’twasteresourcesinanobviousway.

Inthischapter,Iwillperformseveralmeasurementsandcomparisons,andcautiouslydrawsomeconclusions.Pleasedokeepinmindthatonadifferentboxwithadifferentsetuporadifferentoperatingsystem,resultsmayvary.Takealookatthiscode:squares.py

defsquare1(n):

returnn**2#squaringthroughthepoweroperator

defsquare2(n):

returnn*n#squaringthroughmultiplication

Bothfunctionsreturnthesquareofn,butwhichisfaster?FromasimplebenchmarkIranonthem,itlookslikethesecondisslightlyfaster.Ifyouthinkaboutit,itmakessense:calculatingthepowerofanumberinvolvesmultiplicationandtherefore,whateveralgorithmyoumayusetoperformthepoweroperation,it’snotlikelytobeatasimplemultiplicationliketheoneinsquare2.

Dowecareaboutthisresult?Inmostcasesno.Ifyou’recodingane-commercewebsite,chancesareyouwon’teverevenneedtoraiseanumbertothesecondpower,andifyoudo,youprobablywillhavetodoitafewtimesperpage.Youdon’tneedtoconcernyourselfonsavingafewmicrosecondsonafunctionyoucallafewtimes.

So,whendoesoptimizationbecomeimportant?Oneverycommoncaseiswhenyouhavetodealwithhugecollectionsofdata.Ifyou’reapplyingthesamefunctiononamillion

customerobjects,thenyouwantyourfunctiontobetuneduptoitsbest.Gaining1/10ofasecondonafunctioncalledonemilliontimessavesyou100,000seconds,whichareabout27.7hours.That’snotthesame,right?So,let’sfocusoncollections,andlet’sseewhichtoolsPythongivesyoutohandlethemwithefficiencyandgrace.

NoteManyoftheconceptswewillseeinthischapterarebasedonthoseofiteratoranditerable.Simplyput,theabilityforanobjecttoreturnitsnextelementwhenasked,andtoraiseaStopIterationexceptionwhenexhausted.We’llseehowtocodeacustomiteratoranditerableobjectsinthenextchapter.

map,zip,andfilterWe’llstartbyreviewingmap,filter,andzip,whicharethemainbuilt-infunctionsonecanemploywhenhandlingcollections,andthenwe’lllearnhowtoachievethesameresultsusingtwoveryimportantconstructs:comprehensionsandgenerators.Fastenyourseatbelt!

mapAccordingtotheofficialPythondocumentation:

map(function,iterable,...)returnsaniteratorthatappliesfunctiontoeveryitemofiterable,yieldingtheresults.Ifadditionaliterableargumentsarepassed,functionmusttakethatmanyargumentsandisappliedtotheitemsfromalliterablesinparallel.Withmultipleiterables,theiteratorstopswhentheshortestiterableisexhausted.

Wewillexplaintheconceptofyieldinglateroninthechapter.Fornow,let’stranslatethisintocode:we’llusealambdafunctionthattakesavariablenumberofpositionalarguments,andjustreturnsthemasatuple.Also,asmapreturnsaniterator,we’llneedtowrapeachcalltoitwithinalistconstructorsothatweexhausttheiterablebyputtingallofitselementsintoalist(you’llseeanexampleofthisinthecode):map.example.py

>>>map(lambda*a:a,range(3))#withoutwrappinginlist…

<mapobjectat0x7f563513b518>#wegettheiteratorobject

>>>list(map(lambda*a:a,range(3)))#wrappinginlist…

[(0,),(1,),(2,)]#wegetalistwithitselements

>>>list(map(lambda*a:a,range(3),'abc'))#2iterables

[(0,'a'),(1,'b'),(2,'c')]

>>>list(map(lambda*a:a,range(3),'abc',range(4,7)))#3

[(0,'a',4),(1,'b',5),(2,'c',6)]

>>>#mapstopsattheshortestiterator

>>>list(map(lambda*a:a,(),'abc'))#emptytupleisshortest

[]

>>>list(map(lambda*a:a,(1,2),'abc'))#(1,2)shortest

[(1,'a'),(2,'b')]

>>>list(map(lambda*a:a,(1,2,3,4),'abc'))#'abc'shortest

[(1,'a'),(2,'b'),(3,'c')]

Intheprecedingcodeyoucanseewhy,inordertopresentyouwiththeresults,Ihavetowrapthecallstomapwithinalistconstructor,otherwiseIgetthestringrepresentationofamapobject,whichisnotreallyusefulinthiscontext,isit?

Youcanalsonoticehowtheelementsofeachiterableareappliedtothefunction:atfirst,thefirstelementofeachiterable,thenthesecondoneofeachiterable,andsoon.Noticealsothatmapstopswhentheshortestoftheiterableswecalleditwithisexhausted.Thisisactuallyaverynicebehavior:itdoesn’tforceustoleveloffalltheiterablestoacommonlength,anditdoesn’tbreakiftheyaren’tallthesamelength.

mapisveryusefulwhenyouhavetoapplythesamefunctiontooneormorecollectionsofobjects.Asamoreinterestingexample,let’sseethedecorate-sort-undecorateidiom(alsoknownasSchwartziantransform).It’satechniquethatwasextremelypopularwhenPythonsortingwasn’tprovidingkey-functions,andthereforetodayislessused,butit’sacooltrickthatstillcomesathandonceinawhile.

Let’sseeavariationofitinthenextexample:wewanttosortindescendingorderbythe

sumofcreditsaccumulatedbystudents,sotohavethebeststudentatposition0.Wewriteafunctiontoproduceadecoratedobject,wesort,andthenweundecorate.Eachstudenthascreditsinthree(possiblydifferent)subjects.Todecorateanobjectmeanstotransformit,eitheraddingextradatatoit,orputtingitintoanotherobject,inawaythatallowsustobeabletosorttheoriginalobjectsthewaywewant.Afterthesorting,werevertthedecoratedobjectstogettheoriginalonesfromthem.Thisiscalledtoundecorate.decorate.sort.undecorate.py

students=[

dict(id=0,credits=dict(math=9,physics=6,history=7)),

dict(id=1,credits=dict(math=6,physics=7,latin=10)),

dict(id=2,credits=dict(history=8,physics=9,chemistry=10)),

dict(id=3,credits=dict(math=5,physics=5,geography=7)),

]

defdecorate(student):

#createa2-tuple(sumofcredits,student)fromstudentdict

return(sum(student['credits'].values()),student)

defundecorate(decorated_student):

#discardsumofcredits,returnoriginalstudentdict

returndecorated_student[1]

students=sorted(map(decorate,students),reverse=True)

students=list(map(undecorate,students))

Intheprecedingcode,Ihighlightedthetrickyandimportantparts.Let’sstartbyunderstandingwhateachstudentobjectis.Infact,let’sprintthefirstone:{'credits':{'history':7,'math':9,'physics':6},'id':0}

Youcanseethatit’sadictionarywithtwokeys:idandcredit.Thevalueofcreditisalsoadictionaryinwhichtherearethreesubject/gradekey/valuepairs.AsI’msureyourecallfromourvisitinthedatastructuresworld,callingdict.values()returnsanobjectsimilartoaniterable,withonlythevalues.Therefore,sum(student['credits'].values()),forthefirststudentisequivalenttosum(9,6,7)(oranypermutationofthosenumbersbecausedictionariesdon’tretainorder,butluckilyforus,additioniscommutative).

Withthatoutoftheway,it’seasytoseewhatistheresultofcallingdecoratewithanyofthestudents.Let’sprinttheresultofdecorate(students[0]):(22,{'credits':{'history':7,'math':9,'physics':6},'id':0})

That’snice!Ifwedecorateallthestudentslikethis,wecansortthemontheirtotalamountofcreditsbutjustsortingthelistoftuples.Inordertoapplythedecorationtoeachiteminstudents,wecallmap(decorate,students).Thenwesorttheresult,andthenweundecorateinasimilarfashion.Ifyouhavegonethroughthepreviouschapterscorrectly,understandingthiscodeshouldn’tbetoohard.

Printingstudentsafterrunningthewholecodeyields:

$pythondecorate.sort.undecorate.py

[{'credits':{'chemistry':10,'history':8,'physics':9},'id':2},

{'credits':{'latin':10,'math':6,'physics':7},'id':1},

{'credits':{'history':7,'math':9,'physics':6},'id':0},

{'credits':{'geography':7,'math':5,'physics':5},'id':3}]

Andyoucansee,bytheorderofthestudentobjects,thattheyhaveindeedbeensortedbythesumoftheircredits.

NoteFormoreonthedecorate-sort-undecorateidiom,there’saveryniceintroductioninthesortinghow-tosectionoftheofficialPythondocumentation(https://docs.python.org/3.4/howto/sorting.html#the-old-way-using-decorate-sort-undecorate).

Onethingtonoticeaboutthesortingpart:whatiftwoormorestudentssharethesametotalsum?Thesortingalgorithmwouldthenproceedsortingthetuplesbycomparingthestudentobjectswitheachother.Thisdoesn’tmakeanysense,andinmorecomplexcasescouldleadtounpredictableresults,orevenerrors.Ifyouwanttobesuretoavoidthisissue,onesimplesolutionistocreatea3-tupleinsteadofa2-tuple,havingthesumofcreditsinthefirstposition,thepositionofthestudentobjectinthestudentslistinthesecondone,andthestudentobjectitselfinthethirdone.Thisway,ifthesumofcreditsisthesame,thetupleswillbesortedagainsttheposition,whichwillalwaysbedifferentandthereforeenoughtoresolvethesortingbetweenanypairoftuples.Formoreconsiderationsonthistopic,pleasecheckoutthesortinghow-tosectionontheofficialPythondocumentation.

zipWe’vealreadycoveredzipinthepreviouschapters,solet’sjustdefineitproperlyandthenIwanttoshowyouhowyoucouldcombineitwithmap.

AccordingtothePythondocumentation:

zip(*iterables)returnsaniteratoroftuples,wherethei-thtuplecontainsthei-thelementfromeachoftheargumentsequencesoriterables.Theiteratorstopswhentheshortestinputiterableisexhausted.Withasingleiterableargument,itreturnsaniteratorof1-tuples.Withnoarguments,itreturnsanemptyiterator.

Let’sseeanexample:zip.grades.py

>>>grades=[18,23,30,27,15,9,22]

>>>avgs=[22,21,29,24,18,18,24]

>>>list(zip(avgs,grades))

[(22,18),(21,23),(29,30),(24,27),(18,15),(18,9),(24,22)]

>>>list(map(lambda*a:a,avgs,grades))#equivalenttozip

[(22,18),(21,23),(29,30),(24,27),(18,15),(18,9),(24,22)]

Intheprecedingcode,we’rezippingtogethertheaverageandthegradeforthelastexam,pereachstudent.Noticehowthecodeinsidethetwolistcallsproducesexactlythesameresult,showinghoweasyitistoreproducezipusingmap.Noticealsothat,aswedoformap,wehavetofeedtheresultofthezipcalltoalistconstructor.

Asimpleexampleonthecombineduseofmapandzipcouldbeawayofcalculatingtheelement-wisemaximumamongstsequences,thatis,themaximumofthefirstelementofeachsequence,thenthemaximumofthesecondone,andsoon:maxims.py

>>>a=[5,9,2,4,7]

>>>b=[3,7,1,9,2]

>>>c=[6,8,0,5,3]

>>>maxs=map(lambdan:max(*n),zip(a,b,c))

>>>list(maxs)

[6,9,2,9,7]

Noticehoweasyitistocalculatethemaxvaluesofthreesequences.zipisnotstrictlyneededofcourse,wecouldjustusemap,butthiswouldrequireustowriteamuchmorecomplicatedfunctiontofeedmapwith.Sometimeswemaybeinasituationwherechangingthefunctionwefeedtomapisnotevenpossible.Incaseslikethese,beingabletomassagethedata(likewe’redoinginthisexamplewithzip)isveryhelpful.

filterAccordingtothePythondocumentation:

filter(function,iterable)constructaniteratorfromthoseelementsofiterableforwhichfunctionreturnsTrue.iterablemaybeeitherasequence,acontainerwhichsupportsiteration,oraniterator.IffunctionisNone,theidentityfunctionisassumed,thatis,allelementsofiterablethatarefalseareremoved.

Let’sseeaveryquickexample:filter.py

>>>test=[2,5,8,0,0,1,0]

>>>list(filter(None,test))

[2,5,8,1]

>>>list(filter(lambdax:x,test))#equivalenttopreviousone

[2,5,8,1]

>>>list(filter(lambdax:x>4,test))#keeponlyitems>4

[5,8]

Intheprecedingcode,noticehowthesecondcalltofilterisequivalenttothefirstone.Ifwepassafunctionthattakesoneargumentandreturnstheargumentitself,onlythoseargumentsthatareTruewillmakethefunctionreturnTrue,thereforethisbehaviorisexactlythesameaspassingNone.It’softenaverygoodexercisetomimicsomeofthebuilt-inPythonbehaviors.WhenyousucceedyoucansayyoufullyunderstandhowPythonbehavesinaspecificsituation.

Armedwithmap,zip,andfilter(andseveralotherfunctionsfromthePythonstandardlibrary)wecanmassagesequencesveryeffectively.Butthosefunctionsarenottheonlywaytodoit.Solet’sseeoneofthenicestfeaturesofPython:comprehensions.

ComprehensionsPythonoffersyoudifferenttypesofcomprehensions:list,dict,andset.

We’llconcentrateonthefirstonefornow,andthenitwillbeeasytoexplaintheothertwo.

Alistcomprehensionisaquickwayofmakingalist.Usuallythelististheresultofsomeoperationthatmayinvolveapplyingafunction,filtering,orbuildingadifferentdatastructure.

Let’sstartwithaverysimpleexampleIwanttocalculatealistwiththesquaresofthefirst10naturalnumbers.Howwouldyoudoit?Thereareacoupleofequivalentways:squares.map.py

#IfyoucodelikethisyouarenotaPythonguy!;)

>>>squares=[]

>>>forninrange(10):

...squares.append(n**2)

...

>>>list(squares)

[0,1,4,9,16,25,36,49,64,81]

#Thisisbetter,oneline,niceandreadable

>>>squares=map(lambdan:n**2,range(10))

>>>list(squares)

[0,1,4,9,16,25,36,49,64,81]

Theprecedingexampleshouldbenothingnewforyou.Let’sseehowtoachievethesameresultusingalistcomprehension:squares.comprehension.py

>>>[n**2forninrange(10)]

[0,1,4,9,16,25,36,49,64,81]

Assimpleasthat.Isn’titelegant?Basicallywehaveputaforloopwithinsquarebrackets.Let’snowfilterouttheoddsquares.I’llshowyouhowtodoitwithmapandfilter,andthenusingalistcomprehensionagain.even.squares.py

#usingmapandfilter

sq1=list(

filter(lambdan:notn%2,map(lambdan:n**2,range(10)))

)

#equivalent,butusinglistcomprehensions

sq2=[n**2forninrange(10)ifnotn%2]

print(sq1,sq1==sq2)#prints:[0,4,16,36,64]True

Ithinkthatnowthedifferenceinreadabilityisevident.Thelistcomprehensionreadsmuchbetter.It’salmostEnglish:givemeallsquares(n**2)fornbetween0and9ifniseven.

AccordingtothePythondocumentation:

Alistcomprehensionconsistsofbracketscontaininganexpressionfollowedbyaforclause,thenzeroormorefororifclauses.Theresultwillbeanewlistresultingfromevaluatingtheexpressioninthecontextoftheforandifclauseswhichfollowit”.

NestedcomprehensionsLet’sseeanexampleofnestedloops.It’sverycommonwhendealingwithalgorithmstohavetoiterateonasequenceusingtwoplaceholders.Thefirstonerunsthroughthewholesequence,lefttoright.Thesecondoneaswell,butitstartsfromthefirstone,insteadof0.Theconceptisthatoftestingallpairswithoutduplication.Let’sseetheclassicalforloopequivalent.pairs.for.loop.py

items='ABCDE'

pairs=[]

forainrange(len(items)):

forbinrange(a,len(items)):

pairs.append((items[a],items[b]))

Ifyouprintpairsattheend,youget:

[('A','A'),('A','B'),('A','C'),('A','D'),('A','E'),('B','B'),

('B','C'),('B','D'),('B','E'),('C','C'),('C','D'),('C','E'),

('D','D'),('D','E'),('E','E')]

Allthetupleswiththesameletterarethoseforwhichbisatthesamepositionasa.Now,let’sseehowwecantranslatethisinalistcomprehension:pairs.list.comprehension.py

items='ABCDE'

pairs=[(items[a],items[b])

forainrange(len(items))forbinrange(a,len(items))]

Thisversionisjusttwolineslongandachievesthesameresult.Noticethatinthisparticularcase,becausetheforloopoverbhasadependencyona,itmustfollowtheforloopoverainthecomprehension.Ifyouswapthemaround,you’llgetanameerror.

FilteringacomprehensionWecanapplyfilteringtoacomprehension.Let’sfirstdoitwithfilter.Let’sfindallPythagoreantripleswhoseshortsidesarenumberssmallerthan10.Weobviouslydon’twanttotestacombinationtwice,andthereforewe’lluseatrickliketheonewesawinthepreviousexample.

NoteAPythagoreantripleisatriple(a,b,c)ofintegernumberssatisfyingtheequation

.pythagorean.triple.py

frommathimportsqrt

#thiswillgenerateallpossiblepairs

mx=10

legs=[(a,b,sqrt(a**2+b**2))

forainrange(1,mx)forbinrange(a,mx)]

#thiswillfilteroutallnonpythagoreantriples

legs=list(

filter(lambdatriple:triple[2].is_integer(),legs))

print(legs)#prints:[(3,4,5.0),(6,8,10.0)]

Intheprecedingcode,wegeneratedalistof3-tuples,legs.Eachtuplecontainstwointegernumbers(thelegs)andthehypotenuseofthePythagoreantrianglewhoselegsarethefirsttwonumbersinthetuple.Forexample,whena=3andb=4,thetuplewillbe(3,4,5.0),andwhena=5andb=7,thetuplewillbe(5,7,8.602325267042627).

Afterhavingallthetriplesdone,weneedtofilteroutallthosethatdon’thaveahypotenusethatisanintegernumber.Inordertodothis,wefilterbasedonfloat_number.is_integer()beingTrue.ThismeansthatofthetwoexampletuplesIshowedyoubefore,theonewithhypotenuse5.0willberetained,whiletheonewithhypotenuse8.602325267042627willbediscarded.

Thisisgood,butIdon’tlikethatthetriplehastwointegernumbersandafloat.Theyaresupposedtobeallintegers,solet’susemaptofixthis:pythagorean.triple.int.py

frommathimportsqrt

mx=10

legs=[(a,b,sqrt(a**2+b**2))

forainrange(1,mx)forbinrange(a,mx)]

legs=filter(lambdatriple:triple[2].is_integer(),legs)

#thiswillmakethethirdnumberinthetuplesinteger

legs=list(

map(lambdatriple:triple[:2]+(int(triple[2]),),legs))

print(legs)#prints:[(3,4,5),(6,8,10)]

Noticethestepweadded.Wetakeeachelementinlegsandwesliceit,takingonlythefirsttwoelementsinit.Then,weconcatenatetheslicewitha1-tuple,inwhichweputtheintegerversionofthatfloatnumberthatwedidn’tlike.

Seemslikealotofwork,right?Indeeditis.Let’sseehowtodoallthiswithalistcomprehension:pythagorean.triple.comprehension.py

frommathimportsqrt

#thisstepisthesameasbefore

mx=10

legs=[(a,b,sqrt(a**2+b**2))

forainrange(1,mx)forbinrange(a,mx)]

#herewecombinefilterandmapinoneCLEANlistcomprehension

legs=[(a,b,int(c))fora,b,cinlegsifc.is_integer()]

print(legs)#prints:[(3,4,5),(6,8,10)]

Iknow.It’smuchbetter,isn’tit?It’sclean,readable,shorter.Inotherwords,elegant.

TipI’mgoingquitefasthere,asanticipatedinthesummaryofthelastchapter.Areyouplayingwiththiscode?Ifnot,Isuggestyoudo.It’sveryimportantthatyouplayaround,breakthings,changethings,seewhathappens.Makesureyouhaveaclearunderstandingofwhatisgoingon.Youwanttobecomeaninja,right?

dictcomprehensionsDictionaryandsetcomprehensionsworkexactlylikethelistones,onlythereisalittledifferenceinthesyntax.Thefollowingexamplewillsufficetoexplaineverythingyouneedtoknow:dictionary.comprehensions.py

fromstringimportascii_lowercase

lettermap=dict((c,k)fork,cinenumerate(ascii_lowercase,1))

Ifyouprintlettermap,youwillseethefollowing(Iomittedthemiddleresults,yougetthegist):

{'a':1,

'b':2,

'c':3,

...omittedresults…

'x':24,

'y':25,

'z':26}

Whathappensintheprecedingcodeisthatwe’refeedingthedictconstructorwithacomprehension(technically,ageneratorexpression,we’llseeitinabit).Wetellthedictconstructortomakekey/valuepairsfromeachtupleinthecomprehension.WeenumeratethesequenceofalllowercaseASCIIletters,startingfrom1,usingenumerate.Pieceofcake.Thereisalsoanotherwaytodothesamething,whichisclosertotheotherdictionarysyntax:

lettermap={c:kfork,cinenumerate(ascii_lowercase,1)}

Itdoesexactlythesamething,withaslightlydifferentsyntaxthathighlightsabitmoreofthekey:valuepart.

Dictionariesdonotallowduplicationinthekeys,asshowninthefollowingexample:dictionary.comprehensions.duplicates.py

word='Hello'

swaps={c:c.swapcase()forcinword}

print(swaps)#prints:{'o':'O','l':'L','e':'E','H':'h'}

Wecreateadictionarywithkeys,thelettersinthestring'Hello',andvaluesofthesameletters,butwiththecaseswapped.Noticethereisonlyone'l':'L'pair.Theconstructordoesn’tcomplain,simplyreassignsduplicatestothelatestvalue.Let’smakethisclearerwithanotherexample;let’sassigntoeachkeyitspositioninthestring:dictionary.comprehensions.positions.py

word='Hello'

positions={c:kfork,cinenumerate(word)}

print(positions)#prints:{'l':3,'o':4,'e':1,'H':0}

Noticethevalueassociatedtotheletter'l':3.Thepair'l':2isn’tthere,ithasbeenoverriddenby'l':3.

setcomprehensionsSetcomprehensionsareverysimilartolistanddictionaryones.Pythonallowsboththeset()constructortobeused,ortheexplicit{}syntax.Let’sseeonequickexample:set.comprehensions.py

word='Hello'

letters1=set(cforcinword)

letters2={cforcinword}

print(letters1)#prints:{'l','o','H','e'}

print(letters1==letters2)#prints:True

Noticehowforsetcomprehensions,asfordictionaries,duplicationisnotallowedandthereforetheresultingsethasonlyfourletters.Also,noticethattheexpressionsassignedtoletters1andletters2produceequivalentsets.

Thesyntaxusedtocreateletters2isverysimilartotheonewecanusetocreateadictionarycomprehension.Youcanspotthedifferenceonlybythefactthatdictionariesrequirekeysandvalues,separatedbycolumns,whilesetsdon’t.

GeneratorsGeneratorsareoneverypowerfultoolthatPythongiftsuswith.Theyarebasedontheconceptsofiteration,aswesaidbefore,andtheyallowforcodingpatternsthatcombineelegancewithefficiency.

Generatorsareoftwotypes:

Generatorfunctions:Theseareverysimilartoregularfunctions,butinsteadofreturningresultsthroughreturnstatements,theyuseyield,whichallowsthemtosuspendandresumetheirstatebetweeneachcallGeneratorexpressions:Theseareverysimilartothelistcomprehensionswe’veseeninthischapter,butinsteadofreturningalisttheyreturnanobjectthatproducesresultsonebyone

GeneratorfunctionsGeneratorfunctionscomeunderallaspectslikeregularfunctions,withonedifference:insteadofcollectingresultsandreturningthematonce,theycanstartthecomputation,yieldonevalue,suspendtheirstatesavingeverythingtheyneedtobeabletoresumeand,ifcalledagain,resumeandperformanotherstep.GeneratorfunctionsareautomaticallyturnedintotheirowniteratorsbyPython,soyoucancallnextonthem.

Thisisallverytheoreticalso,let’smakeitclearwhysuchamechanismissopowerful,andthenlet’sseeanexample.

SayIaskedyoutocountoutloudfrom1toamillion.Youstart,andatsomepointIaskyoutostop.Aftersometime,Iaskyoutoresume.Atthispoint,whatistheminimuminformationyouneedtobeabletoresumecorrectly?Well,youneedtorememberthelastnumberyoucalled.IfIstoppedyouafter31415,youwilljustgoonwith31416,andsoon.

Thepointis,youdon’tneedtorememberallthenumbersyousaidbefore31415,nordoyouneedthemtobewrittendownsomewhere.Well,youmaynotknowit,butyou’rebehavinglikeageneratoralready!

Takeagoodlookatthefollowingcode:first.n.squares.py

defget_squares(n):#classicfunctionapproach

return[x**2forxinrange(n)]

print(get_squares(10))

defget_squares_gen(n):#generatorapproach

forxinrange(n):

yieldx**2#weyield,wedon'treturn

print(list(get_squares_gen(10)))

Theresultoftheprintswillbethesame:[0,1,4,9,16,25,36,49,64,81].Butthereisahugedifferencebetweenthetwofunctions.get_squaresisaclassicfunctionthatcollectsallthesquaresofnumbersin[0,n)inalist,andreturnsit.Ontheotherhand,get_squares_genisagenerator,andbehavesverydifferently.Eachtimetheinterpreterreachestheyieldline,itsexecutionissuspended.Theonlyreasonthoseprintsreturnthesameresultisbecausewefedget_squares_gentothelistconstructor,whichwhencalledlikethatexhauststhegeneratorcompletelybyaskingthenextelementuntilaStopIterationisraised.Let’sseethisindetail:first.n.squares.manual.py

defget_squares_gen(n):

forxinrange(n):

yieldx**2

squares=get_squares_gen(4)#thiscreatesageneratorobject

print(squares)#<generatorobjectget_squares_genat0x7f158…>

print(next(squares))#prints:0

print(next(squares))#prints:1

print(next(squares))#prints:4

print(next(squares))#prints:9

#thefollowingraisesStopIteration,thegeneratorisexhausted,

#anyfurthercalltonextwillkeepraisingStopIteration

print(next(squares))

Intheprecedingcode,eachtimewecallnextonthegeneratorobject,weeitherstartit(firstnext)ormakeitresumefromthelastsuspensionpoint(anyothernext).

Thefirsttimewecallnextonit,weget0,whichisthesquareof0,then1,then4,then9andsincetheforloopstopsafterthat(nis4),thenthegeneratornaturallyends.AclassicfunctionwouldatthatpointjustreturnNone,butinordertocomplywiththeiterationprotocol,ageneratorwillinsteadraiseaStopIterationexception.

Thisexplainshowaforloopworksforexample.Whenyoucallforkinrange(n),whathappensunderthehoodisthattheforloopgetsaniteratoroutofrange(n)andstartscallingnextonit,untilStopIterationisraised,whichtellstheforloopthattheiterationhasreacheditsend.

Havingthisbehaviorbuilt-inineveryiterationaspectofPythonmakesgeneratorsevenmorepowerfulbecauseoncewewritethem,we’llbeabletoplugtheminwhateveriterationmechanismwewant.

Atthispoint,you’reprobablyaskingyourselfwhywouldyouwanttouseageneratorinsteadofaregularfunction.Well,thetitleofthischaptershouldsuggesttheanswer.I’lltalkaboutperformanceslater,sofornowlet’sconcentrateonanotheraspect:sometimesgeneratorsallowyoutodosomethingthatwouldn’tbepossiblewithasimplelist.Forexample,sayyouwanttoanalyzeallpermutationsofasequence.IfthesequencehaslengthN,thenthenumberofitspermutationsisN!.Thismeansthatifthesequenceis10elementslong,thenumberofpermutationsis3628800.Butasequenceof20elementswouldhave2432902008176640000permutations.Theygrowfactorially.

Nowimagineyouhaveaclassicfunctionthatisattemptingtocalculateallpermutations,puttheminalist,andreturnittoyou.With10elements,itwouldrequireprobablyafewtensofseconds,butfor20elementsthereissimplynowaythatitcanbedone.

Ontheotherhand,ageneratorfunctionwillbeabletostartthecomputationandgiveyoubackthefirstpermutation,thenthesecond,andsoon.Ofcourseyouwon’thavethetimetoparsethemall,theyaretoomany,butatleastyou’llbeabletoworkwithsomeofthem.

Rememberwhenweweretalkingaboutthebreakstatementinforloops?Whenwefoundanumberdividingacandidateprimewewerebreakingtheloop,noneedtogoon.

Sometimesit’sexactlythesame,onlytheamountofdatayouhavetoiterateoverissohugethatyoucannotkeepitallinmemoryinalist.Inthiscase,generatorsareinvaluable:theymakepossiblewhatwouldn’tbepossibleotherwise.

So,inordertosavememory(andtime),usegeneratorfunctionswheneverpossible.

It’salsoworthnotingthatyoucanusethereturnstatementinageneratorfunction.ItwillproduceaStopIterationexceptiontoberaised,effectivelyendingtheiteration.Thisisextremelyimportant.Ifareturnstatementwereactuallytomakethefunctionreturn

something,itwouldbreaktheiterationprotocol.Pythonconsistencypreventsthis,andallowsusgreateasewhencoding.Let’sseeaquickexample:gen.yield.return.py

defgeometric_progression(a,q):

k=0

whileTrue:

result=a*q**k

ifresult<=100000:

yieldresult

else:

return

k+=1

forningeometric_progression(2,5):

print(n)

Theprecedingcodeyieldsalltermsofthegeometricprogressiona,aq, , ,….Whentheprogressionproducesatermthatisgreaterthan100,000,thegeneratorstops(withareturnstatement).Runningthecodeproducesthefollowingresult:

$pythongen.yield.return.py

2

10

50

250

1250

6250

31250

Thenexttermwouldhavebeen156250,whichistoobig.

GoingbeyondnextAtthebeginningofthischapter,Itoldyouthatgeneratorobjectsarebasedontheiterationprotocol.We’llseeinthenextchapteracompleteexampleofhowtowriteacustomiterator/iterableobject.Fornow,Ijustwantyoutounderstandhownext()works.

Whathappenswhenyoucallnext(generator)isthatyou’recallingthegenerator.__next__()method.Remember,amethodisjustafunctionthatbelongstoanobject,andobjectsinPythoncanhavespecialmethods.Ourfriend__next__()isjustoneoftheseanditspurposeistoreturnthenextelementoftheiteration,ortoraiseStopIterationwhentheiterationisoverandtherearenomoreelementstoreturn.

NoteInPython,anobject’sspecialmethodsarealsocalledmagicmethods,ordunder(from“doubleunderscore”)methods.

Whenwewriteageneratorfunction,Pythonautomaticallytransformsitintoanobjectthatisverysimilartoaniterator,andwhenwecallnext(generator),thatcallistransformedingenerator.__next__().Let’srevisitthepreviousexampleaboutgeneratingsquares:first.n.squares.manual.method.py

defget_squares_gen(n):

forxinrange(n):

yieldx**2

squares=get_squares_gen(3)

print(squares.__next__())#prints:0

print(squares.__next__())#prints:1

print(squares.__next__())#prints:4

#thefollowingraisesStopIteration,thegeneratorisexhausted,

#anyfurthercalltonextwillkeepraisingStopIteration

print(squares.__next__())

Theresultisexactlyasthepreviousexample,onlythistimeinsteadofusingtheproxycallnext(squares),we’redirectlycallingsquares.__next__().

Generatorobjectshavealsothreeothermethodsthatallowcontrollingtheirbehavior:send,throw,andclose.sendallowsustocommunicateavaluebacktothegeneratorobject,whilethrowandcloserespectivelyallowraisinganexceptionwithinthegeneratorandclosingit.TheiruseisquiteadvancedandIwon’tbecoveringthemhereindetail,butIwanttospendafewwordsatleastaboutsend,withasimpleexample.

Takealookatthefollowingcode:gen.send.preparation.py

defcounter(start=0):

n=start

whileTrue:

yieldn

n+=1

c=counter()

print(next(c))#prints:0

print(next(c))#prints:1

print(next(c))#prints:2

Theprecedingiteratorcreatesageneratorobjectthatwillrunforever.Youcankeepcallingit,itwillneverstop.Alternatively,youcanputitinaforloop,forexample,fornincounter():...anditwillgoonforeveraswell.

Now,whatifyouwantedtostopitatsomepoint?Onesolutionistouseavariabletocontrolthewhileloop.Somethinglikethis:gen.send.preparation.stop.py

stop=False

defcounter(start=0):

n=start

whilenotstop:

yieldn

n+=1

c=counter()

print(next(c))#prints:0

print(next(c))#prints:1

stop=True

print(next(c))#raisesStopIteration

Thiswilldoit.Westartwithstop=False,anduntilwechangeittoTrue,thegeneratorwilljustkeepgoing,likebefore.ThemomentwechangestoptoTruethough,thewhileloopwillexit,andthenextcallwillraiseaStopIterationexception.Thistrickworks,butIdon’tlikeit.Wedependonanexternalvariable,andthiscanleadtoissues:whatifanotherfunctionchangesthatstop?Moreover,thecodeisscattered.Inanutshell,thisisn’tgoodenough.

Wecanmakeitbetterbyusinggenerator.send().Whenwecallgenerator.send(),thevaluethatwefeedtosendwillbepassedintothegenerator,executionisresumed,andwecanfetchitviatheyieldexpression.Thisisallverycomplicatedwhenexplainedwithwords,solet’sseeanexample:gen.send.py

defcounter(start=0):

n=start

whileTrue:

result=yieldn#A

print(type(result),result)#B

ifresult=='Q':

break

n+=1

c=counter()

print(next(c))#C

print(c.send('Wow!'))#D

print(next(c))#E

print(c.send('Q'))#F

Executionoftheprecedingcodeproducesthefollowing:

$pythongen.send.py

0

<class'str'>Wow!

1

<class'NoneType'>None

2

<class'str'>Q

Traceback(mostrecentcalllast):

File"gen.send.py",line14,in<module>

print(c.send('Q'))#F

StopIteration

Ithinkit’sworthgoingthroughthiscodelinebyline,likeifwewereexecutingit,andseeifwecanunderstandwhat’sgoingon.

Westartthegeneratorexecutionwithacalltonext(#C).Withinthegenerator,nissettothesamevalueofstart.Thewhileloopisentered,executionstops(#A)andn(0)isyieldedbacktothecaller.0isprintedontheconsole.

Wethencallsend(#D),executionresumesandresultissetto'Wow!'(still#A),thenitstypeandvalueareprintedontheconsole(#B).resultisnot'Q',thereforenisincrementedby1andexecutiongoesbacktothewhilecondition,which,beingTrue,evaluatestoTrue(thatwasn’thardtoguess,right?).Anotherloopcyclebegins,executionstopsagain(#A),andn(1)isyieldedbacktothecaller.1isprintedontheconsole.

Atthispoint,wecallnext(#E),executionisresumedagain(#A),andbecausewearenotsendinganythingtothegeneratorexplicitly,Pythonbehavesexactlylikefunctionsthatarenotusingthereturnstatement:theyieldnexpression(#A)returnsNone.resultthereforeissettoNone,anditstypeandvalueareyetagainprintedontheconsole(#B).Executioncontinues,resultisnot'Q'sonisincrementedby1,andwestartanotherloopagain.Executionstopsagain(#A)andn(2)isyieldedbacktothecaller.2isprintedontheconsole.

Andnowforthegrandfinale:wecallsendagain(#F),butthistimewepassin'Q',thereforewhenexecutionisresumed,resultissetto'Q'(#A).Itstypeandvalueareprintedontheconsole(#B),andthenfinallytheifclauseevaluatestoTrueandthewhileloopisstoppedbythebreakstatement.ThegeneratornaturallyterminatesandthismeansaStopIterationexceptionisraised.Youcanseetheprintofitstracebackonthelastfewlinesprintedontheconsole.

Thisisnotatallsimpletounderstandatfirst,soifit’snotcleartoyou,don’tbediscouraged.Youcankeepreadingonandthenyoucancomebacktothisexampleaftersometime.

Usingsendallowsforinterestingpatterns,andit’sworthnotingthatsendcanonlybeusedtoresumetheexecution,nottostartit.Onlynextstartstheexecutionofagenerator.

TheyieldfromexpressionAnotherinterestingconstructistheyieldfromexpression.Thisexpressionallowsyoutoyieldvaluesfromasubiterator.Itsuseallowsforquiteadvancedpatterns,solet’sjustseeaveryquickexampleofit:gen.yield.for.py

defprint_squares(start,end):

forninrange(start,end):

yieldn**2

forninprint_squares(2,5):

print(n)

Thepreviouscodeprintsthenumbers4,9,16ontheconsole(onseparatelines).Bynow,Iexpectyoutobeabletounderstanditbyyourself,butlet’squicklyrecapwhathappens.Theforloopoutsidethefunctiongetsaniteratorfromprint_squares(2,5)andcallsnextonituntiliterationisover.Everytimethegeneratoriscalled,executionissuspended(andlaterresumed)onyieldn**2,whichreturnsthesquareofthecurrentn.

Let’sseehowwecantransformthiscodebenefitingfromtheyieldfromexpression:gen.yield.from.py

defprint_squares(start,end):

yieldfrom(n**2forninrange(start,end))

forninprint_squares(2,5):

print(n)

Thiscodeproducesthesameresult,butasyoucanseetheyieldfromisactuallyrunningasubiterator(n**2…).Theyieldfromexpressionreturnstothecallereachvaluethesubiteratorisproducing.It’sshorteranditreadsbetter.

GeneratorexpressionsLet’snowtalkabouttheothertechniquestogeneratevaluesoneatatime.

Thesyntaxisexactlythesameaslistcomprehensions,only,insteadofwrappingthecomprehensionwithsquarebrackets,youwrapitwithroundbraces.Thatiscalledageneratorexpression.

Ingeneral,generatorexpressionsbehavelikeequivalentlistcomprehensions,butthereisoneveryimportantthingtoremember:generatorsallowforoneiterationonly,thentheywillbeexhausted.Let’sseeanexample:generator.expressions.py

>>>cubes=[k**3forkinrange(10)]#regularlist

>>>cubes

[0,1,8,27,64,125,216,343,512,729]

>>>type(cubes)

<class'list'>

>>>cubes_gen=(k**3forkinrange(10))#createasgenerator

>>>cubes_gen

<generatorobject<genexpr>at0x7ff26b5db990>

>>>type(cubes_gen)

<class'generator'>

>>>list(cubes_gen)#thiswillexhaustthegenerator

[0,1,8,27,64,125,216,343,512,729]

>>>list(cubes_gen)#nothingmoretogive

[]

Lookatthelineinwhichthegeneratorexpressioniscreatedandassignedthenamecubes_gen.Youcanseeit’sageneratorobject.Inordertoseeitselements,wecanuseaforloop,amanualsetofcallstonext,orsimply,feedittoalistconstructor,whichiswhatIdid.

Noticehow,oncethegeneratorhasbeenexhausted,thereisnowaytorecoverthesameelementsfromitagain.Weneedtorecreateit,ifwewanttouseitfromscratchagain.

Inthenextfewexamples,let’sseehowtoreproducemapandfilterusinggeneratorexpressions:gen.map.py

defadder(*n):

returnsum(n)

s1=sum(map(lambdan:adder(*n),zip(range(100),range(1,101))))

s2=sum(adder(*n)forninzip(range(100),range(1,101)))

Inthepreviousexample,s1ands2areexactlythesame:theyarethesumofadder(0,1),adder(1,2),adder(2,3),andsoon,whichtranslatestosum(1,3,5,...).Thesyntaxisdifferentthough,Ifindthegeneratorexpressiontobemuchmorereadable:gen.filter.py

cubes=[x**3forxinrange(10)]

odd_cubes1=filter(lambdacube:cube%2,cubes)

odd_cubes2=(cubeforcubeincubesifcube%2)

Inthepreviousexample,odd_cubes1andodd_cubes2arethesame:theygenerateasequenceofoddcubes.Yetagain,Ipreferthegeneratorsyntax.Thisshouldbeevidentwhenthingsgetalittlemorecomplicated:gen.map.filter.py

N=20

cubes1=map(

lambdan:(n,n**3),

filter(lambdan:n%3==0orn%5==0,range(N))

)

cubes2=(

(n,n**3)forninrange(N)ifn%3==0orn%5==0)

Theprecedingcodecreatestogeneratorscubes1andcubes2.Theyareexactlythesame,

andreturn2-tuples(n, )whennisamultipleof3or5.

Ifyouprintthelist(cubes1),youget:[(0,0),(3,27),(5,125),(6,216),(9,729),(10,1000),(12,1728),(15,3375),(18,5832)].

Seehowmuchbetterthegeneratorexpressionreads?Itmaybedebatablewhenthingsareverysimple,butassoonasyoustartnestingfunctionsabit,likewedidinthisexample,thesuperiorityofthegeneratorsyntaxisevident.Shorter,simpler,moreelegant.

Now,letmeaskyouaquestion:whatisthedifferencebetweenthefollowinglinesofcode?sum.example.py

s1=sum([n**2forninrange(10**6)])

s2=sum((n**2forninrange(10**6)))

s3=sum(n**2forninrange(10**6))

Strictlyspeaking,theyallproducethesamesum.Theexpressionstogets2ands3areexactlythesamebecausethebracesins2areredundant.Theyarebothgeneratorexpressionsinsidethesumfunction.Theexpressiontogets1isdifferentthough.Insidesum,wefindalistcomprehension.Thismeansthatinordertocalculates1,thesumfunctionhastocallnextonalist,amilliontimes.

Doyouseewherewe’reloosingtimeandmemory?Beforesumcanstartcallingnextonthatlist,thelistneedstohavebeencreated,whichisawasteoftimeandspace.It’smuchbetterforsumtocallnextonasimplegeneratorexpression.Thereisnoneedtohaveallthenumbersfromrange(10**6)storedinalist.

So,watchoutforextraparentheseswhenyouwriteyourexpressions:sometimesit’seasytoskiponthesedetails,whichmakesourcodemuchdifferent.Don’tbelieveme?sum.example.2.py

s=sum([n**2forninrange(10**8)])#thisiskilled

#s=sum(n**2forninrange(10**8))#thissucceeds

print(s)

Tryrunningtheprecedingexample.IfIrunthefirstline,thisiswhatIget:

$pythonsum.example.2.py

Killed

Ontheotherhand,ifIcommentoutthefirstline,anduncommentthesecondone,thisistheresult:

$pythonsum.example.2.py

333333328333333350000000

Sweetgeneratorexpressions.Thedifferencebetweenthetwolinesisthatinthefirstone,alistwiththesquaresofthefirsthundredmillionnumbersmustbemadebeforebeingabletosumthemup.Thatlistishuge,andwerunoutofmemory(atleast,myboxdid,ifyoursdoesn’ttryabiggernumber),thereforePythonkillstheprocessforus.Sadface.

Butwhenweremovethesquarebrackets,wedon’tmakealistanymore.Thesumfunctionreceives0,1,4,9,andsoonuntilthelastone,andsumsthemup.Noproblems,happyface.

SomeperformanceconsiderationsSo,we’veseenthatwehavemanydifferentwaystoachievethesameresult.Wecanuseanycombinationofmap,zip,filter,orchoosetogowithacomprehension,ormaybechoosetouseagenerator,eitherfunctionorexpression.Wemayevendecidetogowithforloops:whenthelogictoapplytoeachrunningparameterisn’tsimple,theymaybethebestoption.

Otherthanreadabilityconcernsthough,let’stalkaboutperformances.Whenitcomestoperformances,usuallytherearetwofactorswhichplayamajorrole:spaceandtime.

Spacemeansthesizeofthememorythatadatastructureisgoingtotakeup.Thebestwaytochooseistoaskyourselfifyoureallyneedalist(ortuple)orifasimplegeneratorfunctionwouldworkaswell.Iftheanswerisyes,gowiththegenerator,it’llsavealotofspace.Samegoeswithfunctions:ifyoudon’tactuallyneedthemtoreturnalistortuple,thenyoucantransformthemingeneratorfunctionsaswell.

Sometimes,youwillhavetouselists(ortuples),forexampletherearealgorithmsthatscansequencesusingmultiplepointersormaybetheyrunoverthesequencemorethanonce.Ageneratorfunction(orexpression)canbeiteratedoveronlyonceandthenit’sexhausted,sointhesesituations,itwouldn’tbetherightchoice.

Timeisabitharderthanspacebecauseitdependsonmorevariablesandthereforeitisn’tpossibletostatethatXisfasterthanYwithabsolutecertaintyforallcases.However,basedontestsrunonPythontoday,wecansaythatmapcallscanbetwiceasfastasequivalentforloops,andlistcomprehensionscanbe(alwaysgenerallyspeaking)evenfasterthanequivalentmapcalls.

Inordertofullyappreciatethereasonbehindthesestatements,weneedtounderstandhowPythonworks,andthisisabitoutsidethescopeofthisbook,forit’stootechnicalindetail.Let’sjustsaythatmapandlistcomprehensionsrunatClanguagespeedwithintheinterpreter,whileaPythonforloopisrunasPythonbytecodewithinthePythonVirtualMachine,whichisoftenmuchslower.

NoteThereareseveraldifferentimplementationsofPython.Theoriginalone,andstillthemostcommonone,istheonewritteninC.Cisoneofthemostpowerfulandpopularprogramminglanguagesstillusedtoday.

TheseclaimsImadecomefrombooksandarticlesthatyoucanfindontheWeb,buthowaboutwedoasmallexerciseandtrytofindoutforourselves?Iwillwriteasmallpieceofcodethatcollectstheresultsofdivmod(a,b)foracertainsetofintegerpairs(a,b).IwillusethetimefunctionfromthetimemoduletocalculatetheelapsedtimeoftheoperationsthatIwillperform.Let’sgo!performances.py

fromtimeimporttime

mx=5500#thisisthemaxIcouldreachwithmycomputer…

t=time()#starttimefortheforloop

dmloop=[]

forainrange(1,mx):

forbinrange(a,mx):

dmloop.append(divmod(a,b))

print('forloop:{:.4f}s'.format(time()-t))#elapsedtime

t=time()#starttimeforthelistcomprehension

dmlist=[

divmod(a,b)forainrange(1,mx)forbinrange(a,mx)]

print('listcomprehension:{:.4f}s'.format(time()-t))

t=time()#starttimeforthegeneratorexpression

dmgen=list(

divmod(a,b)forainrange(1,mx)forbinrange(a,mx))

print('generatorexpression:{:.4f}s'.format(time()-t))

#verifycorrectnessofresultsandnumberofitemsineachlist

print(dmloop==dmlist==dmgen,len(dmloop))

Asyoucansee,we’recreatingthreelists:dmloop,dmlist,dmgen(divmod-forloop,divmod-listcomprehension,divmod-generatorexpression).Westartwiththeslowestoption,theforloops.Thenwehavealistcomprehension,andfinallyageneratorexpression.Let’sseetheoutput:

$pythonperformances.py

forloop:4.3433s

listcomprehension:2.7238s

generatorexpression:3.1380s

True15122250

Thelistcomprehensionrunsin63%ofthetimetakenbytheforloop.That’simpressive.Thegeneratorexpressioncamequiteclosetothat,withagood72%.Thereasonthegeneratorexpressionissloweristhatweneedtofeedittothelist()constructorandthishasalittlebitmoreoverheadcomparedtoasheerlistcomprehension.

Iwouldnevergowithageneratorexpressioninasimilarcasethough,thereisnopointifattheendwewantalist.Iwouldjustusealistcomprehension,andtheresultofthepreviousexampleprovesmeright.Ontheotherhand,ifIjusthadtodothosedivmodcalculationswithoutretainingtheresults,thenageneratorexpressionwouldbethewaytogobecauseinsuchasituationalistcomprehensionwouldunnecessarilyconsumealotofspace.

So,torecap:generatorsareveryfastandallowyoutosaveonspace.Listcomprehensionsareingeneralevenfaster,butdon’tsaveonspace.PurePythonforloopsaretheslowestoption.Let’sseeasimilarexamplethatcomparesaforloopandamapcall:performances.map.py

fromtimeimporttime

mx=2*10**7

t=time()

absloop=[]

forninrange(mx):

absloop.append(abs(n))

print('forloop:{:.4f}s'.format(time()-t))

t=time()

abslist=[abs(n)forninrange(mx)]

print('listcomprehension:{:.4f}s'.format(time()-t))

t=time()

absmap=list(map(abs,range(mx)))

print('map:{:.4f}s'.format(time()-t))

print(absloop==abslist==absmap)

Thiscodeisconceptuallyverysimilartothepreviousexample.Theonlythingthathaschangedisthatwe’reapplyingtheabsfunctioninsteadofthedivmodone,andwehaveonlyoneloopinsteadoftwonestedones.Executiongivesthefollowingresult:

$pythonperformances.map.py

forloop:3.1283s

listcomprehension:1.3966s

map:1.2319s

True

Andmapwinstherace!AsItoldyoubefore,givingastatementofwhatisfasterthanwhatisverytricky.Inthiscase,themapcallisfasterthanthelistcomprehension.

Apartfromthecasebycaselittledifferencesthough,it’squiteclearthattheforloopoptionistheslowestone,solet’sseewhatarethereasonswestillwanttouseit.

Don’toverdocomprehensionsandgeneratorsWe’veseenhowpowerfullistcomprehensionsandgeneratorexpressionscanbe.Andtheyare,don’tgetmewrong,butthefeelingthatIhavewhenIdealwiththemisthattheircomplexitygrowsexponentially.Themoreyoutrytodowithinasinglecomprehensionorageneratorexpression,theharderitbecomestoread,understand,andthereforetomaintainorchange.

OpenaPythonconsoleandtypeinimportthis,let’sreadtheZenofPythonagain,inparticular,thereareafewlinesthatIthinkareveryimportanttokeepinmind:

>>>importthis

TheZenofPython,byTimPeters

Beautifulisbetterthanugly.

Explicitisbetterthanimplicit.#

Simpleisbetterthancomplex.#

Complexisbetterthancomplicated.

Flatisbetterthannested.

Sparseisbetterthandense.

Readabilitycounts.#

Specialcasesaren'tspecialenoughtobreaktherules.

Althoughpracticalitybeatspurity.

Errorsshouldneverpasssilently.

Unlessexplicitlysilenced.

Inthefaceofambiguity,refusethetemptationtoguess.

Thereshouldbeone--andpreferablyonlyone--obviouswaytodoit.

Althoughthatwaymaynotbeobviousatfirstunlessyou'reDutch.

Nowisbetterthannever.

Althoughneverisoftenbetterthan*right*now.

Iftheimplementationishardtoexplain,it'sabadidea.#

Iftheimplementationiseasytoexplain,itmaybeagoodidea.

Namespacesareonehonkinggreatidea—let'sdomoreofthose!

Ihaveputacommentsignontherightofthemainfocuspointshere.Comprehensionsandgeneratorexpressionsbecomehardtoread,moreimplicitthanexplicit,complex,andtheycanbehardtoexplain.Sometimesyouhavetobreakthemapartusingtheinside-outtechnique,tounderstandwhytheyproducetheresulttheyproduce.

Togiveyouanexample,let’stalkabitmoreaboutPythagoreantriples.Justtoremind

you,aPythagoreantripleisatupleofpositiveintegers(a,b,c)suchthat .

Wesawearlierinthischapterhowtocalculatethem,butwediditinaveryinefficientwaybecausewewerescanningallpairsofnumbersbelowacertainthreshold,calculatingthehypotenuse,andfilteringoutthosethatwerenotproducingatriple.

AbetterwaytogetalistofPythagoreantriplesistodirectlygeneratethem.Therearemanydifferentformulastodothisandwe’lluseoneofthem:theEuclideanformula.

Thisformulasaysthatanytriple(a,b,c),where ,b=2mn, ,withmandnpositiveintegerssuchthatm>n,isaPythagoreantriple.Forexample,whenm=2andn=1,wefindthesmallesttriple:(3,4,5).

Thereisonecatchthough:considerthetriple(6,8,10),thatisjustlike(3,4,5)withall

thenumbersmultipliedby2.ThistripleisdefinitelyPythagorean,since ,butwecanderiveitfrom(3,4,5)simplybymultiplyingeachofitselementsby2.Samegoesfor(9,12,15),(12,16,20),andingeneralforallthetriplesthatwecanwriteas(3k,4k,5k),withkbeingapositiveintegergreaterthan1.

Atriplethatcannotbeobtainedbymultiplyingtheelementsofanotheronebysomefactork,iscalledprimitive.Anotherwayofstatingthisis:ifthethreeelementsofatriplearecoprime,thenthetripleisprimitive.Twonumbersarecoprimewhentheydon’tshareanyprimefactoramongsttheirdivisors,thatis,theirgreatestcommondivisor(GCD)is1.Forexample,3and5arecoprime,while3and6arenot,becausetheyarebothdivisibleby3.

So,theEuclideanformulatellsusthatifmandnarecoprime,andm–nisodd,thetripletheygenerateisprimitive.Inthefollowingexample,wewillwriteageneratorexpressiontocalculatealltheprimitivePythagoreantripleswhosehypotenuse(c)islessthanorequal

tosomeintegerN.Thismeanswewantalltriplesforwhich .Whennis1,the

formulalookslikethis: ,whichmeanswecanapproximatethecalculationwith

anupperboundof .

So,torecap:mmustbegreaterthann,theymustalsobecoprime,andtheirdifferencem-nmustbeodd.Moreover,inordertoavoiduselesscalculationswe’llputtheupperboundformatfloor(sqrt(N))+1.

NoteThefunctionfloorforarealnumberxgivesthemaximumintegernsuchthatn<x,forexample,floor(3.8)=3,floor(13.1)=13.Takingthefloor(sqrt(N))+1meanstakingtheintegerpartofthesquarerootofNandaddingaminimalmarginjusttomakesurewedon’tmissoutanynumber.

Let’sputallofthisintocode,stepbystep.Let’sstartbywritingasimplegcdfunctionthatusesEuclid’salgorithm:functions.py

defgcd(a,b):

"""CalculatetheGreatestCommonDivisorof(a,b)."""

whileb!=0:

a,b=b,a%b

returna

TheexplanationofEuclid’salgorithmisavailableontheWeb,soIwon’tspendanytimeheretalkingaboutit;weneedtoconcentrateonthegeneratorexpression.ThenextstepistousetheknowledgewegatheredbeforetogeneratealistofprimitivePythagorean

triples:pythagorean.triple.generation.py

fromfunctionsimportgcd

N=50

triples=sorted(#1

((a,b,c)fora,b,cin(#2

((m**2-n**2),(2*m*n),(m**2+n**2))#3

forminrange(1,int(N**.5)+1)#4

forninrange(1,m)#5

if(m-n)%2andgcd(m,n)==1#6

)ifc<=N),key=lambda*triple:sum(*triple)#7

)

print(triples)

Thereyougo.It’snoteasytoread,solet’sgothroughitlinebyline.At#3,westartageneratorexpressionthatiscreatingtriples.Youcanseefrom#4and#5thatwe’reloopingonmin[1,M]withMbeingtheintegerpartofsqrt(N),plus1.Ontheotherhand,nloopswithin[1,m),torespectthem>nrule.WorthnotinghowIcalculatedsqrt(N),thatis,N**.5,whichisjustanotherwaytodoitthatIwantedtoshowyou.

At#6,youcanseethefilteringconditionstomakethetriplesprimitive:(m-n)%2evaluatestoTruewhen(m-n)isodd,andgcd(m,n)==1meansmandnarecoprime.Withtheseinplace,weknowthetripleswillbeprimitive.Thistakescareoftheinnermostgeneratorexpression.Theoutermostonestartsat#2,andfinishesat#7.Wetakethetriples(a,b,c)in(…innermostgenerator…)suchthatc<=N.Thisisnecessarybecause

isthelowestupperboundthatwecanapply,butitdoesn’tguaranteethatcwillactuallybelessthanorequaltoN.

Finally,at#1weapplysorting,topresentthelistinorder.At#7,aftertheoutermostgeneratorexpressionisclosed,youcanseethatwespecifythesortingkeytobethesuma+b+c.Thisisjustmypersonalpreference,thereisnomathematicalreasonbehindit.

So,whatdoyouthink?Wasitstraightforwardtoread?Idon’tthinkso.Andbelieveme,thisisstillasimpleexample;Ihaveseenexpressionswaymorecomplicatedthanthisone.

Unfortunatelysomeprogrammersthinkthatwritingcodelikethisiscool,thatit’ssomesortofdemonstrationoftheirsuperiorintellectualpowers,oftheirabilitytoquicklyreadanddigestintricatecode.

Withinaprofessionalenvironmentthough,Ifindmyselfhavingmuchmorerespectforthosewhowriteefficient,cleancode,andmanagetokeepegooutthedoor.Conversely,thosewhodon’t,willproducelinesatwhichyouwillstareforalongtimewhileswearinginthreelanguages(atleastthisiswhatIdo).

Now,let’sseeifwecanrewritethiscodeintosomethingeasiertoread:pythagorean.triple.generation.for.py

fromfunctionsimportgcd

defgen_triples(N):

forminrange(1,int(N**.5)+1):#1

forninrange(1,m):#2

if(m-n)%2andgcd(m,n)==1:#3

c=m**2+n**2#4

ifc<=N:#5

a=m**2-n**2#6

b=2*m*n#7

yield(a,b,c)#8

triples=sorted(

gen_triples(50),key=lambda*triple:sum(*triple))#9

print(triples)

Ifeelsomuchbetteralready.Let’sgothroughthiscodeaswell,linebyline.You’llseehoweasieritistounderstand.

Westartloopingat#1and#2,inexactlythesamewaywewereloopinginthepreviousexample.Online#3,wehavethefilteringforprimitivetriples.Online#4,wedeviateabitfromwhatweweredoingbefore:wecalculatec,andonline#5,wefilteroncbeinglessthanorequaltoN.Onlywhencsatisfiesthatcondition,wecalculateaandb,andyieldtheresultingtuple.It’salwaysgoodtodelayallcalculationsforasmuchaspossiblesothatwedon’twastetime,incaseeventuallywehavetodiscardthoseresults.

Onthelastline,beforeprintingtheresult,weapplysortingwiththesamekeywewereusinginthegeneratorexpressionexample.

Ihopeyouagree,thisexampleiseasiertounderstand.AndIpromiseyou,ifyouhavetomodifythecodeoneday,you’llfindthatmodifyingthisoneiseasy,whiletomodifytheotherversionwilltakemuchlonger(anditwillbemoreerrorprone).

Bothexamples,whenrun,printthefollowing:

$pythonpythagorean.triple.generation.py

[(3,4,5),(5,12,13),(15,8,17),(7,24,25),(21,20,29),(35,12,

37),(9,40,41)]

Themoralofthestoryis,tryandusecomprehensionsandgeneratorexpressionsasmuchasyoucan,butifthecodestartstobecomplicatedtomodifyortoread,youmaywanttorefactorintosomethingmorereadable.Thereisnothingwrongwiththis.

NamelocalizationNowthatwearefamiliarwithalltypesofcomprehensionsandgeneratorexpression,let’stalkaboutnamelocalizationwithinthem.Python3.*localizesloopvariablesinallfourformsofcomprehensions:list,dict,set,andgeneratorexpressions.Thisbehavioristhereforedifferentfromthatoftheforloop.Let’sseeasimpleexampletoshowallthecases:scopes.py

A=100

ex1=[AforAinrange(5)]

print(A)#prints:100

ex2=list(AforAinrange(5))

print(A)#prints:100

ex3=dict((A,2*A)forAinrange(5))

print(A)#prints:100

ex4=set(AforAinrange(5))

print(A)#prints:100

s=0

forAinrange(5):

s+=A

print(A)#prints:4

Intheprecedingcode,wedeclareaglobalnameA=100,andthenweexercisethefourcomprehensions:list,generatorexpression,dictionary,andset.NoneofthemaltertheglobalnameA.Conversely,youcanseeattheendthattheforloopmodifiesit.Thelastprintstatementprints4.

Let’sseewhathappensifAwasn’tthere:scopes.noglobal.py

ex1=[AforAinrange(5)]

print(A)#breaks:NameError:name'A'isnotdefined

Theprecedingcodewouldworkthesamewithanyofthefourtypesofcomprehensions.Afterwerunthefirstline,Aisnotdefinedintheglobalnamespace.

Onceagain,theforloopbehavesdifferently:scopes.for.py

s=0

forAinrange(5):

s+=A

print(A)#prints:4

print(globals())

Theprecedingcodeshowsthatafteraforloop,iftheloopvariablewasn’tdefinedbeforeit,wecanfinditintheglobalframe.Tomakesureofit,let’stakeapeekatitbycalling

theglobals()built-infunction:

$pythonscopes.for.py

4

{'__spec__':None,'__name__':'__main__','s':10,'A':4,'__doc__':

None,'__cached__':None,'__package__':None,'__file__':'scopes.for.py',

'__loader__':<_frozen_importlib.SourceFileLoaderobjectat

0x7f05a5a183c8>,'__builtins__':<module'builtins'(built-in)>}

Togetherwithalotofotherboilerplatestuff,wecanspot'A':4.

Generationbehaviorinbuilt-insAmongstthebuilt-intypes,thegenerationbehaviorisnowquitecommon.ThisisamajordifferencebetweenPython2andPython3.Alotoffunctionssuchasmap,zip,andfilterhavebeentransformedsothattheyreturnobjectsthatbehavelikeiterables.Theideabehindthischangeisthatifyouneedtomakealistofthoseresultsyoucanalwayswrapthecallinalist()class,andyou’redone.Ontheotherhand,ifyoujustneedtoiterateandwanttokeeptheimpactonmemoryaslightaspossible,youcanusethosefunctionssafely.

Anothernotableexampleistherangefunction.InPython2itreturnsalist,andthereisanotherfunctioncalledxrangethatreturnsanobjectthatyoucaniterateon,whichgeneratesthenumbersonthefly.InPython3thisfunctionhasgone,andrangenowbehaveslikeit.

Butthisconceptingeneralisnowquitewidespread.Youcanfinditintheopen()function,whichisusedtooperateonfileobjects(we’llseeitinoneofthenextchapters),butalsoinenumerate,inthedictionarykeys,values,anditemsmethods,andseveralotherplaces.

Itallmakessense:Python’saimistotryandreducethememoryfootprintbyavoidingwastingspacewhereverispossible,especiallyinthosefunctionsandmethodsthatareusedextensivelyinmostsituations.

Doyourememberatthebeginningofthischapter?Isaidthatitmakesmoresensetooptimizetheperformancesofcodethathastodealwithalotofobjects,ratherthanshavingoffafewmillisecondsfromafunctionthatwecalltwiceaday.

OnelastexampleBeforewepartfromthischapter,I’llshowyouasimpleproblemthatIsubmittedtocandidatesforaPythondeveloperroleinacompanyIusedtoworkfor.

Theproblemisthefollowing:giventhesequence01123581321…writeafunctionthatwouldreturnthetermsofthissequenceuptosomelimitN.

Ifyouhaven’trecognizedit,thatistheFibonaccisequence,whichisdefinedasF(0)=0,F(1)=1and,foranyn>1,F(n)=F(n-1)+F(n-2).Thissequenceisexcellenttotestknowledgeaboutrecursion,memoizationtechniquesandothertechnicaldetails,butinthiscaseitwasagoodopportunitytocheckwhetherthecandidateknewaboutgenerators(andtoomanysocalledPythoncodersdidn’t,whenIwasinterviewingthem).

Let’sstartfromarudimentaryversionofafunction,andthenimproveonit:fibonacci.first.py

deffibonacci(N):

"""ReturnallfibonaccinumbersuptoN."""

result=[0]

next_n=1

whilenext_n<=N:

result.append(next_n)

next_n=sum(result[-2:])

returnresult

print(fibonacci(0))#[0]

print(fibonacci(1))#[0,1,1]

print(fibonacci(50))#[0,1,1,2,3,5,8,13,21,34]

Fromthetop:wesetuptheresultlisttoastartingvalueof[0].Thenwestarttheiterationfromthenextelement(next_n),whichis1.WhilethenextelementisnotgreaterthanN,wekeepappendingittothelistandcalculatingthenext.Wecalculatethenextelementbytakingasliceofthelasttwoelementsintheresultlistandpassingittothesumfunction.Addsomeprintstatementshereandthereifthisisnotcleartoyou,butbynowIwouldexpectitnottobeanissue.

WhentheconditionofthewhileloopevaluatestoFalse,weexittheloopandreturnresult.Youcanseetheresultofthoseprintstatementsinthecommentsnexttoeachofthem.

Atthispoint,Iwouldaskthecandidatethefollowingquestion:“WhatifIjustwantedtoiterateoverthosenumbers?”Agoodcandidatewouldthenchangethecodelikethenextlisting(anexcellentcandidatewouldhavestartedwithit!):fibonacci.second.py

deffibonacci(N):

"""ReturnallfibonaccinumbersuptoN."""

yield0

ifN==0:

return

a=0

b=1

whileb<=N:

yieldb

a,b=b,a+b

print(list(fibonacci(0)))#[0]

print(list(fibonacci(1)))#[0,1,1]

print(list(fibonacci(50)))#[0,1,1,2,3,5,8,13,21,34]

ThisisactuallyoneofthesolutionsIwasgiven.Idon’tknowwhyIkeptit,butI’mgladIdidsoIcanshowittoyou.Now,thefibonaccifunctionisageneratorfunction.Firstweyield0,thenifNis0wereturn(thiswillcauseaStopIterationexceptiontoberaised).Ifthat’snotthecase,westartiterating,yieldingbateveryloopcycle,andthenupdatingaandb.Allweneedinordertobeabletoproducethenextelementofthesequenceisthepasttwo:aandb,respectively.

Thiscodeismuchbetter,hasalightermemoryfootprintandallwehavetodotogetalistofFibonaccinumbersistowrapthecallwithlist(),asusual.

Butwhataboutelegance?Icannotleavethecodelikethat.Itwasdecentforaninterview,wherethefocusismoreonfunctionalitythanelegance,buthereI’dliketoshowyouanicerversion:fibonacci.elegant.py

deffibonacci(N):

"""ReturnallfibonaccinumbersuptoN."""

a,b=0,1

whilea<=N:

yielda

a,b=b,a+b

Muchbetter.Thewholebodyofthefunctionisfourlines,fiveifyoucountthedocstring.Noticehowinthiscaseusingtupleassignment(a,b=0,1anda,b=b,a+b)helpsinmakingthecodeshorter,andmorereadable.It’soneofthefeaturesofPythonIlikealot.

SummaryInthischapter,weexploredtheconceptofiterationandgenerationabitmoredeeply.Wesawthemap,zipandfilterfunctionsquiteindetail,andhowtousethemasanalternativetoaregularforloopapproach.

Thenwesawtheconceptofcomprehensions,forlists,dictionaries,andsets.Wesawtheirsyntaxandhowtousethemasanalternativetoboththeclassicforloopapproachandalsototheuseofmap,zip,andfilterfunctions.

Finally,wetalkedabouttheconceptofgeneration,intwoforms:generatorfunctionsandexpressions.Welearnedhowtosavetimeandspacebyusinggenerationtechniquesandsawhowtheycanmakepossiblewhatwouldn’tnormallybeifweusedaconventionalapproachbasedonlists.

Wetalkedaboutperformances,andsawthatforloopsarelastintermsofspeed,buttheyprovidethebestreadabilityandflexibilitytochange.Ontheotherhand,functionssuchasmapandfiltercanbemuchfaster,andcomprehensionsmaybeevenbetter.

Thecomplexityofthecodewrittenusingthesetechniquesgrowsexponentiallyso,inordertofavorreadabilityandeaseofmaintainability,westillneedtousetheclassicforloopapproachattimes.Anotherdifferenceisinthenamelocalization,wheretheforloopbehavesdifferentlyfromallothertypesofcomprehensions.

Thenextchapterwillbeallaboutobjectsandclasses.Structurallysimilartothisone,inthatwewon’texploremanydifferentsubjects,rather,justafewofthem,butwe’lltrytodivealittlebitmoredeeply.

Makesureyouunderstandwelltheconceptsofthischapterbeforejumpingtothenextone.We’rebuildingawallbrickbybrick,andifthefoundationisnotsolid,wewon’tgetveryfar.

Chapter6.AdvancedConcepts–OOP,Decorators,andIterators “Laclassenonèacqua.(Classwillout)”

—Italiansaying

Icouldprobablywriteasmallbookaboutobject-orientedprogramming(referredtoasOOPhenceforth)andclasses.Inthischapter,I’mfacingthehardchallengeoffindingthebalancebetweenbreadthanddepth.Therearesimplytoomanythingstotell,andthere’splentyofthemthatwouldtakemorethanthiswholechapterifIdescribedthemaloneindepth.Therefore,IwilltrytogiveyouwhatIthinkisagoodpanoramicviewofthefundamentals,plusafewthingsthatmaycomeinhandyinthenextchapters.Python’sofficialdocumentationwillhelpinfillingthegaps.

We’regoingtoexplorethreeimportantconceptsinthischapter:decorators,OOP,anditerators.

DecoratorsInthepreviouschapter,Imeasuredtheexecutiontimeofvariousexpressions.Ifyourecall,Ihadtoinitializeavariabletothestarttime,andsubtractitfromthecurrenttimeafterexecutioninordertocalculatetheelapsedtime.Ialsoprinteditontheconsoleaftereachmeasurement.Thatwasverytedious.

Everytimeyoufindyourselfrepeatingthings,analarmbellshouldgooff.Canyouputthatcodeinafunctionandavoidrepetition?Theanswermostofthetimeisyes,solet’slookatanexample.decorators/time.measure.start.py

fromtimeimportsleep,time

deff():

sleep(.3)

defg():

sleep(.5)

t=time()

f()

print('ftook:',time()-t)#ftook:0.3003859519958496

t=time()

g()

print('gtook:',time()-t)#gtook:0.5005719661712646

Intheprecedingcode,Idefinedtwofunctions,fandg,whichdonothingbutsleep(by0.3and0.5secondsrespectively).Iusedthesleepfunctiontosuspendtheexecutionforthedesiredamountoftime.Ialsohighlightedhowwecalculatethetimeelapsedbysettingttothecurrenttimeandthensubtractingitwhenthetaskisdone.Youcanseethatthemeasureisprettyaccurate.

Now,howdoweavoidrepeatingthatcodeandthosecalculations?Onefirstpotentialapproachcouldbethefollowing:decorators/time.measure.dry.py

fromtimeimportsleep,time

deff():

sleep(.3)

defg():

sleep(.5)

defmeasure(func):

t=time()

func()

print(func.__name__,'took:',time()-t)

measure(f)#ftook:0.30041074752807617

measure(g)#gtook:0.5006198883056641

Ah,muchbetternow.Thewholetimingmechanismhasbeenencapsulatedintoafunctionsowedon’trepeatcode.Weprintthefunctionnamedynamicallyandit’seasyenoughtocode.Whatifweneedtopassargumentstothefunctionwemeasure?Thiscodewouldgetjustabitmorecomplicated,solet’sseeanexample.decorators/time.measure.arguments.py

fromtimeimportsleep,time

deff(sleep_time=0.1):

sleep(sleep_time)

defmeasure(func,*args,**kwargs):

t=time()

func(*args,**kwargs)

print(func.__name__,'took:',time()-t)

measure(f,sleep_time=0.3)#ftook:0.3004162311553955

measure(f,0.2)#ftook:0.20028162002563477

Now,fisexpectingtobefedsleep_time(withadefaultvalueof0.1).Ialsohadtochangethemeasurefunctionsothatitisnowacceptingafunction,anyvariablepositionalarguments,andanyvariablekeywordarguments.Inthisway,whateverwecallmeasurewith,weredirectthoseargumentstothecalltofwedoinside.

Thisisverygood,butwecanpushitalittlebitfurther.Let’ssaywewanttosomehowhavethattimingbehaviorbuilt-inintheffunction,sothatwecouldjustcallitandhavethatmeasuretaken.Here’showwecoulddoit:decorators/time.measure.deco1.py

fromtimeimportsleep,time

deff(sleep_time=0.1):

sleep(sleep_time)

defmeasure(func):

defwrapper(*args,**kwargs):

t=time()

func(*args,**kwargs)

print(func.__name__,'took:',time()-t)

returnwrapper

f=measure(f)#decorationpoint

f(0.2)#ftook:0.2002875804901123

f(sleep_time=0.3)#ftook:0.3003721237182617

print(f.__name__)#wrapper<-ouch!

Theprecedingcodeisprobablynotsostraightforward.Iconfessthat,eventoday,itsometimesrequiresmesomeseriousconcentrationtounderstandsomedecorators,theycanbeprettynasty.Let’sseewhathappenshere.Themagicisinthedecorationpoint.Webasicallyreassignfwithwhateverisreturnedbymeasurewhenwecallitwithfasan

argument.Withinmeasure,wedefineanotherfunction,wrapper,andthenwereturnit.So,theneteffectisthatafterthedecorationpoint,whenwecallf,we’reactuallycallingwrapper.Sincethewrapperinsideiscallingfunc,whichisf,weareactuallyclosingthelooplikethat.Ifyoudon’tbelieveme,takealookatthelastline.

wrapperisactually…awrapper.Ittakesvariableandpositionalarguments,andcallsfwiththem.Italsodoesthetimemeasurementtrickaroundthecall.

Thistechniqueiscalleddecoration,andmeasureis,atalleffects,adecorator.Thisparadigmbecamesopopularandwidelyusedthatatsomepoint,Pythonaddedaspecialsyntaxforit(checkPEP318).Let’sexplorethreecases:onedecorator,twodecorators,andonedecoratorthattakesarguments.decorators/syntax.py

deffunc(arg1,arg2,...):

pass

func=decorator(func)

#isequivalenttothefollowing:

@decorator

deffunc(arg1,arg2,...):

pass

Basically,insteadofmanuallyreassigningthefunctiontowhatwasreturnedbythedecorator,weprependthedefinitionofthefunctionwiththespecialsyntax@decorator_name.

Wecanapplymultipledecoratorstothesamefunctioninthefollowingway:decorators/syntax.py

deffunc(arg1,arg2,...):

pass

func=deco1(deco2(func))

#isequivalenttothefollowing:

@deco1

@deco2

deffunc(arg1,arg2,...):

pass

Whenapplyingmultipledecorators,payattentiontotheorder,shoulditmatter.Intheprecedingexample,funcisdecoratedwithdeco2first,andtheresultisdecoratedwithdeco1.Agoodruleofthumbis:thecloserthedecoratortothefunction,thesooneritisapplied.

Somedecoratorscantakearguments.Thistechniqueisgenerallyusedtoproduceotherdecorators.Let’slookatthesyntax,andthenwe’llseeanexampleofit.decorators/syntax.py

deffunc(arg1,arg2,...):

pass

func=decoarg(argA,argB)(func)

#isequivalenttothefollowing:

@decoarg(argA,argB)

deffunc(arg1,arg2,...):

pass

Asyoucansee,thiscaseisabitdifferent.Firstdecoargiscalledwiththegivenarguments,andthenitsreturnvalue(theactualdecorator)iscalledwithfunc.BeforeIgiveyouanotherexample,let’sfixonethingthatisbotheringme.Idon’twanttolosetheoriginalfunctionnameanddocstring(andtheotherattributesaswell,checkthedocumentationforthedetails)whenIdecorateit.Butbecauseinsideourdecoratorwereturnwrapper,theoriginalattributesfromfuncarelostandfendsupbeingassignedtheattributesofwrapper.Thereisaneasyfixforthatfromfunctools,awonderfulmodulefromthePythonstandardlibrary.Iwillfixthelastexample,andIwillalsorewriteitssyntaxtousethe@operator.decorators/time.measure.deco2.py

fromtimeimportsleep,time

fromfunctoolsimportwraps

defmeasure(func):

@wraps(func)

defwrapper(*args,**kwargs):

t=time()

func(*args,**kwargs)

print(func.__name__,'took:',time()-t)

returnwrapper

@measure

deff(sleep_time=0.1):

"""I'macat.Ilovetosleep!"""

sleep(sleep_time)

f(sleep_time=0.3)#ftook:0.30039525032043457

print(f.__name__,':',f.__doc__)

#f:I'macat.Ilovetosleep!

Nowwe’retalking!Asyoucansee,allweneedtodoistotellPythonthatwrapperactuallywrapsfunc(bymeansofthewrapsfunction),andyoucanseethattheoriginalnameanddocstringarenowmaintained.

Let’sseeanotherexample.Iwantadecoratorthatprintsanerrormessagewhentheresultofafunctionisgreaterthanathreshold.Iwillalsotakethisopportunitytoshowyouhowtoapplytwodecoratorsatonce.decorators/two.decorators.py

fromtimeimportsleep,time

fromfunctoolsimportwraps

defmeasure(func):

@wraps(func)

defwrapper(*args,**kwargs):

t=time()

result=func(*args,**kwargs)

print(func.__name__,'took:',time()-t)

returnresult

returnwrapper

defmax_result(func):

@wraps(func)

defwrapper(*args,**kwargs):

result=func(*args,**kwargs)

ifresult>100:

print('Resultistoobig({0}).Maxallowedis100.'

.format(result))

returnresult

returnwrapper

@measure

@max_result

defcube(n):

returnn**3

print(cube(2))

print(cube(5))

TipTakeyourtimeinstudyingtheprecedingexampleuntilyouaresureyouunderstanditwell.Ifyoudo,Idon’tthinkthereisanydecoratoryouwon’tbeabletowriteafterwards.

Ihadtoenhancethemeasuredecorator,sothatitswrappernowreturnstheresultofthecalltofunc.Themax_resultdecoratordoesthataswell,butbeforereturning,itchecksthatresultisnotgreaterthan100,whichisthemaximumallowed.

Idecoratedcubewithbothofthem.First,max_resultisapplied,thenmeasure.Runningthiscodeyieldsthisresult:

$pythontwo.decorators.py

cubetook:7.62939453125e-06#

8#

Resultistoobig(125).Maxallowedis100.

cubetook:1.1205673217773438e-05

125

Foryourconvenience,Iputa#totherightoftheresultsofthefirstcall:print(cube(2)).Theresultis8,andthereforeitpassesthethresholdchecksilently.Therunningtimeismeasuredandprinted.Finally,weprinttheresult(8).

Onthesecondcall,theresultis125,sotheerrormessageisprinted,theresultreturned,andthenit’stheturnofmeasure,whichprintstherunningtimeagain,andfinally,weprinttheresult(125).

HadIdecoratedthecubefunctionwiththesametwodecoratorsbutinadifferentorder,theerrormessagewouldfollowthelinethatprintstherunningtime,insteadofpreceding

it.

AdecoratorfactoryLet’ssimplifythisexamplenow,goingbacktoasingledecorator:max_result.IwanttomakeitsothatIcandecoratedifferentfunctionswithdifferentthresholds,andIdon’twanttowriteonedecoratorforeachthreshold.Let’samendmax_resultsothatitallowsustodecoratefunctionsspecifyingthethresholddynamically.decorators/decorators.factory.py

fromfunctoolsimportwraps

defmax_result(threshold):

defdecorator(func):

@wraps(func)

defwrapper(*args,**kwargs):

result=func(*args,**kwargs)

ifresult>threshold:

print(

'Resultistoobig({0}).Maxallowedis{1}.'

.format(result,threshold))

returnresult

returnwrapper

returndecorator

@max_result(75)

defcube(n):

returnn**3

print(cube(5))

Thisprecedingcodeshowsyouhowtowriteadecoratorfactory.Ifyourecall,decoratingafunctionwithadecoratorthattakesargumentsisthesameaswritingfunc=decorator(argA,argB)(func),sowhenwedecoratecubewithmax_result(75),we’redoingcube=max_result(75)(cube).

Let’sgothroughwhathappens,stepbystep.Whenwecallmax_result(75),weenteritsbody.Adecoratorfunctionisdefinedinside,whichtakesafunctionasitsonlyargument.Insidethatfunction,theusualdecoratortrickisperformed.Wedefineawrapper,insideofwhichwechecktheresultoftheoriginalfunction’scall.Thebeautyofthisapproachisthatfromtheinnermostlevel,wecanstillrefertobothfuncandthreshold,whichallowsustosetthethresholddynamically.

wrapperreturnsresult,decoratorreturnswrapper,andmax_resultreturnsdecorator.Thismeansthatourcallcube=max_result(75)(cube),actuallybecomescube=decorator(cube).Notjustanydecoratorthough,butoneforwhichthresholdhasthevalue75.Thisisachievedbyamechanismcalledclosure,whichisoutsideofthescopeofthischapterbutnonethelessveryinteresting,soImentioneditforyoutodosomeresearchonit.

Runningthelastexampleproducesthefollowingresult:

$pythondecorators.factory.py

Resultistoobig(125).Maxallowedis75.

125

Theprecedingcodeallowsmetousethemax_resultdecoratorwithdifferentthresholdsatmyownwill,likethis:decorators/decorators.factory.py

@max_result(75)

defcube(n):

returnn**3

@max_result(100)

defsquare(n):

returnn**2

@max_result(1000)

defmultiply(a,b):

returna*b

Notethateverydecorationusesadifferentthresholdvalue.

DecoratorsareverypopularinPython.Theyareusedquiteoftenandtheysimplify(andbeautify,Idaresay)thecodealot.

Object-orientedprogrammingIt’sbeenquitealongandhopefullynicejourneyand,bynow,weshouldbereadytoexploreobject-orientedprogramming.I’llusethedefinitionfromKindler,E.;Krivy,I.(2011).Object-OrientedSimulationofsystemswithsophisticatedcontrol.InternationalJournalofGeneralSystems,andadaptittoPython:

Object-orientedprogramming(OOP)isaprogrammingparadigmbasedontheconceptof“objects”,whicharedatastructuresthatcontaindata,intheformofattributes,andcode,intheformoffunctionsknownasmethods.Adistinguishingfeatureofobjectsisthatanobject’smethodcanaccessandoftenmodifythedataattributesoftheobjectwithwhichtheyareassociated(objectshaveanotionof“self”).InOOprogramming,computerprogramsaredesignedbymakingthemoutofobjectsthatinteractwithoneanother.

Pythonhasfullsupportforthisparadigm.Actually,aswehavealreadysaid,everythinginPythonisanobject,sothisshowsthatOOPisnotjustsupportedbyPython,butit’spartofitsverycore.

ThetwomainplayersinOOPareobjectsandclasses.Classesareusedtocreateobjects(objectsareinstancesoftheclasseswithwhichtheywerecreated),sowecouldseethemasinstancefactories.Whenobjectsarecreatedbyaclass,theyinherittheclassattributesandmethods.Theyrepresentconcreteitemsintheprogram’sdomain.

ThesimplestPythonclassIwillstartwiththesimplestclassyoucouldeverwriteinPython.oop/simplest.class.py

classSimplest():#whenempty,thebracesareoptional

pass

print(type(Simplest))#whattypeisthisobject?

simp=Simplest()#wecreateaninstanceofSimplest:simp

print(type(simp))#whattypeissimp?

#issimpaninstanceofSimplest?

print(type(simp)==Simplest)#There'sabetterwayforthis

Let’sruntheprecedingcodeandexplainitlinebyline:

$pythonoop/simplest.class.py

<class'type'>

<class'__main__.Simplest'>

True

TheSimplestclassIdefinedonlyhasthepassinstructionforitsbody,whichmeansitdoesn’thaveanycustomattributesormethods.Iwillprintitstype(__main__isthenameofthescopeinwhichtop-levelcodeexecutes),andIamawarethat,inthecomment,Iwroteobjectinsteadofclass.Itturnsoutthat,asyoucanseebytheresultofthatprint,classesareactuallyobjects.Tobeprecise,theyareinstancesoftype.Explainingthisconceptwouldleadtoatalkaboutmetaclassesandmetaprogramming,advancedconceptsthatrequireasolidgraspofthefundamentalstobeunderstoodandalasthisisbeyondthescopeofthischapter.Asusual,Imentionedittoleaveapointerforyou,forwhenyou’llbereadytodigdeeper.

Let’sgobacktotheexample:IusedSimplesttocreateaninstance,simp.Youcanseethatthesyntaxtocreateaninstanceisthesameweusetocallafunction.

ThenweprintwhattypesimpbelongstoandweverifythatsimpisinfactaninstanceofSimplest.I’llshowyouabetterwayofdoingthislateroninthechapter.

Uptonow,it’sallverysimple.WhathappenswhenwewriteclassClassName():pass,though?Well,whatPythondoesiscreateaclassobjectandassignitaname.Thisisverysimilartowhathappenswhenwedeclareafunctionusingdef.

ClassandobjectnamespacesAftertheclassobjecthasbeencreated(whichusuallyhappenswhenthemoduleisfirstimported),itbasicallyrepresentsanamespace.Wecancallthatclasstocreateitsinstances.Eachinstanceinheritstheclassattributesandmethodsandisgivenitsownnamespace.Wealreadyknowthat,towalkanamespace,allweneedtodoistousethedot(.)operator.

Let’slookatanotherexample:oop/class.namespaces.py

classPerson():

species='Human'

print(Person.species)#Human

Person.alive=True#Addeddynamically!

print(Person.alive)#True

man=Person()

print(man.species)#Human(inherited)

print(man.alive)#True(inherited)

Person.alive=False

print(man.alive)#False(inherited)

man.name='Darth'

man.surname='Vader'

print(man.name,man.surname)#DarthVader

Intheprecedingexample,Ihavedefinedaclassattributecalledspecies.Anyvariabledefinedinthebodyofaclassisanattributethatbelongstothatclass.Inthecode,IhavealsodefinedPerson.alive,whichisanotherclassattribute.Youcanseethatthereisnorestrictiononaccessingthatattributefromtheclass.Youcanseethatman,whichisaninstanceofPerson,inheritsbothofthem,andreflectstheminstantlywhentheychange.

manhasalsotwoattributeswhichbelongtoitsownnamespaceandthereforearecalledinstanceattributes:nameandsurname.

NoteClassattributesaresharedamongstallinstances,whileinstanceattributesarenot;therefore,youshoulduseclassattributestoprovidethestatesandbehaviorstobesharedbyallinstances,anduseinstanceattributesfordatathatbelongsjusttoonespecificobject.

AttributeshadowingWhenyousearchforanattributeinanobject,ifitisnotfound,Pythonkeepssearchingintheclassthatwasusedtocreatethatobject(andkeepssearchinguntilit’seitherfoundortheendoftheinheritancechainisreached).Thisleadstoaninterestingshadowingbehavior.Let’slookatanexample:oop/class.attribute.shadowing.py

classPoint():

x=10

y=7

p=Point()

print(p.x)#10(fromclassattribute)

print(p.y)#7(fromclassattribute)

p.x=12#pgetsitsown'x'attribute

print(p.x)#12(nowfoundontheinstance)

print(Point.x)#10(classattributestillthesame)

delp.x#wedeleteinstanceattribute

print(p.x)#10(nowsearchhastogoagaintofindclassattr)

p.z=3#let'smakeita3Dpoint

print(p.z)#3

print(Point.z)

#AttributeError:typeobject'Point'hasnoattribute'z'

Theprecedingcodeisveryinteresting.WehavedefinedaclasscalledPointwithtwoclassattributes,xandy.Whenwecreateaninstance,p,youcanseethatwecanprintbothxandyfromp‘snamespace(p.xandp.y).WhathappenswhenwedothatisthatPythondoesn’tfindanyxoryattributesontheinstance,andthereforesearchestheclass,andfindsthemthere.

Thenwegivepitsownxattributebyassigningp.x=12.Thisbehaviormayappearabitweirdatfirst,butifyouthinkaboutit,it’sexactlythesameaswhathappensinafunctionthatdeclaresx=12whenthereisaglobalx=10outside.Weknowthatx=12won’taffecttheglobalone,andforclassesandinstances,itisexactlythesame.

Afterassigningp.x=12,whenweprintit,thesearchdoesn’tneedtoreadtheclassattributes,becausexisfoundontheinstance,thereforeweget12printedout.

WealsoprintPoint.xwhichreferstoxintheclassnamespace.

Andthen,wedeletexfromthenamespaceofp,whichmeansthat,onthenextline,whenweprintitagain,Pythonwillgoagainandsearchforitintheclass,becauseitwon’tbefoundintheinstanceanymore.

Thelastthreelinesshowyouthatassigningattributestoaninstancedoesn’tmeanthattheywillbefoundintheclass.Instancesgetwhateverisintheclass,buttheoppositeisnottrue.

Whatdoyouthinkaboutputtingthexandycoordinatesasclassattributes?Doyouthinkitwasagoodidea?

I,me,andmyself–usingtheselfvariableFromwithinaclassmethodwecanrefertoaninstancebymeansofaspecialargument,calledselfbyconvention.selfisalwaysthefirstattributeofaninstancemethod.Let’sexaminethisbehaviortogetherwithhowwecanshare,notjustattributes,butmethodswithallinstances.oop/class.self.py

classSquare():

side=8

defarea(self):#selfisareferencetoaninstance

returnself.side**2

sq=Square()

print(sq.area())#64(sideisfoundontheclass)

print(Square.area(sq))#64(equivalenttosq.area())

sq.side=10

print(sq.area())#100(sideisfoundontheinstance)

Notehowtheareamethodisusedbysq.Thetwocalls,Square.area(sq)andsq.area(),areequivalent,andteachushowthemechanismworks.Eitheryoupasstheinstancetothemethodcall(Square.area(sq)),whichwithinthemethodwillbecalledself,oryoucanuseamorecomfortablesyntax:sq.area()andPythonwilltranslatethatforyoubehindthecurtains.

Let’slookatabetterexample:oop/class.price.py

classPrice():

deffinal_price(self,vat,discount=0):

"""Returnspriceafterapplyingvatandfixeddiscount."""

return(self.net_price*(100+vat)/100)-discount

p1=Price()

p1.net_price=100

print(Price.final_price(p1,20,10))#110(100*1.2-10)

print(p1.final_price(20,10))#equivalent

Theprecedingcodeshowsyouthatnothingpreventsusfromusingargumentswhendeclaringmethods.Wecanusetheexactsamesyntaxasweusedwiththefunction,butweneedtorememberthatthefirstargumentwillalwaysbetheinstance.

InitializinganinstanceHaveyounoticedhow,beforecallingp1.final_price(...),wehadtoassignnet_pricetop1?Thereisabetterwaytodoit.Inotherlanguages,thiswouldbecalledaconstructor,butinPython,it’snot.Itisactuallyaninitializer,sinceitworksonanalreadycreatedinstance,andthereforeit’scalled__init__.It’samagicmethod,whichisrunrightaftertheobjectiscreated.Pythonobjectsalsohavea__new__method,whichistheactualconstructor.Inpractice,it’snotsocommontohavetooverrideitthough,it’sapracticethatismostlyusedwhencodingmetaclasses,whichisafairlyadvancedtopicthatwewon’texploreinthebook.oop/class.init.py

classRectangle():

def__init__(self,sideA,sideB):

self.sideA=sideA

self.sideB=sideB

defarea(self):

returnself.sideA*self.sideB

r1=Rectangle(10,4)

print(r1.sideA,r1.sideB)#104

print(r1.area())#40

r2=Rectangle(7,3)

print(r2.area())#21

Thingsarefinallystartingtotakeshape.Whenanobjectiscreated,the__init__methodisautomaticallyrunforus.Inthiscase,Icodeditsothatwhenwecreateanobject(bycallingtheclassnamelikeafunction),wepassargumentstothecreationcall,likewewouldonanyregularfunctioncall.Thewaywepassparametersfollowsthesignatureofthe__init__method,andtherefore,inthetwocreationstatements,10and7willbesideAforr1andr2respectively,while4and3willbesideB.Youcanseethatthecalltoarea()fromr1andr2reflectsthattheyhavedifferentinstancearguments.

Settingupobjectsinthiswayismuchnicerandconvenient.

OOPisaboutcodereuseBynowitshouldbeprettyclear:OOPisallaboutcodereuse.Wedefineaclass,wecreateinstances,andthoseinstancesusemethodsthataredefinedonlyintheclass.Theywillbehavedifferentlyaccordingtohowtheinstanceshavebeensetupbytheinitializer.

InheritanceandcompositionButthisisjusthalfofthestory,OOPismuchmorepowerful.Wehavetwomaindesignconstructstoexploit:inheritanceandcomposition.

InheritancemeansthattwoobjectsarerelatedbymeansofanIs-Atypeofrelationship.Ontheotherhand,compositionmeansthattwoobjectsarerelatedbymeansofaHas-Atypeofrelationship.It’sallveryeasytoexplainwithanexample:oop/class.inheritance.py

classEngine():

defstart(self):

pass

defstop(self):

pass

classElectricEngine(Engine):#Is-AEngine

pass

classV8Engine(Engine):#Is-AEngine

pass

classCar():

engine_cls=Engine

def__init__(self):

self.engine=self.engine_cls()#Has-AEngine

defstart(self):

print(

'Startingengine{0}forcar{1}...Wroom,wroom!'

.format(

self.engine.__class__.__name__,

self.__class__.__name__)

)

self.engine.start()

defstop(self):

self.engine.stop()

classRaceCar(Car):#Is-ACar

engine_cls=V8Engine

classCityCar(Car):#Is-ACar

engine_cls=ElectricEngine

classF1Car(RaceCar):#Is-ARaceCarandalsoIs-ACar

engine_cls=V8Engine

car=Car()

racecar=RaceCar()

citycar=CityCar()

f1car=F1Car()

cars=[car,racecar,citycar,f1car]

forcarincars:

car.start()

"""Prints:

StartingengineEngineforcarCar...Wroom,wroom!

StartingengineV8EngineforcarRaceCar...Wroom,wroom!

StartingengineElectricEngineforcarCityCar...Wroom,wroom!

StartingengineV8EngineforcarF1Car...Wroom,wroom!

"""

TheprecedingexampleshowsyouboththeIs-AandHas-Atypesofrelationshipsbetweenobjects.Firstofall,let’sconsiderEngine.It’sasimpleclassthathastwomethods,startandstop.WethendefineElectricEngineandV8Engine,whichbothinheritfromEngine.Youcanseethatbythefactthatwhenwedefinethem,weputEnginewithinthebracesaftertheclassname.

ThismeansthatbothElectricEngineandV8EngineinheritattributesandmethodsfromtheEngineclass,whichissaidtobetheirbaseclass.

Thesamehappenswithcars.CarisabaseclassforbothRaceCarandCityCar.RaceCarisalsothebaseclassforF1Car.AnotherwayofsayingthisisthatF1CarinheritsfromRaceCar,whichinheritsfromCar.Therefore,F1CarIs-ARaceCarandRaceCarIs-ACar.Becauseofthetransitiveproperty,wecansaythatF1CarIs-ACaraswell.CityCartoo,Is-ACar.

WhenwedefineclassA(B):pass,wesayAisthechildofB,andBistheparentofA.parentandbasearesynonyms,aswellaschildandderived.Also,wesaythataclassinheritsfromanotherclass,orthatitextendsit.

Thisistheinheritancemechanism.

Ontheotherhand,let’sgobacktothecode.Eachclasshasaclassattribute,engine_cls,whichisareferencetotheengineclasswewanttoassigntoeachtypeofcar.CarhasagenericEngine,whilethetworacecarshaveapowerfulV8engine,andthecitycarhasanelectricone.

Whenacariscreatedintheinitializermethod__init__,wecreateaninstanceofwhateverengineclassisassignedtothecar,andsetitasengineinstanceattribute.

Itmakessensetohaveengine_clssharedamongstallclassinstancesbecauseit’squitelikelythatthesameinstancesofacarwillhavethesamekindofengine.Ontheotherhand,itwouldn’tbegoodtohaveonesingleengine(aninstanceofanyEngineclass)asaclassattribute,becausewewouldbesharingoneengineamongstallinstances,whichisincorrect.

ThetypeofrelationshipbetweenacaranditsengineisaHas-Atype.AcarHas-Aengine.Thisiscalledcomposition,andreflectsthefactthatobjectscanbemadeofmanyotherobjects.AcarHas-Aengine,gears,wheels,aframe,doors,seats,andsoon.

WhendesigningOOPcode,itisofvitalimportancetodescribeobjectsinthiswaysothatwecanuseinheritanceandcompositioncorrectlytostructureourcodeinthebestway.

Beforeweleavethisparagraph,let’scheckifItoldyouthetruthwithanotherexample:oop/class.issubclass.isinstance.py

car=Car()

racecar=RaceCar()

f1car=F1Car()

cars=[(car,'car'),(racecar,'racecar'),(f1car,'f1car')]

car_classes=[Car,RaceCar,F1Car]

forcar,car_nameincars:

forclass_incar_classes:

belongs=isinstance(car,class_)

msg='isa'ifbelongselse'isnota'

print(car_name,msg,class_.__name__)

"""Prints:

carisaCar

carisnotaRaceCar

carisnotaF1Car

racecarisaCar

racecarisaRaceCar

racecarisnotaF1Car

f1carisaCar

f1carisaRaceCar

f1carisaF1Car

"""

Asyoucansee,carisjustaninstanceofCar,whileracecarisaninstanceofRaceCar(andofCarbyextension)andf1carisaninstanceofF1Car(andofbothRaceCarandCar,byextension).AbananaisaninstanceofBanana.But,also,itisaFruit.Also,itisFood,right?Thisisthesameconcept.

Tocheckifanobjectisaninstanceofaclass,usetheisinstancemethod.Itisrecommendedoversheertypecomparison(type(object)==Class).

Let’salsocheckinheritance,samesetup,withdifferentforloops:oop/class.issubclass.isinstance.py

forclass1incar_classes:

forclass2incar_classes:

is_subclass=issubclass(class1,class2)

msg='{0}asubclassof'.format(

'is'ifis_subclasselse'isnot')

print(class1.__name__,msg,class2.__name__)

"""Prints:

CarisasubclassofCar

CarisnotasubclassofRaceCar

CarisnotasubclassofF1Car

RaceCarisasubclassofCar

RaceCarisasubclassofRaceCar

RaceCarisnotasubclassofF1Car

F1CarisasubclassofCar

F1CarisasubclassofRaceCar

F1CarisasubclassofF1Car

"""

Interestingly,welearnthataclassisasubclassofitself.ChecktheoutputoftheprecedingexampletoseethatitmatchestheexplanationIprovided.

NoteOnethingtonoticeaboutconventionsisthatclassnamesarealwayswrittenusingCapWords,whichmeansThisWayIsCorrect,asopposedtofunctionsandmethods,whicharewrittenthis_way_is_correct.Also,wheninthecodeyouwanttouseanamewhichisaPython-reservedkeywordorbuilt-infunctionorclass,theconventionistoaddatrailingunderscoretothename.Inthefirstforloopexample,I’mloopingthroughtheclassnamesusingforclass_in…,becauseclassisareservedword.ButyoualreadyknewallthisbecauseyouhavethoroughlystudiedPEP8,right?

TohelpyoupicturethedifferencebetweenIs-AandHas-A,takealookatthefollowingdiagram:

AccessingabaseclassWe’vealreadyseenclassdeclarationslikeclassClassA:passandclassClassB(BaseClassName):pass.Whenwedon’tspecifyabaseclassexplicitly,Pythonwillsetthespecialobjectclassasthebaseclassfortheonewe’redefining.Ultimately,allclassesderivefromobject.Notethat,ifyoudon’tspecifyabaseclass,bracesareoptional.

Therefore,writingclassA:passorclassA():passorclassA(object):passisexactlythesamething.objectisaspecialclassinthatithasthemethodsthatarecommontoallPythonclasses,anditdoesn’tallowyoutosetanyattributesonit.

Let’sseehowwecanaccessabaseclassfromwithinaclass.oop/super.duplication.py

classBook:

def__init__(self,title,publisher,pages):

self.title=title

self.publisher=publisher

self.pages=pages

classEbook(Book):

def__init__(self,title,publisher,pages,format_):

self.title=title

self.publisher=publisher

self.pages=pages

self.format_=format_

Takealookattheprecedingcode.IhighlightedthepartofEbookinitializationthatisduplicatedfromitsbaseclassBook.Thisisquitebadpracticebecausewenowhavetwosetsofinstructionsthataredoingthesamething.Moreover,anychangeinthesignatureofBook.__init__willnotreflectinEbook.WeknowthatEbookIs-ABook,andthereforewewouldprobablywantchangestobereflectedinthechildrenclasses.

Let’sseeonewaytofixthisissue:oop/super.explicit.py

classBook:

def__init__(self,title,publisher,pages):

self.title=title

self.publisher=publisher

self.pages=pages

classEbook(Book):

def__init__(self,title,publisher,pages,format_):

Book.__init__(self,title,publisher,pages)

self.format_=format_

ebook=Ebook('LearningPython','PacktPublishing',360,'PDF')

print(ebook.title)#LearningPython

print(ebook.publisher)#PacktPublishing

print(ebook.pages)#360

print(ebook.format_)#PDF

Now,that’sbetter.Wehaveremovedthatnastyduplication.Basically,wetellPythontocallthe__init__methodoftheBookclass,andwefeedselftothecall,makingsurethatwebindthatcalltothepresentinstance.

Ifwemodifythelogicwithinthe__init__methodofBook,wedon’tneedtotouchEbook,itwillautoadapttothechange.

Thisapproachisgood,butwecanstilldoabitbetter.SaythatwechangeBook‘snametoLiber,becausewe’vefalleninlovewithLatin.Wehavetochangethe__init__methodofEbooktoreflectthechange.Thiscanbeavoidedbyusingsuper.oop/super.implicit.py

classBook:

def__init__(self,title,publisher,pages):

self.title=title

self.publisher=publisher

self.pages=pages

classEbook(Book):

def__init__(self,title,publisher,pages,format_):

super().__init__(title,publisher,pages)

#Anotherwaytodothesamethingis:

#super(Ebook,self).__init__(title,publisher,pages)

self.format_=format_

ebook=Ebook('LearningPython','PacktPublishing',360,'PDF')

print(ebook.title)#LearningPython

print(ebook.publisher)#PacktPublishing

print(ebook.pages)#360

print(ebook.format_)#PDF

superisafunctionthatreturnsaproxyobjectthatdelegatesmethodcallstoaparentorsiblingclass.Inthiscase,itwilldelegatethatcallto__init__totheBookclass,andthebeautyofthismethodisthatnowwe’reevenfreetochangeBooktoLiberwithouthavingtotouchthelogicinthe__init__methodofEbook.

Nowthatweknowhowtoaccessabaseclassfromachild,let’sexplorePython’smultipleinheritance.

MultipleinheritanceApartfromcomposingaclassusingmorethanonebaseclass,whatisofinteresthereishowanattributesearchisperformed.Takealookatthefollowingdiagram:

Asyoucansee,ShapeandPlotteractasbaseclassesforalltheothers.Polygoninheritsdirectlyfromthem,RegularPolygoninheritsfromPolygon,andbothRegularHexagonandSquareinheritfromRegulaPolygon.NotealsothatShapeandPlotterimplicitlyinheritfromobject,thereforewehavewhatiscalledadiamondor,insimplerterms,morethanonepathtoreachabaseclass.We’llseewhythismattersinafewmoments.Let’stranslateitintosomesimplecode:oop/multiple.inheritance.py

classShape:

geometric_type='GenericShape'

defarea(self):#Thisactsasplaceholderfortheinterface

raiseNotImplementedError

defget_geometric_type(self):

returnself.geometric_type

classPlotter:

defplot(self,ratio,topleft):

#Imaginesomeniceplottinglogichere…

print('Plottingat{},ratio{}.'.format(

topleft,ratio))

classPolygon(Shape,Plotter):#baseclassforpolygons

geometric_type='Polygon'

classRegularPolygon(Polygon):#Is-APolygon

geometric_type='RegularPolygon'

def__init__(self,side):

self.side=side

classRegularHexagon(RegularPolygon):#Is-ARegularPolygon

geometric_type='RegularHexagon'

defarea(self):

return1.5*(3**.5*self.side**2)

classSquare(RegularPolygon):#Is-ARegularPolygon

geometric_type='Square'

defarea(self):

returnself.side*self.side

hexagon=RegularHexagon(10)

print(hexagon.area())#259.8076211353316

print(hexagon.get_geometric_type())#RegularHexagon

hexagon.plot(0.8,(75,77))#Plottingat(75,77),ratio0.8.

square=Square(12)

print(square.area())#144

print(square.get_geometric_type())#Square

square.plot(0.93,(74,75))#Plottingat(74,75),ratio0.93.

Takealookattheprecedingcode:theclassShapehasoneattribute,geometric_type,andtwomethods:areaandget_geometric_type.It’squitecommontousebaseclasses(likeShape,inourexample)todefineaninterface:methodsforwhichchildrenmustprovideanimplementation.Therearedifferentandbetterwaystodothis,butIwanttokeepthisexampleassimpleaspossible.

WealsohavethePlotterclass,whichaddstheplotmethod,therebyprovidingplottingcapabilitiesforanyclassthatinheritsfromit.Ofcourse,theplotimplementationisjustadummyprintinthisexample.ThefirstinterestingclassisPolygon,whichinheritsfrombothShapeandPlotter.

Therearemanytypesofpolygons,oneofwhichistheregularone,whichisbothequiangular(allanglesareequal)andequilateral(allsidesareequal),sowecreatetheRegularPolygonclassthatinheritsfromPolygon.Becauseforaregularpolygon,allsidesareequal,wecanimplementasimple__init__methodonRegularPolygon,whichtakesthelengthoftheside.Finally,wecreatetheRegularHexagonandSquareclasses,whichbothinheritfromRegularPolygon.

Thisstructureisquitelong,buthopefullygivesyouanideaofhowtospecializetheclassificationofyourobjectswhenyoudesignthecode.

Now,pleasetakealookatthelasteightlines.NotethatwhenIcalltheareamethodonhexagonandsquare,Igetthecorrectareaforboth.Thisisbecausetheybothprovidethecorrectimplementationlogicforit.Also,Icancallget_geometric_typeonbothofthem,eventhoughitisnotdefinedontheirclasses,andPythonhastogoallthewayuptoShapetofindanimplementationforit.Notethat,eventhoughtheimplementationisprovidedintheShapeclass,theself.geometric_typeusedforthereturnvalueiscorrectlytakenfromthecallerinstance.

Theplotmethodcallsarealsointeresting,andshowyouhowyoucanenrichyourobjectswithcapabilitiestheywouldn’totherwisehave.ThistechniqueisverypopularinwebframeworkssuchasDjango(whichwe’llexploreintwolaterchapters),whichprovidesspecialclassescalledmixins,whosecapabilitiesyoucanjustuseoutofthebox.Allyouhavetodoistodefinethedesiredmixinasonethebaseclassesforyourown,andthat’sit.

Multipleinheritanceispowerful,butcanalsogetreallymessy,soweneedtomakesureweunderstandwhathappenswhenweuseit.

MethodresolutionorderBynow,weknowthatwhenyouaskforsomeobject.attribute,andattributeisnotfoundonthatobject,Pythonstartssearchingintheclasssomeobjectwascreatedfrom.Ifit’snotthereeither,Pythonsearchesuptheinheritancechainuntileitherattributeisfoundortheobjectclassisreached.Thisisquitesimpletounderstandiftheinheritancechainisonlycomprisedofsingleinheritancesteps,whichmeansthatclasseshaveonlyoneparent.However,whenmultipleinheritanceisinvolved,therearecaseswhenit’snotstraightforwardtopredictwhatwillbethenextclassthatwillbesearchedforifanattributeisnotfound.

Pythonprovidesawaytoalwaysknowwhatistheorderinwhichclassesaresearchedonattributelookup:themethodresolutionorder.

NoteThemethodresolutionorder(MRO)istheorderinwhichbaseclassesaresearchedforamemberduringlookup.Fromversion2.3PythonusesanalgorithmcalledC3,whichguaranteesmonotonicity.

InPython2.2,new-styleclasseswereintroduced.Thewayyouwriteanew-styleclassinPython2.*istodefineitwithanexplicitobjectbaseclass.ClassicclasseswerenotexplicitlyinheritingfromobjectandhavebeenremovedinPython3.

Oneofthedifferencesbetweenclassicandnewstyle-classesinPython2.*isthatnew-styleclassesaresearchedwiththenewMRO.

Withregardstothepreviousexample,let’sseewhatistheMROfortheSquareclass:oop/multiple.inheritance.py

print(square.__class__.__mro__)

#prints:

#(<class'__main__.Square'>,<class'__main__.RegularPolygon'>,

#<class'__main__.Polygon'>,<class'__main__.Shape'>,

#<class'__main__.Plotter'>,<class'object'>)

TogettotheMROofaclass,wecangofromtheinstancetoits__class__attributeandfromthattoits__mro__attribute.Alternatively,wecouldhavecalledSquare.__mro__,orSquare.mro()directly,butifyouhavetodoitdynamically,it’smorelikelyyouwillhaveanobjectinyourhandsratherthanaclass.

NotethattheonlypointofdoubtisthebisectionafterPolygon,wheretheinheritancechainbreaksintotwoways,oneleadstoShapeandtheothertoPlotter.WeknowbyscanningtheMROfortheSquareclassthatShapeissearchedbeforePlotter.

Whyisthisimportant?Well,imaginethefollowingcode:oop/mro.simple.py

classA:

label='a'

classB(A):

label='b'

classC(A):

label='c'

classD(B,C):

pass

d=D()

print(d.label)#Hypotheticallythiscouldbeeither'b'or'c'

BothBandCinheritfromA,andDinheritsfrombothBandC.Thismeansthatthelookup

forthelabelattributecanreachthetop(A)throughbothBorC.Accordingtowhichisreachedfirst,wegetadifferentresult.

So,intheprecedingexampleweget'b',whichiswhatwewereexpecting,sinceBistheleftmostoneamongstbaseclassesofD.ButwhathappensifIremovethelabelattributefromB?Thiswouldbetheconfusingsituation:WillthealgorithmgoallthewayuptoAorwillitgettoCfirst?Let’sfindout!oop/mro.py

classA:

label='a'

classB(A):

pass#was:label='b'

classC(A):

label='c'

classD(B,C):

pass

d=D()

print(d.label)#'c'

print(d.__class__.mro())#noticeanotherwaytogettheMRO

#prints:

#[<class'__main__.D'>,<class'__main__.B'>,

#<class'__main__.C'>,<class'__main__.A'>,<class'object'>]

So,welearnthattheMROisD-B-C-A-(object),whichmeanswhenweaskford.label,weget'c',whichiscorrect.

Indaytodayprogramming,itisnotquitecommontohavetodealwiththeMRO,butthefirsttimeyoufightagainstsomemixinfromaframework,Ipromiseyou’llbegladIspentaparagraphexplainingit.

StaticandclassmethodsUntilnow,wehavecodedclasseswithattributesintheformofdataandinstancemethods,buttherearetwoothertypesofmethodsthatwecanplaceinsideaclass:staticmethodsandclassmethods.

StaticmethodsAsyoumayrecall,whenyoucreateaclassobject,Pythonassignsanametoit.Thatnameactsasanamespace,andsometimesitmakessensetogroupfunctionalitiesunderit.Staticmethodsareperfectforthisusecasesinceunlikeinstancemethods,theyarenotpassedanyspecialargument.Let’slookatanexampleofanimaginaryStringclass.oop/static.methods.py

classString:

@staticmethod

defis_palindrome(s,case_insensitive=True):

#weallowonlylettersandnumbers

s=''.join(cforcinsifc.isalnum())#Studythis!

#Forcaseinsensitivecomparison,welower-cases

ifcase_insensitive:

s=s.lower()

forcinrange(len(s)//2):

ifs[c]!=s[-c-1]:

returnFalse

returnTrue

@staticmethod

defget_unique_words(sentence):

returnset(sentence.split())

print(String.is_palindrome(

'Radar',case_insensitive=False))#False:CaseSensitive

print(String.is_palindrome('Anutforajaroftuna'))#True

print(String.is_palindrome('NeverOdd,OrEven!'))#True

print(String.is_palindrome(

'InGirumImusNocteEtConsumimurIgni')#Latin!Show-off!

)#True

print(String.get_unique_words(

'Ilovepalindromes.Ireallyreallylovethem!'))

#{'them!','really','palindromes.','I','love'}

Theprecedingcodeisquiteinteresting.Firstofall,welearnthatstaticmethodsarecreatedbysimplyapplyingthestaticmethoddecoratortothem.Youcanseethattheyaren’tpassedanyspecialargumentso,apartfromthedecoration,theyreallyjustlooklikefunctions.

Wehaveaclass,String,whichactsasacontainerforfunctions.Anotherapproachwouldbetohaveaseparatemodulewithfunctionsinside.It’sreallyamatterofpreferencemostofthetime.

Thelogicinsideis_palindromeshouldbestraightforwardforyoutounderstandbynow,but,justincase,let’sgothroughit.Firstweremoveallcharactersfromsthatarenoteitherlettersornumbers.Inordertodothis,weusethejoinmethodofastringobject(anemptystringobject,inthiscase).Bycallingjoinonanemptystring,theresultisthatallelementsintheiterableyoupasstojoinwillbeconcatenatedtogether.Wefeedjoinageneratorexpressionthatsays,takeanycharacterfromsifthecharacteriseitheralphanumericoranumber.Ihopeyouhavebeenabletofindthatoutbyyourself,maybeusingtheinside-outtechniqueIshowedyouinoneoftheprecedingchapters.

Wethenlowercasesifcase_insensitiveisTrue,andthenweproceedtocheckifitisapalindrome.Inordertodothis,wecomparethefirstandlastcharacters,thenthesecondandthesecondtolast,andsoon.Ifatanypointwefindadifference,itmeansthestringisn’tapalindromeandthereforewecanreturnFalse.Ontheotherhand,ifweexittheforloopnormally,itmeansnodifferenceswerefound,andwecanthereforesaythestringisapalindrome.

Noticethatthiscodeworkscorrectlyregardlessofthelengthofthestring,thatis,ifthelengthisoddoreven.len(s)//2reacheshalfofs,andifsisanoddamountofcharacterslong,themiddleonewon’tbechecked(likeinRaDaR,Disnotchecked),butwedon’tcare;itwouldbecomparedwithitselfsoit’salwayspassingthatcheck.

get_unique_wordsismuchsimpler,itjustreturnsasettowhichwefeedalistwiththewordsfromasentence.Thesetclassremovesanyduplicationforus,thereforewedon’tneedtodoanythingelse.

TheStringclassprovidesusanicecontainernamespaceformethodsthataremeanttoworkonstrings.IcouldhavecodedasimilarexamplewithaMathclass,andsomestaticmethodstoworkonnumbers,butIwantedtoshowyousomethingdifferent.

ClassmethodsClassmethodsareslightlydifferentfrominstancemethodsinthattheyalsotakeaspecialfirstargument,butinthiscase,itistheclassobjectitself.Twoverycommonusecasesforcodingclassmethodsaretoprovidefactorycapabilitytoaclassandtoallowbreakingupstaticmethods(whichyouhavetothencallusingtheclassname)withouthavingtohardcodetheclassnameinyourlogic.Let’slookatanexampleofbothofthem.oop/class.methods.factory.py

classPoint:

def__init__(self,x,y):

self.x=x

self.y=y

@classmethod

deffrom_tuple(cls,coords):#clsisPoint

returncls(*coords)

@classmethod

deffrom_point(cls,point):#clsisPoint

returncls(point.x,point.y)

p=Point.from_tuple((3,7))

print(p.x,p.y)#37

q=Point.from_point(p)

print(q.x,q.y)#37

Intheprecedingcode,Ishowedyouhowtouseaclassmethodtocreateafactoryfortheclass.Inthiscase,wewanttocreateaPointinstancebypassingbothcoordinates(regularcreationp=Point(3,7)),butwealsowanttobeabletocreateaninstancebypassingatuple(Point.from_tuple)oranotherinstance(Point.from_point).

Withinthetwoclassmethods,theclsargumentreferstothePointclass.Aswithinstancemethod,whichtakeselfasthefirstargument,classmethodtakeaclsargument.Bothselfandclsarenamedafteraconventionthatyouarenotforcedtofollowbutarestronglyencouragedtorespect.ThisissomethingthatnoPythoncoderwouldchangebecauseitissostrongaconventionthatparsers,linters,andanytoolthatautomaticallydoessomethingwithyourcodewouldexpect,soit’smuchbettertosticktoit.

Let’slookatanexampleoftheotherusecase:splittingastaticmethod.oop/class.methods.split.py

classString:

@classmethod

defis_palindrome(cls,s,case_insensitive=True):

s=cls._strip_string(s)

#Forcaseinsensitivecomparison,welower-cases

ifcase_insensitive:

s=s.lower()

returncls._is_palindrome(s)

@staticmethod

def_strip_string(s):

return''.join(cforcinsifc.isalnum())

@staticmethod

def_is_palindrome(s):

forcinrange(len(s)//2):

ifs[c]!=s[-c-1]:

returnFalse

returnTrue

@staticmethod

defget_unique_words(sentence):

returnset(sentence.split())

print(String.is_palindrome('Anutforajaroftuna'))#True

print(String.is_palindrome('Anutforajarofbeans'))#False

Comparethiscodewiththepreviousversion.Firstofallnotethateventhoughis_palindromeisnowaclassmethod,wecallitinthesamewaywewerecallingitwhenitwasastaticone.Thereasonwhywechangedittoaclassmethodisthatafterfactoringoutacoupleofpiecesoflogic(_strip_stringand_is_palindrome),weneedtogeta

referencetothemandifwehavenoclsinourmethod,theonlyoptionwouldbetocallthemlikethis:String._strip_string(...)andString._is_palindrome(...),whichisnotgoodpractice,becausewewouldhardcodetheclassnameintheis_palindromemethod,therebyputtingourselvesintheconditionofhavingtomodifyitwheneverwewouldchangetheclassname.Usingclswillactastheclassname,whichmeansourcodewon’tneedanyamendments.

Notealsothat,bynamingthefactored-outmethodswithaleadingunderscore,Iamhintingthatthosemethodsarenotsupposedtobecalledfromoutsidetheclass,butthiswillbethesubjectofthenextparagraph.

PrivatemethodsandnamemanglingIfyouhaveanybackgroundwithlanguageslikeJava,C#,C++,orsimilar,thenyouknowtheyallowtheprogrammertoassignaprivacystatustoattributes(bothdataandmethods).Eachlanguagehasitsownslightlydifferentflavorforthis,butthegististhatpublicattributesareaccessiblefromanypointinthecode,whileprivateonesareaccessibleonlywithinthescopetheyaredefinedin.

InPython,thereisnosuchthing.Everythingispublic;therefore,werelyonconventionsandonamechanismcallednamemangling.

Theconventionisasfollows:ifanattribute’snamehasnoleadingunderscoresitisconsideredpublic.Thismeansyoucanaccessitandmodifyitfreely.Whenthenamehasoneleadingunderscore,theattributeisconsideredprivate,whichmeansit’sprobablymeanttobeusedinternallyandyoushouldnotuseitormodifyitfromtheoutside.Averycommonusecaseforprivateattributesarehelpermethodsthataresupposedtobeusedbypublicones(possiblyincallchainsinconjunctionwithothermethods),andinternaldata,likescalingfactors,oranyotherdatathatideallywewouldputinaconstant(avariablethatcannotchange,but,surprise,surprise,Pythondoesn’thavethoseeither).

Thischaracteristicusuallyscarespeoplefromotherbackgroundsoff;theyfeelthreatenedbythelackofprivacy.Tobehonest,inmywholeprofessionalexperiencewithPython,I’veneverheardanyonescreamingOhmyGod,wehaveaterriblebugbecausePythonlacksprivateattributes!Notonce,Iswear.

Thatsaid,thecallforprivacyactuallymakessensebecausewithoutit,youriskintroducingbugsintoyourcodeforreal.Let’slookatasimpleexample:oop/private.attrs.py

classA:

def__init__(self,factor):

self._factor=factor

defop1(self):

print('Op1withfactor{}...'.format(self._factor))

classB(A):

defop2(self,factor):

self._factor=factor

print('Op2withfactor{}...'.format(self._factor))

obj=B(100)

obj.op1()#Op1withfactor100…

obj.op2(42)#Op2withfactor42…

obj.op1()#Op1withfactor42…<-ThisisBAD

Intheprecedingcode,wehaveanattributecalled_factor,andlet’spretendit’sveryimportantthatitisn’tmodifiedatruntimeaftertheinstanceiscreated,becauseop1dependsonittofunctioncorrectly.We’venameditwithaleadingunderscore,buttheissuehereisthatwhenwecallobj.op2(42),wemodifyit,andthisreflectsinsubsequent

callstoop1.

Let’sfixthisundesiredbehaviorbyaddinganotherleadingunderscore:oop/private.attrs.fixed.py

classA:

def__init__(self,factor):

self.__factor=factor

defop1(self):

print('Op1withfactor{}...'.format(self.__factor))

classB(A):

defop2(self,factor):

self.__factor=factor

print('Op2withfactor{}...'.format(self.__factor))

obj=B(100)

obj.op1()#Op1withfactor100…

obj.op2(42)#Op2withfactor42…

obj.op1()#Op1withfactor100…<-Wohoo!Nowit'sGOOD!

Wow,lookatthat!Nowit’sworkingasdesired.Pythoniskindofmagicandinthiscase,whatishappeningisthatthenamemanglingmechanismhaskickedin.

Namemanglingmeansthatanyattributenamethathasatleasttwoleadingunderscoresandatmostonetrailingunderscore,like__my_attr,isreplacedwithanamethatincludesanunderscoreandtheclassnamebeforetheactualname,like_ClassName__my_attr.

Thismeansthatwhenyouinheritfromaclass,themanglingmechanismgivesyourprivateattributetwodifferentnamesinthebaseandchildclassessothatnamecollisionisavoided.Everyclassandinstanceobjectstoresreferencestotheirattributesinaspecialattributecalled__dict__,solet’sinspectobj.__dict__toseenamemanglinginaction:oop/private.attrs.py

print(obj.__dict__.keys())

#dict_keys(['_factor'])

Thisisthe_factorattributethatwefindintheproblematicversionofthisexample.Butlookattheonethatisusing__factor:oop/private.attrs.fixed.py

print(obj.__dict__.keys())

#dict_keys(['_A__factor','_B__factor'])

See?objhastwoattributesnow,_A__factor(mangledwithintheAclass),and_B__factor(mangledwithintheBclass).Thisisthemechanismthatmakespossiblethatwhenyoudoobj.__factor=42,__factorinAisn’tchanged,becauseyou’reactuallytouching_B__factor,whichleaves_A__factorsafeandsound.

Ifyou’redesigningalibrarywithclassesthataremeanttobeusedandextendedbyotherdevelopers,youwillneedtokeepthisinmindinordertoavoidunintentionaloverriding

ofyourattributes.Bugslikethesecanbeprettysubtleandhardtospot.

ThepropertydecoratorAnotherthingthatwouldbeacrimenottomentionisthepropertydecorator.ImaginethatyouhaveanageattributeinaPersonclassandatsomepointyouwanttomakesurethatwhenyouchangeitsvalue,you’realsocheckingthatageiswithinaproperrange,like[18,99].Youcanwriteaccessormethods,likeget_age()andset_age()(alsocalledgettersandsetters)andputthelogicthere.get_age()willmostlikelyjustreturnage,whileset_age()willalsodotherangecheck.Theproblemisthatyoumayalreadyhavealotofcodeaccessingtheageattributedirectly,whichmeansyou’renowuptosomegood(andpotentiallydangerousandtedious)refactoring.LanguageslikeJavaovercomethisproblembyusingtheaccessorpatternbasicallybydefault.ManyJavaIntegratedDevelopmentEnvironments(IDEs)autocompleteanattributedeclarationbywritinggetterandsetteraccessormethodsstubsforyouonthefly.

Pythonissmarter,anddoesthiswiththepropertydecorator.Whenyoudecorateamethodwithproperty,youcanusethenameofthemethodasifitwasadataattribute.Becauseofthis,it’salwaysbesttorefrainfromputtinglogicthatwouldtakeawhiletocompleteinsuchmethodsbecause,byaccessingthemasattributes,wearenotexpectingtowait.

Let’slookatanexample:oop/property.py

classPerson:

def__init__(self,age):

self.age=age#anyonecanmodifythisfreely

classPersonWithAccessors:

def__init__(self,age):

self._age=age

defget_age(self):

returnself._age

defset_age(self):

if18<=age<=99:

self._age=age

else:

raiseValueError('Agemustbewithin[18,99]')

classPersonPythonic:

def__init__(self,age):

self._age=age

@property

defage(self):

returnself._age

@age.setter

defage(self,age):

if18<=age<=99:

self._age=age

else:

raiseValueError('Agemustbewithin[18,99]')

person=PersonPythonic(39)

print(person.age)#39-Noticeweaccessasdataattribute

person.age=42#Noticeweaccessasdataattribute

print(person.age)#42

person.age=100#ValueError:Agemustbewithin[18,99]

ThePersonclassmaybethefirstversionwewrite.Thenwerealizeweneedtoputtherangelogicinplaceso,withanotherlanguage,wewouldhavetorewritePersonasthePersonWithAccessorsclass,andrefactorallthecodethatwasusingPerson.age.InPython,werewritePersonasPersonPythonic(younormallywouldn’tchangethename,ofcourse)sothattheageisstoredinaprivate_agevariable,andwedefinepropertygettersandsettersusingthatdecoration,whichallowustokeepusingthepersoninstancesaswewerebefore.Agetterisamethodthatiscalledwhenweaccessanattributeforreading.Ontheotherhand,asetterisamethodthatiscalledwhenweaccessanattributetowriteit.Inotherlanguages,likeJavaforexample,it’scustomarytodefinethemasget_age()andset_age(intvalue),butIfindthePythonsyntaxmuchneater.Itallowsyoutostartwritingsimplecodeandrefactorlateron,onlywhenyouneedit,thereisnoneedtopolluteyourcodewithaccessorsonlybecausetheymaybehelpfulinthefuture.

Thepropertydecoratoralsoallowsforread-onlydata(nosetter)andforspecialactionswhentheattributeisdeleted.Pleaserefertotheofficialdocumentationtodigdeeper.

OperatoroverloadingIfindPython’sapproachtooperatoroverloadingtobebrilliant.Tooverloadanoperatormeanstogiveitameaningaccordingtothecontextinwhichitisused.Forexample,the+operatormeansadditionwhenwedealwithnumbers,butconcatenationwhenwedealwithsequences.

InPython,whenyouuseoperators,you’remostlikelycallingthespecialmethodsofsomeobjectsbehindthescenes.Forexample,thecalla[k]roughlytranslatestotype(a).__getitem__(a,k).

Asanexample,let’screateaclassthatstoresastringandevaluatestoTrueif'42'ispartofthatstring,andFalseotherwise.Also,let’sgivetheclassalengthpropertywhichcorrespondstothatofthestoredstring.oop/operator.overloading.py

classWeird:

def__init__(self,s):

self._s=s

def__len__(self):

returnlen(self._s)

def__bool__(self):

return'42'inself._s

weird=Weird('Hello!Iam9yearsold!')

print(len(weird))#24

print(bool(weird))#False

weird2=Weird('Hello!Iam42yearsold!')

print(len(weird2))#25

print(bool(weird2))#True

Thatwasfun,wasn’tit?Forthecompletelistofmagicmethodsthatyoucanoverrideinordertoprovideyourcustomimplementationofoperatorsforyourclasses,pleaserefertothePythondatamodelintheofficialdocumentation.

Polymorphism–abriefoverviewThewordpolymorphismcomesfromtheGreekpolys(many,much)andmorphē(form,shape),anditsmeaningistheprovisionofasingleinterfaceforentitiesofdifferenttypes.

Inourcarexample,wecallengine.start(),regardlessofwhatkindofengineitis.Aslongasitexposesthestartmethod,wecancallit.That’spolymorphisminaction.

Inotherlanguages,likeJava,inordertogiveafunctiontheabilitytoacceptdifferenttypesandcallamethodonthem,thosetypesneedtobecodedinsuchawaythattheyshareaninterface.Inthisway,thecompilerknowsthatthemethodwillbeavailableregardlessofthetypeoftheobjectthefunctionisfed(aslongasitextendstheproperinterface,ofcourse).

InPython,thingsaredifferent.Polymorphismisimplicit,nothingpreventsyoutocallamethodonanobject,therefore,technically,thereisnoneedtoimplementinterfacesorotherpatterns.

Thereisaspecialkindofpolymorphismcalledadhocpolymorphism,whichiswhatwesawinthelastparagraph:operatoroverloading.Theabilityofanoperatortochangeshape,accordingtothetypeofdataitisfed.

Icannotspendtoomuchtimeonpolymorphism,butIencourageyoutocheckitoutbyyourself,itwillexpandyourunderstandingofOOP.Goodluck!

WritingacustomiteratorNowwehaveallthetoolstoappreciatehowwecanwriteourowncustomiterator.Let’sfirstdefinewhatisaniterableandaniterator:

Iterable:Anobjectissaidtobeiterableifit’scapableofreturningitsmembersoneatatime.Lists,tuples,strings,dicts,arealliterables.Customobjectsthatdefineeitherof__iter__or__getitem__methodsarealsoiterables.Iterator:Anobjectissaidtobeaniteratorifitrepresentsastreamofdata.Acustomiteratorisrequiredtoprovideanimplementationfor__iter__thatreturnstheobjectitself,andanimplementationfor__next__,whichreturnsthenextitemofthedatastreamuntilthestreamisexhausted,atwhichpointallsuccessivecallsto__next__simplyraisetheStopIterationexception.Built-infunctionssuchasiterandnextaremappedtocall__iter__and__next__onanobject,behindthescenes.

Let’swriteaniteratorthatreturnsalltheoddcharactersfromastringfirst,andthentheevenones.iterators/iterator.py

classOddEven:

def__init__(self,data):

self._data=data

self.indexes=(list(range(0,len(data),2))+

list(range(1,len(data),2)))

def__iter__(self):

returnself

def__next__(self):

ifself.indexes:

returnself._data[self.indexes.pop(0)]

raiseStopIteration

oddeven=OddEven('ThIsIsCoOl!')

print(''.join(cforcinoddeven))#TIICO!hssol

oddeven=OddEven('HoLa')#ormanually…

it=iter(oddeven)#thiscallsoddeven.__iter__internally

print(next(it))#H

print(next(it))#L

print(next(it))#o

print(next(it))#a

So,weneededtoprovideanimplementationfor__iter__whichreturnedtheobjectitself,andthenonefor__next__.Let’sgothroughit.Whatneedstohappenisthatwereturn_data[0],_data[2],_data[4],…,_data[1],_data[3],_data[5],…untilwehavereturnedeveryiteminthedata.Inordertodothis,wepreparealist,indexes,like[0,2,4,6,…,1,3,5,…],andwhilethereisatleastanelementinit,wepopthefirstoneandreturntheelementfromthedatathatisatthatposition,therebyachievingourgoal.Whenindexesisempty,weraiseStopIteration,asrequiredbytheiteratorprotocol.

Thereareotherwaystoachievethesameresult,sogoaheadandtrytocodeadifferentoneyourself.Makesuretheendresultworksforalledgecases,emptysequences,sequencesoflength1,2,andsoon.

SummaryInthischapter,wesawdecorators,discoveredthereasonsforhavingthem,andafewexamplesusingoneormoreatthesametime.Wealsosawdecoratorsthattakearguments,whichareusuallyusedasdecoratorfactories.

Wescratchedthesurfaceofobject-orientedprogramminginPython.Wecoveredallthebasicsinawaythatyoushouldnowbeabletounderstandfairlyeasilythecodethatwillcomeinfuturechapters.Wetalkedaboutallkindsofmethodsandattributesthatonecanwriteinaclass,weexploredinheritanceversuscomposition,methodoverriding,properties,operatoroverloading,andpolymorphism.

Attheend,weverybrieflytouchedbaseoniterators,sonowyouhavealltheknowledgetoalsounderstandgeneratorsmoredeeply.

Inthenextchapter,wetakeasteepturn.Itwillstartthesecondhalfofthebook,whichismuchmoreproject-orientedso,fromnowon,itwillbelesstheoryandmorecode,Ihopeyouwillenjoyfollowingtheexamplesandgettingyourhandsdirty,verydirty.

Theysaythatasmoothseanevermadeaskillfulsailor,sokeepexploring,breakthings,readtheerrormessagesaswellasthedocumentation,andlet’sseeifwecangettoseethatwhiterabbit.

Chapter7.Testing,Profiling,andDealingwithExceptions “Codewithouttestsisbrokenbydesign.”

—JacobKaplan-Moss

JacobKaplan-MossisoneofthecoredevelopersoftheDjangowebframework.We’regoingtoexploreitinthenextchapters.Istronglyagreewiththisquoteofhis.Ibelievecodewithouttestsshouldn’tbedeployedtoproduction.

Whyaretestssoimportant?Well,forone,theygiveyoupredictability.Or,atleast,theyhelpyouachievehighpredictability.Unfortunately,thereisalwayssomebugthatsneaksintoourcode.Butwedefinitelywantourcodetobeaspredictableaspossible.Whatwedon’twantistohaveasurprise,ourcodebehavinginanunpredictableway.Wouldyoubehappytoknowthatthesoftwarethatchecksonthesensorsoftheplanethatistakingyouonholidayssometimesgoescrazy?No,probablynot.

Thereforeweneedtotestourcode,weneedtocheckthatitsbehavioriscorrect,thatitworksasexpectedwhenitdealswithedgecases,thatitdoesn’thangwhenthecomponentsit’stalkingtoaredown,thattheperformancesarewellwithintheacceptablerange,andsoon.

Thischapterisallaboutthistopic,makingsurethatyourcodeispreparedtofacethescaryoutsideworld,thatisfastenoughandthatitcandealwithunexpectedorexceptionalconditions.

We’regoingtoexploretesting,includingabriefintroductiontotest-drivendevelopment(TDD),whichisoneofmyfavoriteworkingmethodologies.Then,we’regoingtoexploretheworldofexceptions,andfinallywe’regoingtotalkalittlebitaboutperformancesandprofiling.Deepbreath,andherewego…

TestingyourapplicationTherearemanydifferentkindsoftests,somanyinfactthatcompaniesoftenhaveadedicateddepartment,calledqualityassurance(QA),madeupofindividualsthatspendtheirdaytestingthesoftwarethecompanydevelopersproduce.

Tostartmakinganinitialclassification,wecandividetestsintotwobroadcategories:white-boxandblack-boxtests.

White-boxtestsarethosewhichexercisetheinternalsofthecode,theyinspectitdowntoaveryfinelevelofgranularity.Ontheotherhand,black-boxtestsarethosewhichconsiderthesoftwareundertestingasifbeingwithinabox,theinternalsofwhichareignored.Eventhetechnology,orthelanguageusedinsidetheboxisnotimportantforblack-boxtests.Whattheydoistopluginputtooneendoftheboxandverifytheoutputattheotherend,andthat’sit.

NoteThereisalsoanin-betweencategory,calledgray-boxtesting,thatinvolvestestingasysteminthesamewaywedowiththeblack-boxapproach,buthavingsomeknowledgeaboutthealgorithmsanddatastructuresusedtowritethesoftwareandonlypartialaccesstoitssourcecode.

Therearemanydifferentkindsoftestsinthesecategories,eachofwhichservesadifferentpurpose.Justtogiveyouanidea,here’safew:

Front-endtestsmakesurethattheclientsideofyourapplicationisexposingtheinformationthatitshould,allthelinks,thebuttons,theadvertising,everythingthatneedstobeshowntotheclient.Itmayalsoverifythatitispossibletowalkacertainpaththroughtheuserinterface.Scenariotestsmakeuseofstories(orscenarios)thathelpthetesterworkthroughacomplexproblemortestapartofthesystem.Integrationtestsverifythebehaviorofthevariouscomponentsofyourapplicationwhentheyareworkingtogethersendingmessagesthroughinterfaces.Smoketestsareparticularlyusefulwhenyoudeployanewupdateonyourapplication.Theycheckwhetherthemostessential,vitalpartsofyourapplicationarestillworkingastheyshouldandthattheyarenotonfire.Thistermcomesfromwhenengineerstestedcircuitsbymakingsurenothingwassmoking.Acceptancetests,oruseracceptancetesting(UAT)iswhatadeveloperdoeswithaproductowner(forexample,inaSCRUMenvironment)todetermineiftheworkthatwascommissionedwascarriedoutcorrectly.Functionaltestsverifythefeaturesorfunctionalitiesofyoursoftware.Destructiveteststakedownpartsofyoursystem,simulatingafailure,inordertoestablishhowwelltheremainingpartsofthesystemperform.Thesekindsoftestsareperformedextensivelybycompaniesthatneedtoprovideanextremelyreliableservice,suchasAmazon,forexample.Performancetestsaimtoverifyhowwellthesystemperformsunderaspecificload

ofdataortrafficsothat,forexample,engineerscangetabetterunderstandingofwhicharethebottlenecksinthesystemthatcouldbringitdowntoitskneesinaheavyloadsituation,orthosewhichpreventscalability.Usabilitytests,andthecloselyrelateduserexperience(UX)tests,aimtocheckiftheuserinterfaceissimpleandeasytounderstandanduse.Theyaimtoprovideinputtothedesignerssothattheuserexperienceisimproved.Securityandpenetrationtestsaimtoverifyhowwellthesystemisprotectedagainstattacksandintrusions.Unittestshelpthedevelopertowritethecodeinarobustandconsistentway,providingthefirstlineoffeedbackanddefenseagainstcodingmistakes,refactoringmistakes,andsoon.Regressiontestsprovidethedeveloperwithusefulinformationaboutafeaturebeingcompromisedinthesystemafteranupdate.Someofthecausesforasystembeingsaidtohavearegressionareanoldbugcomingbacktolife,anexistingfeaturebeingcompromised,oranewissuebeingintroduced.

Manybooksandarticleshavebeenwrittenabouttesting,andIhavetopointyoutothoseresourcesifyou’reinterestedinfindingoutmoreaboutallthedifferentkindsoftests.Inthischapter,wewillconcentrateonunittests,sincetheyarethebackboneofsoftwarecraftingandformthevastmajorityofteststhatarewrittenbyadeveloper.

Testingisanart,anartthatyoudon’tlearnfrombooks,I’mafraid.Youcanlearnallthedefinitions(andyoushould),andtryandcollectasmuchknowledgeabouttestingasyoucanbutIpromiseyou,youwillbeabletotestyoursoftwareproperlyonlywhenyouhavedoneitforlongenoughinthefield.

Whenyouarehavingtroublerefactoringabitofcode,becauseeverylittlethingyoutouchmakesatestblowup,youlearnhowtowritelessrigidandlimitingtests,whichstillverifythecorrectnessofyourcodebut,atthesametime,allowyouthefreedomandjoytoplaywithit,toshapeitasyouwant.

Whenyouarebeingcalledtoooftentofixunexpectedbugsinyourcode,youlearnhowtowritetestsmorethoroughly,howtocomeupwithamorecomprehensivelistofedgecases,andstrategiestocopewiththembeforetheyturnintobugs.

Whenyouarespendingtoomuchtimereadingtestsandtryingtorefactortheminordertochangeasmallfeatureinthecode,youlearntowritesimpler,shorter,andbetterfocusedtests.

Icouldgoonwiththiswhenyou…youlearn…,butIguessyougetthepicture.Youneedtogetyourhandsdirtyandbuildexperience.Mysuggestion?Studythetheoryasmuchasyoucan,andthenexperimentusingdifferentapproaches.Also,trytolearnfromexperiencedcoders;it’sveryeffective.

TheanatomyofatestBeforeweconcentrateonunittests,let’sseewhatatestis,andwhatitspurposeis.

Atestisapieceofcodewhosepurposeistoverifysomethinginoursystem.Itmaybethatwe’recallingafunctionpassingtwointegers,thatanobjecthasapropertycalleddonald_duck,orthatwhenyouplaceanorderonsomeAPI,afteraminuteyoucanseeitdissectedintoitsbasicelements,inthedatabase.

Atestistypicallycomprisedofthreesections:

Preparation:Thisiswhereyousetupthescene.Youprepareallthedata,theobjects,theservicesyouneedintheplacesyouneedthemsothattheyarereadytobeused.Execution:Thisiswhereyouexecutethebitoflogicthatyou’recheckingagainst.Youperformanactionusingthedataandtheinterfacesyouhavesetupinthepreparationphase.Verification:Thisiswhereyouverifytheresultsandmakesuretheyareaccordingtoyourexpectations.Youcheckthereturnedvalueofafunction,orthatsomedataisinthedatabase,someisnot,somehaschanged,arequesthasbeenmade,somethinghashappened,amethodhasbeencalled,andsoon.

TestingguidelinesLikesoftware,testscanbegoodorbad,withthewholerangeofshadesinthemiddle.Inordertowritegoodtests,herearesomeguidelines:

Keepthemassimpleaspossible:It’sokaytoviolatesomegoodcodingrules,suchashardcodingvaluesorduplicatingcode.Testsneedfirstandforemosttobeasreadableaspossibleandeasytounderstand.Whentestsarehardtoreadorunderstand,youcanneverbesureiftheyareactuallymakingsureyourcodeisperformingcorrectly.Testsshouldverifyonethingandonethingonly:It’sveryimportantthatyoukeepthemshortandcontained.It’sperfectlyfinetowritemultipleteststoexerciseasingleobjectorfunction.Justmakesurethateachtesthasoneandonlyonepurpose.Testsshouldnotmakeanyunnecessaryassumptionwhenverifyingdata:Thisistrickytounderstandatfirst,butsayyouaretestingthereturnvalueofafunctionanditisanunorderedlistofnumbers(like[2,3,1]).Iftheorderinthatlistisrandom,inthetestyoumaybetemptedtosortitandcompareitwith[1,2,3].Ifyoudo,youwillintroduceanextraassumptionontheorderingoftheresultofyourfunctioncall,andthisisbadpractice.Youshouldalwaysfindawaytoverifythingswithoutintroducinganyassumptionsoranyfeaturethatdoesn’tbelongintheusecaseyou’redescribingwithyourtest.Testsshouldexercisethewhat,ratherthanthehow:Testsshouldfocusoncheckingwhatafunctionissupposedtodo,ratherthanhowitisdoingit.Forexample,focusonthefactthatit’scalculatingthesquarerootofanumber(thewhat),insteadofonthefactthatitiscallingmath.sqrttodoit(thehow).Unlessyou’rewritingperformancetestsoryouhaveaparticularneedtoverifyhowacertainactionisperformed,trytoavoidthistypeoftestingandfocusonthewhat.Testingthehowleadstorestrictivetestsandmakesrefactoringhard.Moreover,thetypeoftestyouhavetowritewhenyouconcentrateonthehowismorelikelytodegradethequalityofyourtestingcodebasewhenyouamendyoursoftwarefrequently(moreonthislater).Testsshouldassumetheleastpossibleinthepreparationphase:Sayyouhave10teststhatarecheckinghowadatastructureismanipulatedbyafunction.Andlet’ssaythisdatastructureisadictwithfivekey/valuepairs.Ifyouputthecompletedictineachtest,themomentyouhavetochangesomethinginthatdict,youalsohavetoamendalltentests.Ontheotherhand,ifyoustripdownthetestdataasmuchasyoucan,youwillfindthat,mostofthetime,it’spossibletohavethemajorityoftestscheckingonlyapartialversionofthedata,andonlyafewrunningwithafullversionofit.Thismeansthatwhenyouneedtochangeyourdata,youwillhavetoamendonlythoseteststhatareactuallyexercisingit.Testshouldrunasfastaspossible:Agoodtestcodebasecouldendupbeingmuchlongerthanthecodebeingtesteditself.Itvariesaccordingtothesituationandthedeveloperbutwhateverthelength,you’llenduphavinghundreds,ifnotthousands,ofteststorun,whichmeansthefastertheyrun,thefasteryoucangetbacktowritingcode.WhenusingTDD,forexample,youruntestsveryoften,sospeedisessential.

Testsshoulduseuptheleastpossibleamountofresources:Thereasonforthisisthateverydeveloperwhochecksoutyourcodeshouldbeabletorunyourtests,nomatterhowpowerfultheirboxis.ItcouldbeaskinnyvirtualmachineoraneglectedJenkinsbox,yourtestsshouldrunwithoutchewinguptoomanyresources.

NoteAJenkinsboxisamachinethatrunsJenkins,softwarethatiscapableof,amongstmanyotherthings,runningyourtestsautomatically.Jenkinsisfrequentlyusedincompanieswheredevelopersusepracticeslikecontinuousintegration,extremeprogramming,andsoon.

UnittestingNowthatyouhaveanideaaboutwhattestingisandwhyweneedit,let’sfinallyintroducethedeveloper’sbestfriend:theunittest.

Beforeweproceedwiththeexamples,allowmetospendsomewordsofcaution:I’lltrytogiveyouthefundamentalsaboutunittesting,butIdon’tfollowanyparticularschoolofthoughtormethodologytotheletter.Overtheyears,Ihavetriedmanydifferenttestingapproaches,eventuallycomingupwithmyownwayofdoingthings,whichisconstantlyevolving.ToputitasBruceLeewouldhave:

“Absorbwhatisuseful,discardwhatisuselessandaddwhatisspecificallyyourown”.

WritingaunittestInordertoexplainhowtowriteaunittest,let’shelpourselveswithasimplesnippet:data.py

defget_clean_data(source):

data=load_data(source)

cleaned_data=clean_data(data)

returncleaned_data

Thefunctionget_clean_dataisresponsibleforgettingdatafromsource,cleaningit,andreturningittothecaller.Howdowetestthisfunction?

Onewayofdoingthisistocallitandthenmakesurethatload_datawascalledoncewithsourceasitsonlyargument.Thenwehavetoverifythatclean_datawascalledonce,withthereturnvalueofload_data.And,finally,wewouldneedtomakesurethatthereturnvalueofclean_dataiswhatisreturnedbytheget_clean_datafunctionaswell.

Inordertodothis,weneedtosetupthesourceandrunthiscode,andthismaybeaproblem.Oneofthegoldenrulesofunittestingisthatanythingthatcrossestheboundariesofyourapplicationneedstobesimulated.Wedon’twanttotalktoarealdatasource,andwedon’twanttoactuallyrunrealfunctionsiftheyarecommunicatingwithanythingthatisnotcontainedinourapplication.Afewexampleswouldbeadatabase,asearchservice,anexternalAPI,afileinthefilesystem,andsoon.

Weneedtheserestrictionstoactasashield,sothatwecanalwaysrunourtestssafelywithoutthefearofdestroyingsomethinginarealdatasource.

Anotherreasonisthatitmaybequitedifficultforasingledevelopertoreproducethewholearchitectureontheirbox.Itmayrequirethesettingupofdatabases,APIs,services,filesandfolders,andsoonandsoforth,andthiscanbedifficult,timeconsuming,orsometimesnotevenpossible.

NoteVerysimplyput,anapplicationprogramminginterface(API)isasetoftoolsfor

buildingsoftwareapplications.AnAPIexpressesasoftwarecomponentintermsofitsoperations,inputsandoutputs,andunderlyingtypes.Forexample,ifyoucreateasoftwarethatneedstointerfacewithadataproviderservice,it’sverylikelythatyouwillhavetogothroughtheirAPIinordertogainaccesstothedata.

Therefore,inourunittests,weneedtosimulateallthosethingsinsomeway.Unittestsneedtoberunbyanydeveloperwithouttheneedforthewholesystemtobesetupontheirbox.

Adifferentapproach,whichIalwaysfavorwhenit’spossibletodoso,istosimulateentitieswithoutusingfakeobjects,butusingspecialpurposetestobjectsinstead.Forexample,ifyourcodetalkstoadatabase,insteadoffakingallthefunctionsandmethodsthattalktothedatabaseandprogrammingthefakeobjectssothattheyreturnwhattherealoneswould,I’dmuchratherprefertospawnatestdatabase,setupthetablesanddataIneed,andthenpatchtheconnectionsettingssothatmytestsarerunningrealcode,againstthetestdatabase,therebydoingnoharmatall.In-memorydatabasesareexcellentoptionsforthesecases.

NoteOneoftheapplicationsthatallowyoutospawnadatabasefortesting,isDjango.Withinthedjango.testpackageyoucanfindseveraltoolsthathelpyouwriteyourtestssothatyouwon’thavetosimulatethedialogwithadatabase.Bywritingteststhisway,youwillalsobeabletocheckontransactions,encodings,andallotherdatabaserelatedaspectsofprogramming.Anotheradvantageofthisapproachconsistsintheabilityofcheckingagainstthingsthatcanchangefromonedatabasetoanother.

Sometimes,though,it’sstillnotpossible,andweneedtousefakes,thereforelet’stalkaboutthem.

MockobjectsandpatchingFirstofall,inPython,thesefakeobjectsarecalledmocks.Uptoversion3.3,themocklibrarywasathird-partylibrarythatbasicallyeveryprojectwouldinstallviapipbut,fromversion3.3,ithasbeenincludedinthestandardlibraryundertheunittestmodule,andrightfullyso,givenitsimportanceandhowwidespreaditis.

Theactofreplacingarealobjectorfunction(oringeneral,anypieceofdatastructure)withamock,iscalledpatching.Themocklibraryprovidesthepatchtool,whichcanactasafunctionorclassdecorator,andevenasacontextmanager(moreonthisinChapter8,TheEdges–GUIsandScripts),thatyoucanusetomockthingsout.Onceyouhavereplacedeverythingyouneednottorun,withsuitablemocks,youcanpasstothesecondphaseofthetestandrunthecodeyouareexercising.Aftertheexecution,youwillbeabletocheckthosemockstoverifythatyourcodehasworkedcorrectly.

AssertionsTheverificationphaseisdonethroughtheuseofassertions.Anassertionisafunction(ormethod)thatyoucanusetoverifyequalitybetweenobjects,aswellasotherconditions.Whenaconditionisnotmet,theassertionwillraiseanexceptionthatwillmakeyourtest

fail.Youcanfindalistofassertionsintheunittestmoduledocumentation,andtheircorrespondingPythonicversioninthenosethird-partylibrary,whichprovidesafewadvantagesoverthesheerunittestmodule,startingfromanimprovedtestdiscoverystrategy(whichisthewayatestrunnerdetectsanddiscoversthetestsinyourapplication).

AclassicunittestexampleMocks,patches,andassertionsarethebasictoolswe’llbeusingtowritetests.So,finally,let’sseeanexample.I’mgoingtowriteafunctionthattakesalistofintegersandfiltersoutallthosewhicharen’tpositive.filter_funcs.py

deffilter_ints(v):

return[numfornuminvifis_positive(num)]

defis_positive(n):

returnn>0

Intheprecedingexample,wedefinethefilter_intsfunction,whichbasicallyusesalistcomprehensiontoretainallthenumbersinvthatarepositive,discardingzerosandnegativeones.Ihope,bynow,anyfurtherexplanationofthecodewouldbeinsulting.

Whatisinteresting,though,istostartthinkingabouthowwecantestit.Well,howaboutwecallfilter_intswithalistofnumbersandwemakesurethatis_positiveiscalledforeachofthem?Ofcourse,wewouldhavetotestis_positiveaswell,butIwillshowyoulateronhowtodothat.Let’swriteasimpletestforfilter_intsnow.

NoteJusttobesurewe’reonthesamepage,Iamputtingthecodeforthischapterinafoldercalledch7,whichlieswithintherootofourproject.Atthesamelevelofch7,Ihavecreatedafoldercalledtests,inwhichIhaveplacedafoldercalledtest_ch7.InthisfolderIhaveonetestfile,calledtest_filter_func.py.

Basically,withinthetestsfolder,IwillrecreatethetreestructureofthecodeI’mtesting,prependingeverythingwithtest_.Thisway,findingtestsisreallyeasy,aswellasiskeepingthemtidy.tests/test_ch7/test_filter_funcs.py

fromunittestimportTestCase#1

fromunittest.mockimportpatch,call#2

fromnose.toolsimportassert_equal#3

fromch7.filter_funcsimportfilter_ints#4

classFilterIntsTestCase(TestCase):#5

@patch('ch7.filter_funcs.is_positive')#6

deftest_filter_ints(self,is_positive_mock):#7

#preparation

v=[3,-4,0,5,8]

#execution

filter_ints(v)#8

#verification

assert_equal(

[call(3),call(-4),call(0),call(5),call(8)],

is_positive_mock.call_args_list

)#9

My,ohmy,solittlecode,andyetsomuchtosay.Firstofall:#1.TheTestCaseclassisthebaseclassthatweusetohaveacontainedentityinwhichtorunourtests.It’snotjustabarecontainer;itprovidesyouwithmethodstowritetestsmoreeasily.

On#2,weimportpatchandcallfromtheunittest.mockmodule.patchisresponsibleforsubstitutinganobjectwithaMockinstance,therebygivingustheabilitytocheckonitaftertheexecutionphasehasbeencompleted.callprovidesuswithanicewayofexpressinga(forexample,function)call.

On#3,youcanseethatIprefertouseassertionsfromnose,ratherthantheonesthatcomewiththeunittestmodule.Togiveyouanexample,assert_equal(...)wouldbecomeself.assertEqual(...)ifIdidn’tusenose.Idon’tenjoytypingself.foranyassertion,ifthereisawaytoavoidit,andIdon’tparticularlyenjoycamelcase,thereforeIalwaysprefertousenosetomakemyassertions.

assert_equalisafunctionthattakestwoparameters(andanoptionalthirdonethatactsasamessage)andverifiesthattheyarethesame.Iftheyareequal,nothinghappens,butiftheydiffer,thenanAssertionErrorexceptionisraised,tellingussomethingiswrong.WhenIwritemytests,Ialwaysputtheexpectedvalueasthefirstargument,andtherealoneasthesecond.ThisconventionsavesmetimewhenI’mreadingtests.

On#4,weimportthefunctionwewanttotest,andthen(#5)weproceedtocreatetheclasswhereourtestswilllive.Eachmethodofthisclassstartingwithtest_,willbeinterpretedasatest.Asyoucansee,weneedtodecoratetest_filter_intswithpatch(#6).Understandingthispartiscrucial,weneedtopatchtheobjectwhereitisactuallyused.Inthiscase,thepathisverysimple:ch7.filter_func.is_positive.

TipPatchingcanbeverytricky,soIurgeyoutoreadtheWheretopatchsectioninthemockdocumentation:https://docs.python.org/3/library/unittest.mock.html#where-to-patch.

Whenwedecorateafunctionusingpatch,likeinourexample,wegetanextraargumentinthetestsignature(#7),whichIliketocallasthepatchedfunctionname,plusa_mocksuffix,justtomakeitclearthattheobjecthasbeenpatched(ormockedout).).

Finally,wegettothebodyofthetest,andwehaveaverysimplepreparationphaseinwhichwesetupalistwithatleastonerepresentativeofalltheintegernumbercategories(negative,zero,andpositive).

Then,in#8,weperformtheexecutionphase,whichrunsthefilter_intsfunction,withoutcollectingitsresults.Ifallhasgoneasexpected,thefakeis_positivefunctionmusthavebeencalledwitheachoftheintegersinv.

Wecanverifythisbycomparingalistofcallobjectstothecall_args_listattributeonthemock(#9).Thisattributeisthelistofallthecallsperformedontheobjectsinceitscreation.

Let’srunthistest.Firstofall,makesurethatyouinstallnose($pipfreezewilltellyouifyouhaveitalready):

$pipinstallnose

Then,changeintotherootoftheproject(mineiscalledlearning.python),andrunthetestslikethis:

$noseteststests/test_ch7/

.

------------------------------------------------------------

Ran1testin0.006s

OK

Theoutputshowsonedot(eachdotisatest),aseparationline,andthetimetakentorunthewholetestsuite.ItalsosaysOKattheend,whichmeansthatourtestswereallsuccessful.

MakingatestfailGood,sojustforfunlet’smakeonefail.Inthetestfile,changethelastcallfromcall(8)tocall(9),andrunthetestsagain:

$noseteststests/test_ch7/

F

============================================================

FAIL:test_filter_ints(test_filter_funcs.FilterIntsTestCase)

------------------------------------------------------------

Traceback(mostrecentcalllast):

File"/usr/lib/python3.4/unittest/mock.py",line1125,inpatched

returnfunc(*args,**keywargs)

File"/home/fab/srv/learning.python/tests/test_ch7/test_filter_funcs.py",

line21,intest_filter_ints

is_positive_mock.call_args_list

AssertionError:[call(3),call(-4),call(0),call(5),call(9)]!=[call(3),

call(-4),call(0),call(5),call(8)]

------------------------------------------------------------

Ran1testin0.008s

FAILED(failures=1)

Wow,wemadethebeastangry!Somuchwonderfulinformation,though.Thistellsyouthatthetesttest_filter_ints(withthepathtoit),wasrunandthatitfailed(thebigFatthetop,wherethedotwasbefore).ItgivesyouaTraceback,thattellsyouthatinthetest_filter_funcs.pymodule,atline21,whenassertingonis_positive_mock.call_args_list,wehaveadiscrepancy.Thetestexpectsthelistofcallstoendwithacall(9)instance,butthereallistendswithacall(8).Thisisnothinglessthanwonderful.

Ifyouhaveatestlikethis,canyouimaginewhatwouldhappenifyourefactoredandintroducedabugintoyourfunctionbymistake?Well,yourtestswillbreak!Theywilltell

youthatyouhavescrewedsomethingup,andhere’sthedetails.So,yougoandcheckoutwhatyoubroke.

InterfacetestingLet’saddanothertestthatchecksonthereturnedvalue.It’sanothermethodintheclass,soIwon’treproducethewholecodeagain:tests/test_ch7/test_filter_funcs.py

deftest_filter_ints_return_value(self):

v=[3,-4,0,-2,5,0,8,-1]

result=filter_ints(v)

assert_list_equal([3,5,8],result)

Thistestisabitdifferentfromthepreviousone.Firstly,wecannotmocktheis_positivefunction,otherwisewewouldn’tbeabletocheckontheresult.Secondly,wedon’tcheckoncalls,butonlyontheresultofthefunctionwheninputisgiven.

Ilikethistestmuchmorethanthepreviousone.Thistypeoftestiscalledaninterfacetestbecauseitchecksontheinterface(thesetofinputsandoutputs)ofthefunctionwe’retesting.Itdoesn’tuseanymocks,whichiswhyIusethistechniquemuchmorethanthepreviousone.Let’srunthenewtestsuiteandthenlet’sseewhyIlikeinterfacetestingmorethanthosewithmocks.

$noseteststests/test_ch7/

..

------------------------------------------------------------

Ran2testsin0.006s

OK

Twotestsran,allgood(Ichangedthat9backtoan8inthefirsttest,ofcourse).

ComparingtestswithandwithoutmocksNow,let’sseewhyIdon’treallylikemocksandusethemonlywhenIhavenochoice.Let’srefactorthecodeinthisway:filter_funcs_refactored.py

deffilter_ints(v):

v=[numfornuminvifnum!=0]#1

return[numfornuminvifis_positive(num)]

Thecodeforis_positiveisthesameasbefore.Butthelogicinfilter_intshasnowchangedinawaythatis_positivewillneverbecalledwitha0,sincetheyareallfilteredoutin#1.Thisleadstoaninterestingresult,solet’srunthetestsagain:

$noseteststests/test_ch7/test_filter_funcs_refactored.py

F.

============================================================

FAIL:test_filter_ints(test_filter_funcs_refactored.FilterIntsTestCase)

------------------------------------------------------------

...omit…

AssertionError:[call(3),call(-4),call(0),call(5),call(8)]!=[call(3),

call(-4),call(5),call(8)]

------------------------------------------------------------

Ran2testsin0.002s

FAILED(failures=1)

Onetestsucceededbuttheotherone,theonewiththemockedis_positivefunction,failed.TheAssertionErrormessageshowsusthatwenowneedtoamendthelistofexpectedcalls,removingcall(0),becauseitisnolongerperformed.

Thisisnotgood.Wehavechangedneithertheinterfaceofthefunctionnoritsbehavior.Thefunctionisstillkeepingtoitsoriginalcontract.Whatwe’vedonebytestingitwithamockedobjectislimitourselves.Infact,wenowhavetoamendthetestinordertousethenewlogic.

Thisisjustasimpleexamplebutitshowsoneimportantflawinthewholemockmechanism.Youmustkeepyourmocksup-to-dateandinsyncwiththecodetheyarereplacing,otherwiseyouriskhavingissuesliketheprecedingone,orevenworse.Yourtestsmaynotfailbecausetheyareusingmockedobjectsthatperformfine,butbecausetherealones,nownotinsyncanymore,areactuallyfailing.

Sousemocksonlywhennecessary,onlywhenthereisnootherwayoftestingyourfunctions.Whenyoucrosstheboundariesofyourapplicationinatest,trytouseareplacement,likeatestdatabase,orafakeAPI,andonlywhenit’snotpossible,resorttomocks.Theyareverypowerful,butalsoverydangerouswhennothandledproperly.

So,let’sremovethatfirsttestandkeeponlythesecondone,sothatIcanshowyouanotherissueyoucouldrunintowhenwritingtests.Thewholetestmodulenowlookslikethis:tests/test_ch7/test_filter_funcs_final.py

fromunittestimportTestCase

fromnose.toolsimportassert_list_equal

fromch7.filter_funcsimportfilter_ints

classFilterIntsTestCase(TestCase):

deftest_filter_ints_return_value(self):

v=[3,-4,0,-2,5,0,8,-1]

result=filter_ints(v)

assert_list_equal([3,5,8],result)

Ifwerunit,itwillpass.

Abriefchatabouttriangulation.Nowletmeaskyou:whathappensifIchangemyfilter_intsfunctiontothis?filter_funcs_triangulation.py

deffilter_ints(v):

return[3,5,8]

Ifyourunthetestsuite,thetestwehavewillstillpass!YoumaythinkI’mcrazybutI’mshowingyouthisbecauseIwanttotalkaboutaconceptcalledtriangulation,whichis

veryimportantwhendoinginterfacetestingwithTDD.

Thewholeideaistoremovecheatingcode,orbadlyperformingcode,bypinpointingitfromdifferentangles(likegoingtoonevertexofatrianglefromtheothertwo)inawaythatmakesitimpossibleforourcodetocheat,andthebugisexposed.Wecansimplymodifythetestlikethis:tests/test_ch7/test_filter_funcs_final_triangulation.py

deftest_filter_ints_return_value(self):

v1=[3,-4,0,-2,5,0,8,-1]

v2=[7,-3,0,0,9,1]

assert_list_equal([3,5,8],filter_ints(v1))

assert_list_equal([7,9,1],filter_ints(v2))

Ihavemovedtheexecutionsectionintheassertionsdirectly,andyoucanseethatwe’renowpinpointingourfunctionfromtwodifferentangles,therebyrequiringthattherealcodebeinit.It’snolongerpossibleforourfunctiontocheat.

Triangulationisaverypowerfultechniquethatteachesustoalwaystrytoexerciseourcodefrommanydifferentangles,tocoverallpossibleedgecasestoexposeanyproblems.

BoundariesandgranularityLet’snowaddatestfortheis_positivefunction.Iknowit’saone-liner,butitpresentsuswithopportunitytodiscusstwoveryimportantconcepts:boundariesandgranularity.

That0inthebodyofthefunctionisaboundary,the>intheinequalityishowwebehavewithregardstothisboundary.Typically,whenyousetaboundary,youdividethespaceintothreeareas:whatliesbeforetheboundary,aftertheboundary,andontheboundaryitself.Intheexample,beforetheboundarywefindthenegativenumbers,theboundaryistheelement0and,aftertheboundary,wefindthepositivenumbers.Weneedtotesteachoftheseareastobesurewe’retestingthefunctioncorrectly.So,let’sseeonepossiblesolution(Iwilladdthetesttotheclass,butIwon’tshowtherepeatedcode):tests/test_ch7/test_filter_funcs_is_positive_loose.py

deftest_is_positive(self):

assert_equal(False,is_positive(-2))#beforeboundary

assert_equal(False,is_positive(0))#ontheboundary

assert_equal(True,is_positive(2))#aftertheboundary

Youcanseethatweareexercisingonenumberforeachdifferentareaaroundtheboundary.Doyouthinkthistestisgood?Thinkaboutitforaminutebeforereadingon.

Theanswerisno,thistestisnotgood.Notgoodenough,anyway.IfIchangethebodyoftheis_positivefunctiontoreadreturnn>1,Iwouldexpectmytesttofail,butitwon’t.-2isstillFalse,aswellas0,and2isstillTrue.Whydoesthathappen?Itisbecausewehaven’ttakengranularityproperlyintoaccount.We’redealingwithintegers,sowhatistheminimumgranularitywhenwemovefromoneintegertothenextone?It’s1.Therefore,whenwesurroundtheboundary,takingallthreeareasintoaccountisnotenough.Weneedtodoitwiththeminimumpossiblegranularity.Let’schangethetest:

tests/test_ch7/test_filter_funcs_is_positive_correct.py

deftest_is_positive(self):

assert_equal(False,is_positive(-1))

assert_equal(False,is_positive(0))

assert_equal(True,is_positive(1))

Ah,nowit’sbetter.Nowifwechangethebodyofis_positivetoreadreturnn>1,thethirdassertionwillfail,whichiswhatwewant.Canyouthinkofabettertest?tests/test_ch7/test_filter_funcs_is_positive_better.py

deftest_is_positive(self):

assert_equal(False,is_positive(0))

forninrange(1,10**4):

assert_equal(False,is_positive(-n))

assert_equal(True,is_positive(n))

Thistestisevenbetter.Wetestthefirsttenthousandintegers(bothpositiveandnegative)and0.Itbasicallyprovidesuswithabettercoveragethanjusttheoneacrosstheboundary.So,keepthisinmind.Zoomcloselyaroundeachboundarywithminimalgranularity,buttrytoexpandaswell,findingagoodcompromisebetweenoptimalcoverageandexecutionspeed.Wewouldlovetocheckthefirstbillionintegers,butwecan’twaitdaysforourteststorun.

AmoreinterestingexampleOkay,thiswasasgentleanintroductionasIcouldgiveyou,solet’smoveontosomethingmoreinteresting.Let’swriteandtestafunctionthatflattensanesteddictionarystructure.Foracoupleofyears,IhaveworkedverycloselywithTwitterandFacebookAPIs.Handlingsuchhumongousdatastructuresisnoteasy,especiallysincethey’reoftendeeplynested.Itturnsoutthatit’smucheasiertoflattentheminawaythatyoucanworkonthemwithoutlosingtheoriginalstructuralinformation,andthenrecreatethenestedstructurefromtheflatone.Togiveyouanexample,wewantsomethinglikethis:data_flatten.py

nested={

'fullname':'Alessandra',

'age':41,

'phone-numbers':['+447421234567','+447423456789'],

'residence':{

'address':{

'first-line':'AlexandraRd',

'second-line':'',

},

'zip':'N80PP',

'city':'London',

'country':'UK',

},

}

flat={

'fullname':'Alessandra',

'age':41,

'phone-numbers':['+447421234567','+447423456789'],

'residence.address.first-line':'AlexandraRd',

'residence.address.second-line':'',

'residence.zip':'N80PP',

'residence.city':'London',

'residence.country':'UK',

}

Astructurelikeflatismuchsimplertomanipulate.Beforewritingtheflattener,let’smakesomeassumptions:thekeysarestrings,weleaveeverydatastructureasitisunlessit’sadictionary,inwhichcaseweflattenit,weusethedotasseparator,butwewanttobeabletopassadifferentonetoourfunction.Here’sthecode:data_flatten.py

defflatten(data,prefix='',separator='.'):

"""Flattensanesteddictstructure."""

ifnotisinstance(data,dict):

return{prefix:data}ifprefixelsedata

result={}

for(key,value)indata.items():

result.update(

flatten(

value,

_get_new_prefix(prefix,key,separator),

separator=separator))

returnresult

def_get_new_prefix(prefix,key,separator):

return(separator.join((prefix,str(key)))

ifprefixelsestr(key))

Theprecedingexampleisnotdifficult,butalsonottrivialsolet’sgothroughit.Atfirst,wecheckifdataisadictionary.Ifit’snotadictionary,thenit’sdatathatdoesn’tneedtobeflattened;therefore,wesimplyreturneitherdataor,ifprefixisnotanemptystring,adictionarywithonekey/valuepair:prefix/data.

Ifinsteaddataisadict,weprepareanemptyresultdicttoreturn,thenweparsethelistofdata‘sitems,which,atI’msureyouwillremember,are2-tuples(key,value).Foreach(key,value)pair,werecursivelycallflattenonthem,andweupdatetheresultdictwithwhat’sreturnedbythatcall.Recursionisexcellentwhenrunningthroughnestedstructures.

Ataglance,canyouunderstandwhatthe_get_new_prefixfunctiondoes?Let’susetheinside-outtechniqueonceagain.Iseeaternaryoperatorthatreturnsthestringifiedkeywhenprefixisanemptystring.Ontheotherhand,whenprefixisanon-emptystring,weusetheseparatortojointheprefixwiththestringifiedversionofkey.Noticethatthebracesinsidethecalltojoinaren’tredundant,weneedthem.Canyoufigureoutwhy?

Let’swriteacoupleoftestsforthisfunction:tests/test_ch7/test_data_flatten.py

#...importsomitted…

classFlattenTestCase(TestCase):

deftest_flatten(self):

test_cases=[

({'A':{'B':'C','D':[1,2,3],'E':{'F':'G'}},

'H':3.14,

'J':['K','L'],

'M':'N'},

{'A.B':'C',

'A.D':[1,2,3],

'A.E.F':'G',

'H':3.14,

'J':['K','L'],

'M':'N'}),

(0,0),

('Hello','Hello'),

({'A':None},{'A':None}),

]

for(nested,flat)intest_cases:

assert_equal(flat,flatten(nested))

deftest_flatten_custom_separator(self):

nested={'A':{'B':{'C':'D'}}}

assert_equal(

{'A#B#C':'D'},flatten(nested,separator='#'))

Let’sstartfromtest_flatten.Idefinedalistof2-tuples(nested,flat),eachofwhichrepresentsatestcase(Ihighlightednestedtoeasereading).Ihaveonebigdictwiththreelevelsofnesting,andthensomesmallerdatastructuresthatwon’tchangewhenpassedtotheflattenfunction.Thesetestcasesareprobablynotenoughtocoveralledgecases,buttheyshouldgiveyouagoodideaofhowyoucouldstructureatestlikethis.Withasimpleforloop,Icyclethrougheachtestcaseandassertthattheresultofflatten(nested)isequaltoflat.

TipOnethingtosayaboutthisexampleisthat,whenyourunit,itwillshowyouthattwotestshavebeenrun.Thisisactuallynotcorrectbecauseeveniftechnicallytherewereonlytwotestsrunning,inoneofthemwehavemultipletestcases.Itwouldbenicertohavethemruninawaythattheywererecognizedasseparate.Thisispossiblethroughtheuseoflibrariessuchasnose-parameterized,whichIencourageyoutocheckout.It’sonhttps://pypi.python.org/pypi/nose-parameterized.

Ialsoprovidedasecondtesttomakesurethecustomseparatorfeatureworked.Asyoucansee,Iusedonlyonedatastructure,whichismuchsmaller.Wedon’tneedtogobigagain,nortotestotheredgecases.Remember,testsshouldmakesureofonethingandonethingonly,andtest_flatten_custom_separatorjusttakescareofverifyingwhetherornotwecanfeedtheflattenfunctionadifferentseparator.

IcouldkeepblatheringonabouttestsforaboutanotherbookifonlyIhadthespace,butunfortunately,weneedtostophere.Ihaven’ttoldyouaboutdoctests(testswritteninthedocumentationusingaPythoninteractiveshellstyle),andaboutanotherhalfamillion

thingsthatcouldbesaidaboutthissubject.You’llhavetodiscoverthatforyourself.

Takealookatthedocumentationfortheunittestmodule,thenoseandnose-parameterizedlibraries,andpytest(http://pytest.org/),andyouwillbefine.Inmyexperience,mockingandpatchingseemtobequitehardtogetagoodgraspoffordeveloperswhoarenewtothem,soallowyourselfalittletimetodigestthesetechniques.Tryandlearnthemgradually.

Test-drivendevelopmentLet’stalkbrieflyabouttest-drivendevelopmentorTDD.ItisamethodologythatwasrediscoveredbyKentBeck,whowroteTestDrivenDevelopmentbyExample,AddisonWesley–2002,whichIencourageyoutocheckoutifyouwanttolearnaboutthefundamentalsofthissubject,whichI’mquiteobsessedwith.

TDDisasoftwaredevelopmentmethodologythatisbasedonthecontinuousrepetitionofaveryshortdevelopmentcycle.

Atfirst,thedeveloperwritesatest,andmakesitrun.Thetestissupposedtocheckafeaturethatisnotyetpartofthecode.Maybeisanewfeaturetobeadded,orsomethingtoberemovedoramended.Runningthetestwillmakeitfailand,becauseofthis,thisphaseiscalledRed.

Whenthetesthasfailed,thedeveloperwritestheminimalamountofcodetomakeitpass.Whenrunningthetestsucceeds,wehavetheso-calledGreenphase.Inthisphase,itisokaytowritecodethatcheats,justtomakethetestpass(that’swhyyouwouldthenusetriangulation).Thistechniqueiscalled,fakeit‘tilyoumakeit.

Thelastpieceofthiscycleiswherethedevelopertakescareofboththecodeandthetests(inseparatetimes)andrefactorsthemuntiltheyareinthedesiredstate.ThislastphaseiscalledRefactor.

TheTDDmantrathereforerecites,Red-Green-Refactor.

Atfirst,itfeelsreallyweirdtowritetestsbeforethecode,andImustconfessittookmeawhiletogetusedtoit.Ifyousticktoit,though,andforceyourselftolearnthisslightlycounter-intuitivewayofworking,atsomepointsomethingalmostmagicalhappens,andyouwillseethequalityofyourcodeincreaseinawaythatwouldn’tbepossibleotherwise.

Whenyouwriteyourcodebeforethetests,youhavetotakecareofwhatthecodehastodoandhowithastodoit,bothatthesametime.Ontheotherhand,whenyouwritetestsbeforethecode,youcanconcentrateonthewhatpartalone,whileyouwritethem.Whenyouwritethecodeafterwards,youwillmostlyhavetotakecareofhowthecodehastoimplementwhatisrequiredbythetests.Thisshiftinfocusallowsyourmindtoconcentrateonthewhatandhowpartsinseparatemoments,yieldingabrainpowerboostthatwillsurpriseyou.

Thereareseveralotherbenefitsthatcomefromtheadoptionofthistechnique:

Youwillrefactorwithmuchmoreconfidence:Becausewhenyoutouchyourcodeyouknowthatifyouscrewthingsup,youwillbreakatleastonetest.Moreover,youwillbeabletotakecareofthearchitecturaldesignintherefactorphase,wherehavingteststhatactasguardianswillallowyoutoenjoymassagingthecodeuntilitreachesastatethatsatisfiesyou.Thecodewillbemorereadable:Thisiscrucialinourtime,whencodingisasocial

activityandeveryprofessionaldeveloperspendsmuchmoretimereadingcodethanwritingit.Thecodewillbemoreloose-coupledandeasiertotestandmaintain:Thisissimplybecausewritingthetestsfirstforcesyoutothinkmoredeeplyaboutitsstructure.Writingtestsfirstrequiresyoutohaveabetterunderstandingofthebusinessrequirements:Thisisfundamentalindeliveringwhatwasactuallyaskedfor.Ifyourunderstandingoftherequirementsislackinginformation,you’llfindwritingatestextremelychallengingandthissituationactsasasentinelforyou.Havingeverythingunittestedmeansthecodewillbeeasiertodebug:Moreover,smalltestsareperfectforprovidingalternativedocumentation.Englishcanbemisleading,butfivelinesofPythoninasimpletestareveryhardtobemisunderstood.Higherspeed:It’sfastertowritetestsandcodethanitistowritethecodefirstandthenlosetimedebuggingit.Ifyoudon’twritetests,youwillprobablydeliverthecodesooner,butthenyouwillhavetotrackthebugsdownandsolvethem(and,restassured,therewillbebugs).ThecombinedtimetakentowritethecodeandthendebugitisusuallylongerthanthetimetakentodevelopthecodewithTDD,wherehavingtestsrunningbeforethecodeiswritten,ensuringthattheamountofbugsinitwillbemuchlowerthanintheothercase.

Ontheotherhand,themainshortcomingsofthistechniqueare:

Thewholecompanyneedstobelieveinit:Otherwiseyouwillhavetoconstantlyarguewithyourboss,whowillnotunderstandwhyittakesyousolongtodeliver.Thetruthis,itmaytakeyouabitlongertodeliverintheshortterm,butinthelongtermyougainalotwithTDD.However,itisquitehardtoseethelongtermbecauseit’snotunderournosesliketheshorttermis.Ihavefoughtbattleswithstubbornbossesinmycareer,tobeabletocodeusingTDD.Sometimesithasbeenpainful,butalwayswellworthit,andIhaveneverregretteditbecause,intheend,thequalityoftheresulthasalwaysbeenappreciated.Ifyoufailtounderstandthebusinessrequirements,thiswillreflectinthetestsyouwrite,andthereforeitwillreflectinthecodetoo:ThiskindofproblemisquitehardtospotuntilyoudoUAT,butonethingthatyoucandotoreducethelikelihoodofithappeningistopairwithanotherdeveloper.Pairingwillinevitablyrequirediscussionsaboutthebusinessrequirements,andthiswillhelphavingabetterideaaboutthembeforethetestsarewritten.Badlywrittentestsarehardtomaintain:Thisisafact.Testswithtoomanymocksorwithextraassumptionsorbadlystructureddatawillsoonbecomeaburden.Don’tletthisdiscourageyou;justkeepexperimentingandchangethewayyouwritethemuntilyoufindawaythatdoesn’trequireyouahugeamountofworkeverytimeyoutouchyourcode.

I’msopassionateaboutTDDthatwhenIinterviewforajob,IalwaysaskifthecompanyI’mabouttojoinadoptsit.Iftheanswerisno,it’skindofadeal-breakerforme.Iencourageyoutocheckitoutanduseit.Useituntilyoufeelsomethingclickinginyour

mind.Youwon’tregretit,Ipromise.

ExceptionsEventhoughIhaven’tformallyintroducedthemtoyou,bynowIexpectyoutoatleasthaveavagueideaofwhatanexceptionis.Inthepreviouschapters,we’veseenthatwhenaniteratorisexhausted,callingnextonitraisesaStopIterationexception.We’vemetIndexErrorwhenwetriedaccessingalistatapositionthatwasoutsidethevalidrange.We’vealsometAttributeErrorwhenwetriedaccessinganattributeonanobjectthatdidn’thaveit,andKeyErrorwhenwedidthesamewithakeyandadictionary.We’vealsojustmetAssertionErrorwhenrunningtests.

Now,thetimehascomeforustotalkaboutexceptions.

Sometimes,eventhoughanoperationorapieceofcodeiscorrect,thereareconditionsinwhichsomethingmaygowrong.Forexample,ifwe’reconvertinguserinputfromstringtoint,theusercouldaccidentallytypealetterinplaceofadigit,makingitimpossibleforustoconvertthatvalueintoanumber.Whendividingnumbers,wemaynotknowinadvanceifwe’reattemptingadivisionbyzero.Whenopeningafile,itcouldbemissingorcorrupted.

Whenanerrorisdetectedduringexecution,itiscalledanexception.Exceptionsarenotnecessarilylethal;infact,we’veseenthatStopIterationisdeeplyintegratedinPythongeneratoranditeratormechanisms.Normally,though,ifyoudon’ttakethenecessaryprecautions,anexceptionwillcauseyourapplicationtobreak.Sometimes,thisisthedesiredbehaviorbutinothercases,wewanttopreventandcontrolproblemssuchasthese.Forexample,wemayalerttheuserthatthefilethey’retryingtoopeniscorruptedorthatitismissingsothattheycaneitherfixitorprovideanotherfile,withouttheneedfortheapplicationtodiebecauseofthisissue.Let’sseeanexampleofafewexceptions:exceptions/first.example.py

>>>gen=(nforninrange(2))

>>>next(gen)

0

>>>next(gen)

1

>>>next(gen)

Traceback(mostrecentcalllast):

File"<stdin>",line1,in<module>

StopIteration

>>>print(undefined_var)

Traceback(mostrecentcalllast):

File"<stdin>",line1,in<module>

NameError:name'undefined_var'isnotdefined

>>>mylist=[1,2,3]

>>>mylist[5]

Traceback(mostrecentcalllast):

File"<stdin>",line1,in<module>

IndexError:listindexoutofrange

>>>mydict={'a':'A','b':'B'}

>>>mydict['c']

Traceback(mostrecentcalllast):

File"<stdin>",line1,in<module>

KeyError:'c'

>>>1/0

Traceback(mostrecentcalllast):

File"<stdin>",line1,in<module>

ZeroDivisionError:divisionbyzero

Asyoucansee,thePythonshellisquiteforgiving.WecanseetheTraceback,sothatwehaveinformationabouttheerror,buttheprogramdoesn’tdie.Thisisaspecialbehavior,aregularprogramorascriptwouldnormallydieifnothingweredonetohandleexceptions.

Tohandleanexception,Pythongivesyouthetrystatement.WhathappenswhenyouenterthetryclauseisthatPythonwillwatchoutforoneormoredifferenttypesofexceptions(accordingtohowyouinstructit),andiftheyareraised,itwillallowyoutoreact.Thetrystatementiscomprisedofthetryclause,whichopensthestatement;oneormoreexceptclauses(alloptional)thatdefinewhattodowhenanexceptioniscaught;anelseclause(optional),whichisexecutedwhenthetryclauseisexitedwithoutanyexceptionraised;andafinallyclause(optional),whosecodeisexecutedregardlessofwhateverhappenedintheotherclauses.Thefinallyclauseistypicallyusedtocleanupresources.Mindtheorder,it’simportant.Also,trymustbefollowedbyatleastoneexceptclauseorafinallyclause.Let’sseeanexample:exceptions/try.syntax.py

deftry_syntax(numerator,denominator):

try:

print('Inthetryblock:{}/{}'

.format(numerator,denominator))

result=numerator/denominator

exceptZeroDivisionErroraszde:

print(zde)

else:

print('Theresultis:',result)

returnresult

finally:

print('Exiting')

print(try_syntax(12,4))

print(try_syntax(11,0))

Theprecedingexampledefinesasimpletry_syntaxfunction.Weperformthedivisionoftwonumbers.WearepreparedtocatchaZeroDivisionErrorexceptionifwecallthefunctionwithdenominator=0.Initially,thecodeentersthetryblock.Ifdenominatorisnot0,resultiscalculatedandtheexecution,afterleavingthetryblock,resumesintheelseblock.Weprintresultandreturnit.Takealookattheoutputandyou’llnoticethatjustbeforereturningresult,whichistheexitpointofthefunction,Pythonexecutesthefinallyclause.

Whendenominatoris0,thingschange.Weentertheexceptblockandprintzde.Theelseblockisn’texecutedbecauseanexceptionwasraisedinthetryblock.Before(implicitly)returningNone,westillexecutethefinallyblock.Takealookattheoutputandseeifitmakessensetoyou:

$pythonexceptions/try.syntax.py

Inthetryblock:12/4

Theresultis:3.0

Exiting

3.0

Inthetryblock:11/0

divisionbyzero

Exiting

None

Whenyouexecuteatryblock,youmaywanttocatchmorethanoneexception.Forexample,whentryingtodecodeaJSONobject,youmayincurintoValueErrorformalformedJSON,orTypeErrorifthetypeofthedatayou’refeedingtojson.loads()isnotastring.Inthiscase,youmaystructureyourcodelikethis:exceptions/json.example.py

importjson

json_data='{}'

try:

data=json.loads(json_data)

except(ValueError,TypeError)ase:

print(type(e),e)

ThiscodewillcatchbothValueErrorandTypeError.Trychangingjson_data='{}'tojson_data=2orjson_data='{{',andyou’llseethedifferentoutput.

NoteJSONstandsforJavaScriptObjectNotationandit’sanopenstandardformatthatuseshuman-readabletexttotransmitdataobjectsconsistingofkey/valuepairs.It’sanexchangeformatwidelyusedwhenmovingdataacrossapplications,especiallywhendataneedstobetreatedinalanguageorplatform-agnosticway.

Ifyouwanttohandlemultipleexceptionsdifferently,youcanjustaddmoreexceptclauses,likethis:exceptions/multiple.except.py

try:

#somecode

exceptException1:

#reacttoException1

except(Exception2,Exception3):

#reacttoException2andException3

exceptException3:

#reacttoException3…

Keepinmindthatanexceptionishandledinthefirstblockthatdefinesthatexceptionclassoranyofitsbases.Therefore,whenyoustackmultipleexceptclauseslikewe’vejustdone,makesurethatyouputspecificexceptionsatthetopandgenericonesatthebottom.InOOPterms,childrenontop,grandparentsatthebottom.Moreover,rememberthatonlyoneexcepthandlerisexecutedwhenanexceptionisraised.

Youcanalsowritecustomexceptions.Inordertodothat,youjusthavetoinheritfrom

anyotherexceptionclass.Pythonbuilt-inexceptionsaretoomanytobelistedhere,soIhavetopointyoutowardstheofficialdocumentation.OneimportantthingtoknowisthateveryPythonexceptionderivesfromBaseException,butyourcustomexceptionsshouldneverinheritdirectlyfromthatone.Thereasonforitisthathandlingsuchanexceptionwilltrapalsosystem-exitingexceptionssuchasSystemExitandKeyboardInterrupt,whichderivefromBaseException,andthiscouldleadtosevereissues.Incaseofdisaster,youwanttobeabletoCtrl+Cyourwayoutofanapplication.

YoucaneasilysolvetheproblembyinheritingfromException,whichinheritsfromBaseException,butdoesn’tincludeanysystem-exitingexceptioninitschildrenbecausetheyaresiblingsinthebuilt-inexceptionshierarchy(seehttps://docs.python.org/3/library/exceptions.html#exception-hierarchy).

Programmingwithexceptionscanbeverytricky.Youcouldinadvertentlysilenceouterrors,ortrapexceptionsthataren’tmeanttobehandled.Playitsafebykeepinginmindafewguidelines:alwaysputinthetryclauseonlythecodethatmaycausetheexception(s)thatyouwanttohandle.Whenyouwriteexceptclauses,beasspecificasyoucan,don’tjustresorttoexceptExceptionbecauseit’seasy.Useteststomakesureyourcodehandlesedgecasesinawaythatrequirestheleastpossibleamountofexceptionhandling.Writinganexceptstatementwithoutspecifyinganyexceptionwouldcatchanyexception,thereforeexposingyourcodetothesamerisksyouincurwhenyouderiveyourcustomexceptionsfromBaseException.

Youwillfindinformationaboutexceptionsalmosteverywhereontheweb.Somecodersusethemabundantly,otherssparingly(Ibelongtothelattercategory).Findyourownwayofdealingwiththembytakingexamplesfromotherpeople’ssourcecode.There’splentyofinterestingprojectswhosesourcesareopen,andyoucanfindthemoneitherGitHub(https://github.com)orBitbucket(https://bitbucket.org/).

Beforewetalkaboutprofiling,letmeshowyouanunconventionaluseofexceptions,justtogiveyousomethingtohelpyouexpandyourviewsonthem.Theyarenotjustsimplyerrors.exceptions/for.loop.py

n=100

found=False

forainrange(n):

iffound:break

forbinrange(n):

iffound:break

forcinrange(n):

if42*a+17*b+c==5096:

found=True

print(a,b,c)#799995

Theprecedingcodeisquiteacommonidiomifyoudealwithnumbers.Youhavetoiterateoverafewnestedrangesandlookforaparticularcombinationofa,b,andcthatsatisfiesacondition.Intheexample,conditionisatriviallinearequation,butimaginesomethingmuchcoolerthanthat.Whatbugsmeishavingtocheckifthesolutionhasbeenfoundat

thebeginningofeachloop,inordertobreakoutofthemasfastaswecanwhenitis.ThebreakoutlogicinterfereswiththerestofthecodeandIdon’tlikeit,soIcameupwithadifferentsolutionforthis.Takealookatit,andseeifyoucanadaptittoothercasestoo.

exceptions/for.loop.py

classExitLoopException(Exception):

pass

try:

n=100

forainrange(n):

forbinrange(n):

forcinrange(n):

if42*a+17*b+c==5096:

raiseExitLoopException(a,b,c)

exceptExitLoopExceptionasele:

print(ele)#(79,99,95)

Canyouseehowmuchmoreelegantitis?Nowthebreakoutlogicisentirelyhandledwithasimpleexceptionwhosenameevenhintsatitspurpose.Assoonastheresultisfound,weraiseit,andimmediatelythecontrolisgiventotheexceptclausewhichhandlesit.Thisisfoodforthought.Thisexampleindirectlyshowsyouhowtoraiseyourownexceptions.Readupontheofficialdocumentationtodiveintothebeautifuldetailsofthissubject.

ProfilingPythonThereareafewdifferentwaystoprofileaPythonapplication.Profilingmeanshavingtheapplicationrunwhilekeepingtrackofseveraldifferentparameters,likethenumberoftimesafunctioniscalled,theamountoftimespentinsideit,andsoon.Profilingcanhelpusfindthebottlenecksinourapplication,sothatwecanimproveonlywhatisreallyslowingusdown.

Ifyoutakealookattheprofilingsectioninthestandardlibraryofficialdocumentation,youwillseethatthereareacoupleofdifferentimplementationsofthesameprofilinginterface:profileandcProfile.

cProfileisrecommendedformostusers,it’saCextensionwithreasonableoverheadthatmakesitsuitableforprofilinglong-runningprogramsprofileisapurePythonmodulewhoseinterfaceisimitatedbycProfile,butwhichaddssignificantoverheadtoprofiledprograms

Thisinterfacedoesdeterministprofiling,whichmeansthatallfunctioncalls,functionreturnsandexceptioneventsaremonitored,andprecisetimingsaremadefortheintervalsbetweentheseevents.Anotherapproach,calledstatisticalprofiling,randomlysamplestheeffectiveinstructionpointer,anddeduceswheretimeisbeingspent.

Thelatterusuallyinvolveslessoverhead,butprovidesonlyapproximateresults.Moreover,becauseofthewaythePythoninterpreterrunsthecode,deterministicprofilingdoesn’taddthatasmuchoverheadasonewouldthink,soI’llshowyouasimpleexampleusingcProfilefromthecommandline.

We’regoingtocalculatePythagoreantriples(Iknow,you’vemissedthem…)usingthefollowingcode:profiling/triples.py

defcalc_triples(mx):

triples=[]

forainrange(1,mx+1):

forbinrange(a,mx+1):

hypotenuse=calc_hypotenuse(a,b)

ifis_int(hypotenuse):

triples.append((a,b,int(hypotenuse)))

returntriples

defcalc_hypotenuse(a,b):

return(a**2+b**2)**.5

defis_int(n):#nisexpectedtobeafloat

returnn.is_integer()

triples=calc_triples(1000)

Thescriptisextremelysimple;weiterateovertheinterval[1,mx]withaandb(avoidingrepetitionofpairsbysettingb>=a)andwecheckiftheybelongtoarighttriangle.Weusecalc_hypotenusetogethypotenuseforaandb,andthen,withis_int,wecheckifit

isaninteger,whichmeans(a,b,c)isaPythagoreantriple.Whenweprofilethisscript,wegetinformationintabularform.Thecolumnsarencalls,tottime,percall,cumtime,percall,andfilename:lineno(function).Theyrepresenttheamountofcallswemadetoafunction,howmuchtimewespentinit,andsoon.I’lltrimacoupleofcolumnstosavespace,soifyouruntheprofilingyourself,don’tworryifyougetadifferentresult.

$python-mcProfileprofiling/triples.py

1502538functioncallsin0.750seconds

Orderedby:standardname

ncallstottimepercallfilename:lineno(function)

5005000.4690.000triples.py:14(calc_hypotenuse)

5005000.0870.000triples.py:18(is_int)

10.0000.000triples.py:4(<module>)

10.1630.163triples.py:4(calc_triples)

10.0000.000{built-inmethodexec}

10340.0000.000{method'append'of'list'objects}

10.0000.000{method'disable'of'_lsprof.Profil…

5005000.0320.000{method'is_integer'of'float'objects}

Evenwiththislimitedamountofdata,wecanstillinfersomeusefulinformationaboutthiscode.Firstly,wecanseethatthetimecomplexityofthealgorithmwehavechosengrowswiththesquareoftheinputsize.Theamountoftimeswegetinsidetheinnerloopbodyisexactlymx(mx+1)/2.Werunthescriptwithmx=1000,whichmeansweget500500timesinsidetheinnerforloop.Threemainthingshappeninsidethatloop,wecallcalc_hypotenuse,wecallis_intand,iftheconditionismet,weappendtothetripleslist.

Takingalookattheprofilingreport,wenoticethatthealgorithmhasspent0.469secondsinsidecalc_hypotenuse,whichiswaymorethanthe0.087secondsspentinsideis_int,giventhattheywerecalledthesamenumberoftimes,solet’sseeifwecanboostcalc_hypotenusealittle.

Asitturnsout,wecan.AsImentionedearlieroninthebook,thepoweroperator**isquiteexpensive,andincalc_hypotenuse,we’reusingitthreetimes.Fortunately,wecaneasilytransformtwoofthoseintosimplemultiplications,likethis:profiling/triples.py

defcalc_hypotenuse(a,b):

return(a*a+b*b)**.5

Thissimplechangeshouldimprovethings.Ifweruntheprofilingagain,weseethatnowthe0.469isnowdownto0.177.Notbad!Thismeansnowwe’respendingonlyabout37%ofthetimeinsidecalc_hypotenuseaswewerebefore.

Let’sseeifwecanimproveis_intaswell,bychangingitlikethis:profiling/triples.py

defis_int(n):

returnn==int(n)

Thisimplementationisdifferentandtheadvantageisthatitalsoworkswhennisaninteger.Alas,whenweruntheprofilingagainstit,weseethatthetimetakeninsidethe

is_intfunctionhasgoneupto0.141seconds.Thismeansthatithasroughlydoubled,comparedtowhatitwasbefore.Inthiscase,weneedtoreverttothepreviousimplementation.

Thisexamplewastrivial,ofcourse,butenoughtoshowyouhowonecouldprofileanapplication.Havingtheamountofcallsthatareperformedagainstafunctionhelpsusunderstandbetterthetimecomplexityofouralgorithms.Forexample,youwouldn’tbelievehowmanycodersfailtoseethatthosetwoforloopsrunproportionallytothesquareoftheinputsize.

Onethingtomention:dependingonwhatsystemyou’reusing,resultsmaybedifferent.Therefore,it’squiteimportanttobeabletoprofilesoftwareonasystemthatisascloseaspossibletotheonethesoftwareisdeployedon,ifnotactuallyonthatone.

Whentoprofile?Profilingissupercool,butweneedtoknowwhenitisappropriatetodoit,andinwhatmeasureweneedtoaddresstheresultswegetfromit.

DonaldKnuthoncesaidthatprematureoptimizationistherootofalleviland,althoughIwouldn’thaveputitdownsodrastically,Idoagreewithhim.Afterall,whoamItodisagreewiththemanthatgaveusTheArtofComputerProgramming,TeX,andsomeofthecoolestalgorithmsIhaveeverstudiedwhenIwasauniversitystudent?

So,firstandforemost:correctness.Youwantyoucodetodelivertheresultcorrectly,thereforewritetests,findedgecases,andstressyourcodeineverywayyouthinkmakessense.Don’tbeprotective,don’tputthingsinthebackofyourbrainforlaterbecauseyouthinkthey’renotlikelytohappen.Bethorough.

Secondly,takecareofcodingbestpractices.Rememberreadability,extensibility,loosecoupling,modularity,anddesign.ApplyOOPprinciples:encapsulation,abstraction,singleresponsibility,open/closed,andsoon.Readupontheseconcepts.Theywillopenhorizonsforyou,andtheywillexpandthewayyouthinkaboutcode.

Thirdly,refactorlikeabeast!TheBoyScoutsRulesaystoAlwaysleavethecampgroundcleanerthanyoufoundit.Applythisruletoyourcode.

And,finally,whenalloftheabovehasbeentakencareof,thenandonlythen,youtakecareofprofiling.

Runyourprofilerandidentifybottlenecks.Whenyouhaveanideaofthebottlenecksyouneedtoaddress,startwiththeworstonefirst.Sometimes,fixingabottleneckcausesarippleeffectthatwillexpandandchangethewaytherestofthecodeworks.Sometimesthisisonlyalittle,sometimesabitmore,accordingtohowyourcodewasdesignedandimplemented.Therefore,startwiththebiggestissuefirst.

OneofthereasonsPythonissopopularisthatitispossibletoimplementitinmanydifferentways.So,ifyoufindyourselfhavingtroublesboostingupsomepartofyourcodeusingsheerPython,nothingpreventsyoufromrollingupyoursleeves,buyingacoupleofhundredlitersofcoffee,andrewritingtheslowpieceofcodeinC.Guaranteedtobefun!

SummaryInthischapter,weexploredtheworldoftesting,exceptions,andprofiling.

Itriedtogiveyouafairlycomprehensiveoverviewoftesting,especiallyunittesting,whichisthekindoftestingthatadevelopermostlydoes.IhopeIhavesucceededinchannelingthemessagethattestingisnotsomethingthatisperfectlydefinedandthatyoucanlearnfromabook.Youneedtoexperimentwithitalotbeforeyougetcomfortable.Ofalltheeffortsacodermustmakeintermsofstudyandexperimentation,I’dsaytestingisoneofthosethataremostworthit.

We’vebrieflyseenhowwecanpreventourprogramfromdyingbecauseoferrors,calledexceptions,thathappenatruntime.And,tosteerawayfromtheusualground,Ihavegivenyouanexampleofasomewhatunconventionaluseofexceptionstobreakoutofnestedforloops.That’snottheonlycase,andI’msureyou’lldiscoverothersasyougrowasacoder.

Intheend,weverybrieflytouchedbaseonprofiling,withasimpleexampleandafewguidelines.Iwantedtotalkaboutprofilingforthesakeofcompleteness,soatleastyoucanplayaroundwithit.

We’renowabouttoenterChapter8,TheEdges–GUIsandScripts,wherewe’regoingtogetourhandsdirtywithscriptsandGUIsand,hopefully,comeupwithsomethinginteresting.

NoteIamawarethatIgaveyoualotofpointersinthischapter,withnolinksordirections.I’mafraidthisisbychoice.Asacoder,therewon’tbeasingledayatworkwhenyouwon’thavetolooksomethingupinadocumentationpage,inamanual,onawebsite,andsoon.Ithinkit’svitalforacodertobeabletosearcheffectivelyfortheinformationtheyneed,soIhopeyou’llforgivemeforthisextratraining.Afterall,it’sallforyourbenefit.

Chapter8.TheEdges–GUIsandScripts “Auserinterfaceislikeajoke.Ifyouhavetoexplainit,it’snotthatgood.”

—MartinLeBlanc

Inthischapter,we’regoingtoworkonaprojecttogether.We’regoingtoprepareaverysimpleHTMLpagewithafewimages,andthenwe’regoingtoscrapeit,inordertosavethoseimages.

We’regoingtowriteascripttodothis,whichwillallowustotalkaboutafewconceptsthatI’dliketorunbyyou.We’realsogoingtoaddafewoptionstosaveimagesbasedontheirformat,andtochoosethewaywesavethem.And,whenwe’redonewiththescript,we’regoingtowriteaGUIapplicationthatdoesbasicallythesamething,thuskillingtwobirdswithonestone.Havingonlyoneprojecttoexplainwillallowmetoshowawiderrangeoftopicsinthischapter.

NoteAgraphicaluserinterface(GUI)isatypeofinterfacethatallowstheusertointeractwithanelectronicdevicethroughgraphicalicons,buttonsandwidgets,asopposedtotext-basedorcommand-lineinterfaces,whichrequirecommandsortexttobetypedonthekeyboard.Inanutshell,anybrowser,anyofficesuitesuchasLibreOffice,and,ingeneral,anythingthatpopsupwhenyouclickonanicon,isaGUIapplication.

So,ifyouhaven’talreadydoneso,thiswouldbetheperfecttimetostartaconsoleandpositionyourselfinafoldercalledch8intherootofyourprojectforthisbook.Withinthatfolder,we’llcreatetwoPythonmodules(scrape.pyandguiscrape.py)andonestandardfolder(simple_server).Withinsimple_server,we’llwriteourHTMLpage(index.html)insimple_server.Imageswillbestoredinch8/simple_server/img.Thestructureinch8shouldlooklikethis:

$tree-A

.

├──guiscrape.py

├──scrape.py

└──simple_server

├──img

│├──owl-alcohol.png

│├──owl-book.png

│├──owl-books.png

│├──owl-ebook.jpg

│└──owl-rose.jpeg

├──index.html

└──serve.sh

Ifyou’reusingeitherLinuxorMac,youcandowhatIdoandputthecodetostarttheHTTPserverinaserve.shfile.OnWindows,you’llprobablywanttouseabatchfile.

TheHTMLpagewe’regoingtoscrapehasthefollowingstructure:simple_server/index.html

<!DOCTYPEhtml>

<htmllang="en">

<head><title>CoolOwls!</title></head>

<body>

<h1>Welcometomyowlgallery</h1>

<div>

<imgsrc="img/owl-alcohol.png"height="128"/>

<imgsrc="img/owl-book.png"height="128"/>

<imgsrc="img/owl-books.png"height="128"/>

<imgsrc="img/owl-ebook.jpg"height="128"/>

<imgsrc="img/owl-rose.jpeg"height="128"/>

</div>

<p>Doyoulikemyowls?</p>

</body>

</html>

It’sanextremelysimplepage,solet’sjustnotethatwehavefiveimages,threeofwhicharePNGsandtwoareJPGs(notethateventhoughtheyarebothJPGs,oneendswith.jpgandtheotherwith.jpeg,whicharebothvalidextensionsforthisformat).

So,PythongivesyouaverysimpleHTTPserverforfreethatyoucanstartwiththefollowingcommand(inthesimple_serverfolder):

$python-mhttp.server8000

ServingHTTPon0.0.0.0port8000…

127.0.0.1--[31/Aug/201516:11:10]"GET/HTTP/1.1"200-

Thelastlineisthelogyougetwhenyouaccesshttp://localhost:8000,whereourbeautifulpagewillbeserved.Alternatively,youcanputthatcommandinafilecalledserve.sh,andjustrunthatwiththiscommand(makesureit’sexecutable):

$./serve.sh

Itwillhavethesameeffect.Ifyouhavethecodeforthisbook,yourpageshouldlooksomethinglikethis:

Feelfreetouseanyothersetofimages,aslongasyouuseatleastonePNGandoneJPG,andthatinthesrctagyouuserelativepaths,notabsolute.Igotthoselovelyowlsfromhttps://openclipart.org/.

Firstapproach–scriptingNow,let’sstartwritingthescript.I’llgothroughthesourceinthreesteps:importsfirst,thentheargumentparsinglogic,andfinallythebusinesslogic.

Theimportsscrape.py(Imports)

importargparse

importbase64

importjson

importos

frombs4importBeautifulSoup

importrequests

Goingthroughthemfromthetop,youcanseethatwe’llneedtoparsethearguments.whichwe’llfeedtothescriptitself(argparse).Wewillneedthebase64librarytosavetheimageswithinaJSONfile(base64andjson),andwe’llneedtoopenfilesforwriting(os).Finally,we’llneedBeautifulSoupforscrapingthewebpageeasily,andrequeststofetchitscontent.requestsisanextremelypopularlibraryforperformingHTTPrequests,builttoavoidthedifficultiesandquirksofusingthestandardlibraryurllibmodule.It’sbasedonthefasturllib3third-partylibrary.

NoteWewillexploretheHTTPprotocolandrequestsmechanisminChapter10,WebDevelopmentDoneRightso,fornow,let’sjust(simplistically)saythatweperformanHTTPrequesttofetchthecontentofawebpage.Wecandoitprogrammaticallyusingalibrarysuchasrequests,andit’smoreorlesstheequivalentoftypingaURLinyourbrowserandpressingEnter(thebrowserthenfetchesthecontentofawebpageandalsodisplaysittoyou).

Ofalltheseimports,onlythelasttwodon’tbelongtothePythonstandardlibrary,buttheyaresowidelyusedthroughouttheworldthatIdarenotexcludetheminthisbook.Makesureyouhavetheminstalled:

$pipfreeze|egrep-i"soup|requests"

beautifulsoup4==4.4.0

requests==2.7.0

Ofcourse,theversionnumbersmightbedifferentforyou.Ifthey’renotinstalled,usethiscommandtodoso:

$pipinstallbeautifulsoup4requests

Atthispoint,theonlythingthatIreckonmightconfuseyouisthebase64/jsoncouple,soallowmetospendafewwordsonthat.

Aswesawinthepreviouschapter,JSONisoneofthemostpopularformatsfordataexchangebetweenapplications.It’salsowidelyusedforotherpurposestoo,forexample,tosavedatainafile.Inourscript,we’regoingtooffertheusertheabilitytosaveimagesasimagefiles,orasaJSONsinglefile.WithintheJSON,we’llputadictionarywithkeysastheimagesnamesandvaluesastheircontent.Theonlyissueisthatsavingimagesinthebinaryformatistricky,andthisiswherethebase64librarycomestotherescue.Base64isaverypopularbinary-to-textencodingschemethatrepresentsbinarydatainan

ASCIIstringformatbytranslatingitintoaradix-64representation.

NoteTheradix-64representationusesthelettersA-Z,a-z,andthedigits0-9,plusthetwosymbols+and/foragrandtotalof64symbolsaltogether.Therefore,notsurprisingly,theBase64alphabetismadeupofthese64symbols.

Ifyouthinkyouhaveneverusedit,thinkagain.Everytimeyousendanemailwithanimageattachedtoit,theimagegetsencodedwithBase64beforetheemailissent.Ontherecipientside,imagesareautomaticallydecodedintotheiroriginalbinaryformatsothattheemailclientcandisplaythem.

ParsingargumentsNowthatthetechnicalitiesareoutoftheway,let’sseethesecondsectionofourscript(itshouldbeattheendofthescrape.pymodule).scrape.py(Argumentparsingandscrapertriggering)

if__name__=="__main__":

parser=argparse.ArgumentParser(

description='Scrapeawebpage.')

parser.add_argument(

'-t',

'--type',

choices=['all','png','jpg'],

default='all',

help='Theimagetypewewanttoscrape.')

parser.add_argument(

'-f',

'--format',

choices=['img','json'],

default='img',

help='Theformatimagesaresavedto.')

parser.add_argument(

'url',

help='TheURLwewanttoscrapeforimages.')

args=parser.parse_args()

scrape(args.url,args.format,args.type)

Lookatthatfirstline;itisaverycommonidiomwhenitcomestoscripting.AccordingtotheofficialPythondocumentation,thestring'__main__'isthenameofthescopeinwhichtop-levelcodeexecutes.Amodule’s__name__issetequalto'__main__'whenreadfromstandardinput,ascript,orfromaninteractiveprompt.

Therefore,ifyouputtheexecutionlogicunderthatif,theresultisthatyouwillbeabletousethemoduleasalibraryshouldyouneedtoimportanyofthefunctionsorobjectsdefinedinit,becausewhenimportingitfromanothermodule,__name__won’tbe'__main__'.Ontheotherhand,whenyourunthescriptdirectly,likewe’regoingto,__name__willbe'__main__',sotheexecutionlogicwillrun.

Thefirstthingwedothenisdefineourparser.Iwouldrecommendusingthestandardlibrarymodule,argparse,whichissimpleenoughandquitepowerful.Thereareotheroptionsoutthere,butinthiscase,argparsewillprovideuswithallweneed.

Wewanttofeedourscriptthreedifferentdata:thetypeofimageswewanttosave,theformatinwhichwewanttosavethem,andtheURLforthepagetobescraped.

ThetypecanbePNG,JPGorboth(default),whiletheformatcanbeeitherimageorJSON,imagebeingthedefault.URListheonlymandatoryargument.

So,weaddthe-toption,allowingalsothelongversion--type.Thechoicesare'all','png',and'jpg'.Wesetthedefaultto'all'andweaddahelpmessage.

Wedoasimilarprocedurefortheformatargumentallowingboththeshortandlong

syntax(-fand--format),andfinallyweaddtheurlargument,whichistheonlyonethatisspecifieddifferentlysothatitwon’tbetreatedasanoption,butratherasapositionalargument.

Inordertoparseallthearguments,allweneedisparser.parse_args().Verysimple,isn’tit?

Thelastlineiswherewetriggertheactuallogic,bycallingthescrapefunction,passingalltheargumentswejustparsed.Wewillseeitsdefinitionshortly.

Thenicethingaboutargparseisthatifyoucallthescriptbypassing-h,itwillprintaniceusagetextforyouautomatically.Let’stryitout:

$pythonscrape.py-h

usage:scrape.py[-h][-t{all,png,jpg}][-f{img,json}]url

Scrapeawebpage.

positionalarguments:

urlTheURLwewanttoscrapeforimages.

optionalarguments:

-h,--helpshowthishelpmessageandexit

-t{all,png,jpg},--type{all,png,jpg}

Theimagetypewewanttoscrape.

-f{img,json},--format{img,json}

Theformatimagesaresavedto.

Ifyouthinkaboutit,theonetrueadvantageofthisisthatwejustneedtospecifytheargumentsandwedon’thavetoworryabouttheusagetext,whichmeanswewon’thavetokeepitinsyncwiththearguments’definitioneverytimewechangesomething.Thisisprecious.

Here’safewdifferentwaystocallourscrape.pyscript,whichdemonstratethattypeandformatareoptional,andhowyoucanusetheshortandlongsyntaxtousethem:

$pythonscrape.pyhttp://localhost:8000

$pythonscrape.py-tpnghttp://localhost:8000

$pythonscrape.py--type=jpg-fjsonhttp://localhost:8000

Thefirstoneisusingdefaultvaluesfortypeandformat.ThesecondonewillsaveonlyPNGimages,andthethirdonewillsaveonlyJPGs,butinJSONformat.

ThebusinesslogicNowthatwe’veseenthescaffolding,let’sdivedeepintotheactuallogic(ifitlooksintimidatingdon’tworry;we’llgothroughittogether).Withinthescript,thislogicliesaftertheimportsandbeforetheparsing(beforetheif__name__clause):scrape.py(Businesslogic)

defscrape(url,format_,type_):

try:

page=requests.get(url)

exceptrequests.RequestExceptionasrex:

print(str(rex))

else:

soup=BeautifulSoup(page.content,'html.parser')

images=_fetch_images(soup,url)

images=_filter_images(images,type_)

_save(images,format_)

def_fetch_images(soup,base_url):

images=[]

forimginsoup.findAll('img'):

src=img.get('src')

img_url=(

'{base_url}/{src}'.format(

base_url=base_url,src=src))

name=img_url.split('/')[-1]

images.append(dict(name=name,url=img_url))

returnimages

def_filter_images(images,type_):

iftype_=='all':

returnimages

ext_map={

'png':['.png'],

'jpg':['.jpg','.jpeg'],

}

return[

imgforimginimages

if_matches_extension(img['name'],ext_map[type_])

]

def_matches_extension(filename,extension_list):

name,extension=os.path.splitext(filename.lower())

returnextensioninextension_list

def_save(images,format_):

ifimages:

ifformat_=='img':

_save_images(images)

else:

_save_json(images)

print('Done')

else:

print('Noimagestosave.')

def_save_images(images):

forimginimages:

img_data=requests.get(img['url']).content

withopen(img['name'],'wb')asf:

f.write(img_data)

def_save_json(images):

data={}

forimginimages:

img_data=requests.get(img['url']).content

b64_img_data=base64.b64encode(img_data)

str_img_data=b64_img_data.decode('utf-8')

data[img['name']]=str_img_data

withopen('images.json','w')asijson:

ijson.write(json.dumps(data))

Let’sstartwiththescrapefunction.Thefirstthingitdoesisfetchthepageatthegivenurlargument.Whatevererrormayhappenwhiledoingthis,wetrapitintheRequestExceptionrexandweprintit.TheRequestExceptionisthebaseexceptionclassforalltheexceptionsintherequestslibrary.

However,ifthingsgowell,andwehaveapagebackfromtheGETrequest,thenwecanproceed(elsebranch)andfeeditscontenttotheBeautifulSoupparser.TheBeautifulSouplibraryallowsustoparseawebpageinnotime,withouthavingtowriteallthelogicthatwouldbeneededtofindalltheimagesinapage,whichwereallydon’twanttodo.It’snotaseasyasitseems,andreinventingthewheelisnevergood.Tofetchimages,weusethe_fetch_imagesfunctionandwefilterthemwith_filter_images.Finally,wecall_savewiththeresult.

Splittingthecodeintodifferentfunctionswithmeaningfulnamesallowsustoreaditmoreeasily.Evenifyouhaven’tseenthelogicofthe_fetch_images,_filter_images,and_savefunctions,it’snothardtopredictwhattheydo,right?

_fetch_imagestakesaBeautifulSoupobjectandabaseURL.Allitdoesisloopingthroughalloftheimagesfoundonthepageandfillinginthe'name'and'url'informationabouttheminadictionary(oneperimage).Alldictionariesareaddedtotheimageslist,whichisreturnedattheend.

Thereissometrickerygoingonwhenwegetthenameofanimage.Whatwedoissplittheimg_url(http://localhost:8000/img/my_image_name.png)stringusing'/'asaseparator,andwetakethelastitemastheimagename.Thereisamorerobustwayofdoingthis,butforthisexampleitwouldbeoverkill.Ifyouwanttoseethedetailsofeachstep,trytobreakthislogicdownintosmallersteps,andprinttheresultofeachofthemtohelpyourselfunderstand.

Towardstheendofthebook,I’llshowyouanothertechniquetodebuginamuchmoreefficientway.

Anyway,byjustaddingprint(images)attheendofthe_fetch_imagesfunction,wegetthis:

[{'url':'http://localhost:8000/img/owl-alcohol.png','name':'owl-

alcohol.png'},{'url':'http://localhost:8000/img/owl-book.png','name':

'owl-book.png'},...]

Itruncatedtheresultforbrevity.Youcanseeeachdictionaryhasa'url'and'name'key/valuepair,whichwecanusetofetch,identifyandsaveourimagesaswelike.Atthispoint,Ihearyouaskingwhatwouldhappeniftheimagesonthepagewerespecifiedwithanabsolutepathinsteadofarelativeone,right?Goodquestion!

Theansweristhatthescriptwillfailtodownloadthembecausethislogicexpectsrelativepaths.IwasabouttoaddabitoflogictosolvethisissuewhenIthoughtthat,atthisstage,itwouldbeaniceexerciseforyoutodoit,soI’llleaveituptoyoutofixit.

TipHint:inspectthestartofthatsrcvariable.Ifitstartswith'http',thenit’sprobablyanabsolutepath.

Ihopethebodyofthe_filter_imagesfunctionisinterestingtoyou.Iwantedtoshowyouhowtocheckonmultipleextensionsbyusingamappingtechnique.

Inthisfunction,iftype_is'all',thennofilteringisrequired,sowejustreturnalltheimages.Ontheotherhand,whentype_isnot'all',wegettheallowedextensionsfromtheext_mapdictionary,anduseittofiltertheimagesinthelistcomprehensionthatendsthefunctionbody.Youcanseethatbyusinganotherhelperfunction,_matches_extension,Ihavemadethelistcomprehensionsimplerandmorereadable.

All_matches_extensiondoesissplitthenameoftheimagegettingitsextensionandcheckingwhetheritiswithinthelistofallowedones.Canyoufindonemicroimprovement(speed-wise)thatcouldbedonetothisfunction?

I’msurethatyou’rewonderingwhyIhavecollectedalltheimagesinthelistandthenremovedthem,insteadofcheckingwhetherIwantedtosavethembeforeaddingthemtothelist.ThefirstreasonisthatIneeded_fetch_imagesintheGUIappasitisnow.Asecondreasonisthatcombining,fetching,andfilteringwouldproducealongerandabitmorecomplicatedfunction,andI’mtryingtokeepthecomplexityleveldown.Athirdreasonisthatthiscouldbeaniceexerciseforyoutodo.Feelslikewe’repairinghere…

Let’skeepgoingthroughthecodeandinspectthe_savefunction.Youcanseethat,whenimagesisn’tempty,thisbasicallyactsasadispatcher.Weeithercall_save_imagesor_save_json,dependingonwhichinformationisstoredintheformat_variable.

Wearealmostdone.Let’sjumpto_save_images.WeloopontheimageslistandforeachdictionarywefindthereweperformaGETrequestontheimageURLandsaveitscontentinafile,whichwenameastheimageitself.Theoneimportantthingtonotehereishowwesavethatfile.

Weuseacontextmanager,representedbythekeywordwith,todothat.Python’swithstatementsupportstheconceptofaruntimecontextdefinedbyacontextmanager.Thisisimplementedusingapairofmethods(contextmanager.__enter__()andcontextmanager.__exit__(exc_type,exc_val,exc_tb))thatallowuser-defined

classestodefinearuntimecontextthatisenteredbeforethestatementbodyisexecutedandexitedwhenthestatementends.

Inourcase,usingacontextmanager,inconjunctionwiththeopenfunction,givesustheguaranteethatifanythingbadweretohappenwhilewritingthatfile,theresourcesinvolvedintheprocesswillbecleanedupandreleasedproperlyregardlessoftheerror.HaveyouevertriedtodeleteafileonWindows,onlytobepresentedwithanalertthattellsyouthatyoucannotdeletethefilebecausethereisanotherprocessthatisholdingontoit?We’reavoidingthatsortofveryannoyingthing.

Whenweopenafile,wegetahandlerforitand,nomatterwhathappens,wewanttobesurewereleaseitwhenwe’redonewiththefile.Acontextmanageristhetoolweneedtomakesureofthat.

Finally,let’snowstepintothe_save_jsonfunction.It’sverysimilartothepreviousone.Webasicallyfillinthedatadictionary.Theimagenameisthekey,andtheBase64representationofitsbinarycontentisthevalue.Whenwe’redonepopulatingourdictionary,weusethejsonlibrarytodumpitintheimages.jsonfile.I’llgiveyouasmallpreviewofthat:images.json(truncated)

{

"owl-ebook.jpg":"/9j/4AAQSkZJRgABAQEAMQAxAAD/2wBDAAEBAQ…

"owl-book.png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAEbCAYAAAB…

"owl-books.png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAElCAYAAA…

"owl-alcohol.png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAEICAYA…

"owl-rose.jpeg":"/9j/4AAQSkZJRgABAQEANAA0AAD/2wBDAAEBAQ…

}

Andthat’sit!Now,beforeproceedingtothenextsection,makesureyouplaywiththisscriptandunderstandwellhowitworks.Tryandmodifysomething,printoutintermediateresults,addanewargumentorfunctionality,orscramblethelogic.We’regoingtomigrateitintoaGUIapplicationnow,whichwilladdalayerofcomplexitysimplybecausewe’llhavetobuildtheGUIinterface,soit’simportantthatyou’rewellacquaintedwiththebusinesslogic:itwillallowyoutoconcentrateontherestofthecode.

Secondapproach–aGUIapplicationThereareseverallibrariestowriteGUIapplicationsinPython.Themostfamousonesaretkinter,wxPython,PyGTK,andPyQt.TheyallofferawiderangeoftoolsandwidgetsthatyoucanusetocomposeaGUIapplication.

TheoneI’mgoingtousefortherestofthischapteristkinter.tkinterstandsforTkinterfaceanditisthestandardPythoninterfacetotheTkGUItoolkit.BothTkandtkinterareavailableonmostUnixplatforms,MacOSX,aswellasonWindowssystems.

Let’smakesurethattkinterisinstalledproperlyonyoursystembyrunningthiscommand:

$python-mtkinter

ItshouldopenadialogwindowdemonstratingasimpleTkinterface.Ifyoucanseethat,thenwe’regoodtogo.However,ifitdoesn’twork,pleasesearchfortkinterinthePythonofficialdocumentation.Youwillfindseverallinkstoresourcesthatwillhelpyougetupandrunningwithit.

We’regoingtomakeaverysimpleGUIapplicationthatbasicallymimicsthebehaviorofthescriptwesawinthefirstpartofthischapter.Wewon’taddtheabilitytosaveJPGsorPNGssingularly,butafteryou’vegonethroughthischapter,youshouldbeabletoplaywiththecodeandputthatfeaturebackinbyyourself.

So,thisiswhatwe’reaimingfor:

Gorgeous,isn’tit?Asyoucansee,it’saverysimpleinterface(thisishowitshouldlookonUbuntu).Thereisaframe(thatis,acontainer)fortheURLfieldandtheFetchinfobutton,anotherframefortheListboxtoholdtheimagenamesandtheradiobuttontocontrolthewaywesavethem,andfinallythereisaScrape!buttonatthebottom.Wealsohaveastatusbar,whichshowsussomeinformation.

Inordertogetthislayout,wecouldjustplaceallthewidgetsonarootwindow,butthat

wouldmakethelayoutlogicquitemessyandunnecessarilycomplicated.So,instead,wewilldividethespaceusingframesandplacethewidgetsinthoseframes.Thiswaywewillachieveamuchnicerresult.So,thisisthedraftforthelayout:

WehaveaRootWindow,whichisthemainwindowoftheapplication.Wedivideitintotworows,thefirstoneinwhichweplacetheMainFrame,andthesecondoneinwhichweplacetheStatusFrame(whichwillholdthestatusbar).TheMainFrameissubsequentlydividedintothreerowsitself.InthefirstoneweplacetheURLFrame,whichholdstheURLwidgets.InthesecondoneweplacetheImgFrame,whichwillholdtheListboxandtheRadioFrame,whichwillhostalabelandtheradiobuttonwidgets.Andfinallyathirdone,whichwilljustholdtheScrapebutton.

Inordertolayoutframesandwidgets,wewillusealayoutmanagercalledgrid,thatsimplydividesupthespaceintorowsandcolumns,asinamatrix.

Now,allthecodeI’mgoingtowritecomesfromtheguiscrape.pymodule,soIwon’trepeatitsnameforeachsnippet,tosavespace.Themoduleislogicallydividedintothreesections,notunlikethescriptversion:imports,layoutlogic,andbusinesslogic.We’regoingtoanalyzethemlinebyline,inthreechunks.

Theimportsfromtkinterimport*

fromtkinterimportttk,filedialog,messagebox

importbase64

importjson

importos

frombs4importBeautifulSoup

importrequests

We’realreadyfamiliarwithmostofthese.Theinterestingbithereisthosefirsttwolines.Thefirstoneisquitecommonpractice,althoughitisbadpracticeinPythontoimportusingthestarsyntax.Youcanincurinnamecollisionsand,ifthemoduleistoobig,importingeverythingwouldbeexpensive.

Afterthat,weimportttk,filedialog,andmessageboxexplicitly,followingtheconventionalapproachusedwiththislibrary.ttkisthenewsetofstyledwidgets.Theybehavebasicallyliketheoldones,butarecapableofdrawingthemselvescorrectlyaccordingtothestyleyourOSisseton,whichisnice.

Therestoftheimportsiswhatweneedinordertocarryoutthetaskyouknowwellbynow.Notethatthereisnothingweneedtoinstallwithpipinthissecondpart,wealreadyhaveeverythingweneed.

ThelayoutlogicI’mgoingtopasteitchunkbychunksothatIcanexplainiteasilytoyou.You’llseehowallthosepieceswetalkedaboutinthelayoutdraftarearrangedandgluedtogether.WhatI’mabouttopaste,aswedidinthescriptbefore,isthefinalpartoftheguiscrape.pymodule.We’llleavethemiddlepart,thebusinesslogic,forlast.

if__name__=="__main__":

_root=Tk()

_root.title('Scrapeapp')

Asyouknowbynow,weonlywanttoexecutethelogicwhenthemoduleisrundirectly,sothatfirstlineshouldn’tsurpriseyou.

Inthelasttwolines.wesetupthemainwindow,whichisaninstanceoftheTkclass.Weinstantiateitandgiveitatitle.NotethatIusetheprependingunderscoretechniqueforallthenamesofthetkinterobjects,inordertoavoidpotentialcollisionswithnamesinthebusinesslogic.Ijustfinditcleanerlikethis,butyou’reallowedtodisagree.

_mainframe=ttk.Frame(_root,padding='5555')

_mainframe.grid(row=0,column=0,sticky=(E,W,N,S))

Here,wesetuptheMainFrame.It’sattk.Frameinstance.Weset_rootasitsparent,andgiveitsomepadding.Thepaddingisameasureinpixelsofhowmuchspaceshouldbeinsertedbetweentheinnercontentandthebordersinordertoletourlayoutbreathealittle,otherwisewehavethesardineeffect,wherewidgetsarepackedtootightly.

Thesecondlineismuchmoreinteresting.Weplacethis_mainframeonthefirstrow(0)andfirstcolumn(0)oftheparentobject(_root).Wealsosaythatthisframeneedstoextenditselfineachdirectionbyusingthestickyargumentwithallfourcardinaldirections.Ifyou’rewonderingwheretheycamefrom,it’sthefromtkinterimport*magicthatbroughtthemtous.

_url_frame=ttk.LabelFrame(

_mainframe,text='URL',padding='5555')

_url_frame.grid(row=0,column=0,sticky=(E,W))

_url_frame.columnconfigure(0,weight=1)

_url_frame.rowconfigure(0,weight=1)

Next,westartbyplacingtheURLFramedown.Thistime,theparentobjectis_mainframe,asyouwillrecallfromourdraft.ThisisnotjustasimpleFrame,butit’sactuallyaLabelFrame,whichmeanswecansetthetextargumentandexpectarectangletobedrawnaroundit,withthecontentofthetextargumentwritteninthetop-leftpartofit(checkoutthepreviouspictureifithelps).Wepositionthisframeat(0,0),andsaythatitshouldexpandtotheleftandtotheright.Wedon’tneedtheothertwodirections.

Finally,weuserowconfigureandcolumnconfiguretomakesureitbehavescorrectly,shoulditneedtoresize.Thisisjustaformalityinourpresentlayout.

_url=StringVar()

_url.set('http://localhost:8000')

_url_entry=ttk.Entry(

_url_frame,width=40,textvariable=_url)

_url_entry.grid(row=0,column=0,sticky=(E,W,S,N),padx=5)

_fetch_btn=ttk.Button(

_url_frame,text='Fetchinfo',command=fetch_url)

_fetch_btn.grid(row=0,column=1,sticky=W,padx=5)

Here,wehavethecodetolayouttheURLtextboxandthe_fetchbutton.AtextboxinthisenvironmentiscalledEntry.Weinstantiateitasusual,setting_url_frameasitsparentandgivingitawidth.Also,andthisisthemostinterestingpart,wesetthetextvariableargumenttobe_url._urlisaStringVar,whichisanobjectthatisnowconnectedtoEntryandwillbeusedtomanipulateitscontent.Therefore,wedon’tmodifythetextinthe_url_entryinstancedirectly,butbyaccessing_url.Inthiscase,wecallthesetmethodonittosettheinitialvaluetotheURLofourlocalwebpage.

Weposition_url_entryat(0,0),settingallfourcardinaldirectionsforittostickto,andwealsosetabitofextrapaddingontheleftandrightedgesbyusingpadx,whichaddspaddingonthex-axis(horizontal).Ontheotherhand,padytakescareoftheverticaldirection.

Bynow,youshouldgetthateverytimeyoucallthe.gridmethodonanobject,we’rebasicallytellingthegridlayoutmanagertoplacethatobjectsomewhere,accordingtorulesthatwespecifyasargumentsinthegrid()call.

Similarly,wesetupandplacethe_fetchbutton.Theonlyinterestingparameteriscommand=fetch_url.Thismeansthatwhenweclickthisbutton,weactuallycallthefetch_urlfunction.Thistechniqueiscalledcallback.

_img_frame=ttk.LabelFrame(

_mainframe,text='Content',padding='9000')

_img_frame.grid(row=1,column=0,sticky=(N,S,E,W))

ThisiswhatwecalledImgFrameinthelayoutdraft.Itisplacedonthesecondrowofitsparent_mainframe.ItwillholdtheListboxandtheRadioFrame.

_images=StringVar()

_img_listbox=Listbox(

_img_frame,listvariable=_images,height=6,width=25)

_img_listbox.grid(row=0,column=0,sticky=(E,W),pady=5)

_scrollbar=ttk.Scrollbar(

_img_frame,orient=VERTICAL,command=_img_listbox.yview)

_scrollbar.grid(row=0,column=1,sticky=(S,N),pady=6)

_img_listbox.configure(yscrollcommand=_scrollbar.set)

Thisisprobablythemostinterestingbitofthewholelayoutlogic.Aswedidwiththe_url_entry,weneedtodrivethecontentsofListboxbytyingittoavariable_images.WesetupListboxsothat_img_frameisitsparent,and_imagesisthevariableit’stiedto.Wealsopasssomedimensions.

Theinterestingbitcomesfromthe_scrollbarinstance.Notethat,whenweinstantiateit,wesetitscommandto_img_listbox.yview.ThisisthefirsthalfofthecontractbetweenaListboxandaScrollbar.Theotherhalfisprovidedbythe_img_listbox.configuremethod,whichsetstheyscrollcommand=_scrollbar.set.

Byprovidingthisreciprocalbond,whenwescrollonListbox,theScrollbarwillmoveaccordinglyandvice-versa,whenweoperatetheScrollbar,theListboxwillscrollaccordingly.

_radio_frame=ttk.Frame(_img_frame)

_radio_frame.grid(row=0,column=2,sticky=(N,S,W,E))

WeplacetheRadioFrame,readytobepopulated.NotethattheListboxisoccupying(0,0)on_img_frame,theScrollbar(0,1)andtherefore_radio_framewillgoin(0,2).

_choice_lbl=ttk.Label(

_radio_frame,text="Choosehowtosaveimages")

_choice_lbl.grid(row=0,column=0,padx=5,pady=5)

_save_method=StringVar()

_save_method.set('img')

_img_only_radio=ttk.Radiobutton(

_radio_frame,text='AsImages',variable=_save_method,

value='img')

_img_only_radio.grid(

row=1,column=0,padx=5,pady=2,sticky=W)

_img_only_radio.configure(state='normal')

_json_radio=ttk.Radiobutton(

_radio_frame,text='AsJSON',variable=_save_method,

value='json')

_json_radio.grid(row=2,column=0,padx=5,pady=2,sticky=W)

Firstly,weplacethelabel,andwegiveitsomepadding.Notethatthelabelandradiobuttonsarechildrenof_radio_frame.

AsfortheEntryandListboxobjects,theRadiobuttonisalsodrivenbyabondtoanexternalvariable,whichIcalled_save_method.EachRadiobuttoninstancesetsavalueargument,andbycheckingthevalueon_save_method,weknowwhichbuttonisselected.

_scrape_btn=ttk.Button(

_mainframe,text='Scrape!',command=save)

_scrape_btn.grid(row=2,column=0,sticky=E,pady=5)

Onthethirdrowof_mainframeweplacetheScrapebutton.Itscommandissave,whichsavestheimagestobelistedinListbox,afterwehavesuccessfullyparsedawebpage.

_status_frame=ttk.Frame(

_root,relief='sunken',padding='2222')

_status_frame.grid(row=1,column=0,sticky=(E,W,S))

_status_msg=StringVar()

_status_msg.set('TypeaURLtostartscraping…')

_status=ttk.Label(

_status_frame,textvariable=_status_msg,anchor=W)

_status.grid(row=0,column=0,sticky=(E,W))

Weendthelayoutsectionbyplacingdownthestatusframe,whichisasimplettk.Frame.Togiveitalittlestatusbareffect,wesetitsreliefpropertyto'sunken'andgiveitauniformpaddingof2pixels.Itneedstosticktothe_rootwindowleft,rightandbottomparts,sowesetitsstickyattributeto(E,W,S).

Wethenplacealabelinitand,thistime,wetieittoaStringVarobject,becausewewill

havetomodifyiteverytimewewanttoupdatethestatusbartext.Youshouldbeacquaintedtothistechniquebynow.

Finally,onthelastline,weruntheapplicationbycallingthemainloopmethodontheTkinstance.

_root.mainloop()

Pleaserememberthatalltheseinstructionsareplacedundertheif__name__=="__main__":clauseintheoriginalscript.

Asyoucansee,thecodetodesignourGUIapplicationisnothard.Granted,atthebeginningyouhavetoplayaroundalittlebit.Noteverythingwillworkoutperfectlyatthefirstattempt,butIpromiseyouit’sveryeasyandyoucanfindplentyoftutorialsontheweb.Let’snowgettotheinterestingbit,thebusinesslogic.

ThebusinesslogicWe’llanalyzethebusinesslogicoftheGUIapplicationinthreechunks.Thereisthefetchinglogic,thesavinglogic,andthealertinglogic.

Fetchingthewebpageconfig={}

deffetch_url():

url=_url.get()

config['images']=[]

_images.set(())#initializedasanemptytuple

try:

page=requests.get(url)

exceptrequests.RequestExceptionasrex:

_sb(str(rex))

else:

soup=BeautifulSoup(page.content,'html.parser')

images=fetch_images(soup,url)

ifimages:

_images.set(tuple(img['name']forimginimages))

_sb('Imagesfound:{}'.format(len(images)))

else:

_sb('Noimagesfound')

config['images']=images

deffetch_images(soup,base_url):

images=[]

forimginsoup.findAll('img'):

src=img.get('src')

img_url=(

'{base_url}/{src}'.format(base_url=base_url,src=src))

name=img_url.split('/')[-1]

images.append(dict(name=name,url=img_url))

returnimages

Firstofall,letmeexplainthatconfigdictionary.WeneedsomewayofpassingdatabetweentheGUIapplicationandthebusinesslogic.Now,insteadofpollutingtheglobalnamespacewithmanydifferentvariables,mypersonalpreferenceistohaveasingledictionarythatholdsalltheobjectsweneedtopassbackandforth,sothattheglobalnamespaceisn’tbecloggedupwithallthosenames,andwehaveonesingle,clean,easywayofknowingwherealltheobjectsthatareneededbyourapplicationare.

Inthissimpleexample,we’lljustpopulatetheconfigdictionarywiththeimageswefetchfromthepage,butIwantedtoshowyouthetechniquesothatyouhaveatleastanexample.ThistechniquecomesfrommyexperiencewithJavaScript.Whenyoucodeawebpage,youveryoftenimportseveraldifferentlibraries.Ifeachoftheseclutteredtheglobalnamespacewithallsortsofvariables,therewouldbesevereissuesinmakingeverythingwork,becauseofnameclashesandvariableoverriding.Theymakethecoder’slifealivinghell.

So,it’smuchbettertotryandleavetheglobalnamespaceascleanaswecan.Inthiscase,

Ifindthatusingoneconfigvariableismorethanacceptable.

Thefetch_urlfunctionisquitesimilartowhatwedidinthescript.Firstly,wegettheurlvaluebycalling_url.get().Rememberthatthe_urlobjectisaStringVarinstancethatistiedtothe_url_entryobject,whichisanEntry.ThetextfieldyouseeontheGUIistheEntry,butthetextbehindthescenesisthevalueoftheStringVarobject.

Bycallingget()on_url,wegetthevalueofthetextwhichisdisplayedin_url_entry.

Thenextstepistoprepareconfig['images']tobeanemptylist,andtoemptythe_imagesvariable,whichistiedto_img_listbox.This,ofcourse,hastheeffectofcleaningupalltheitemsin_img_listbox.

Afterthispreparationstep,wecantrytofetchthepage,usingthesametry/exceptlogicweadoptedinthescriptatthebeginningofthechapter.

Theonedifferenceisintheactionwetakeifthingsgowrong.Wecall_sb(str(rex))._sbisahelperfunctionwhosecodewe’llseeshortly.Basically,itsetsthetextinthestatusbarforus.Notagoodname,right?Ihadtoexplainitsbehaviortoyou:foodforthought.

Ifwecanfetchthepage,thenwecreatethesoupinstance,andfetchtheimagesfromit.Thelogicoffetch_imagesisexactlythesameastheoneexplainedbefore,soIwon’trepeatmyselfhere.

Ifwehaveimages,usingaquicktuplecomprehension(whichisactuallyageneratorexpressionfedtoatupleconstructor)wefeedthe_imagesStringVarandthishastheeffectofpopulatingour_img_listboxwithalltheimagenames.Finally,weupdatethestatusbar.

Iftherewerenoimages,westillupdatethestatusbar,andattheendofthefunction,regardlessofhowmanyimageswerefound,weupdateconfig['images']toholdtheimageslist.Inthisway,we’llbeabletoaccesstheimagesfromotherfunctionsbyinspectingconfig['images']withouthavingtopassthatlistaround.

SavingtheimagesThelogictosavetheimagesisprettystraightforward.Hereitis:

defsave():

ifnotconfig.get('images'):

_alert('Noimagestosave')

return

if_save_method.get()=='img':

dirname=filedialog.askdirectory(mustexist=True)

_save_images(dirname)

else:

filename=filedialog.asksaveasfilename(

initialfile='images.json',

filetypes=[('JSON','.json')])

_save_json(filename)

def_save_images(dirname):

ifdirnameandconfig.get('images'):

forimginconfig['images']:

img_data=requests.get(img['url']).content

filename=os.path.join(dirname,img['name'])

withopen(filename,'wb')asf:

f.write(img_data)

_alert('Done')

def_save_json(filename):

iffilenameandconfig.get('images'):

data={}

forimginconfig['images']:

img_data=requests.get(img['url']).content

b64_img_data=base64.b64encode(img_data)

str_img_data=b64_img_data.decode('utf-8')

data[img['name']]=str_img_data

withopen(filename,'w')asijson:

ijson.write(json.dumps(data))

_alert('Done')

WhentheuserclickstheScrapebutton,thesavefunctioniscalledusingthecallbackmechanism.

Thefirstthingthatthisfunctiondoesischeckwhetherthereareactuallyanyimagestobesaved.Ifnot,italertstheuseraboutit,usinganotherhelperfunction,_alert,whosecodewe’llseeshortly.Nofurtheractionisperformediftherearenoimages.

Ontheotherhand,iftheconfig['images']listisnotempty,saveactsasadispatcher,anditcalls_save_imagesor_save_json,accordingtowhichvalueisheldby_same_method.Remember,thisvariableistiedtotheradiobuttons,thereforeweexpectitsvaluetobeeither'img'or'json'.

Thisdispatcherisabitdifferentfromtheoneinthescript.Accordingtowhichmethodwehaveselected,adifferentactionmustbetaken.

Ifwewanttosavetheimagesasimages,weneedtoasktheusertochooseadirectory.Wedothisbycallingfiledialog.askdirectoryandassigningtheresultofthecalltothevariabledirname.Thisopensupanicedialogwindowthatasksustochooseadirectory.Thedirectorywechoosemustexist,asspecifiedbythewaywecallthemethod.Thisisdonesothatwedon’thavetowritecodetodealwithapotentiallymissingdirectorywhensavingthefiles.

Here’showthisdialogshouldlookonUbuntu:

Ifwecanceltheoperation,dirnamewillbesettoNone.

Beforefinishinganalyzingthelogicinsave,let’squicklygothrough_save_images.

It’sverysimilartotheversionwehadinthescriptsojustnotethat,atthebeginning,inordertobesurethatweactuallyhavesomethingtodo,wecheckonbothdirnameandthepresenceofatleastoneimageinconfig['images'].

Ifthat’sthecase,itmeanswehaveatleastoneimagetosaveandthepathforit,sowecanproceed.Thelogictosavetheimageshasalreadybeenexplained.Theonethingwedodifferentlythistimeistojointhedirectory(whichmeansthecompletepath)totheimagename,bymeansofos.path.join.Intheos.pathmodulethere’splentyofusefulmethodstoworkwithpathsandfilenames.

Attheendof_save_images,ifwesavedatleastoneimage,wealerttheuserthatwe’redone.

Let’sgobacknowtotheotherbranchinsave.ThisbranchisexecutedwhentheuserselectstheAsJSONradiobuttonbeforepressingtheScrapebutton.Inthiscase,wewanttosaveafile;therefore,wecannotjustaskforadirectory.Wewanttogivetheusertheabilitytochooseafilenameaswell.Hence,wefireupadifferentdialog:filedialog.asksaveasfilename.

Wepassaninitialfilename,whichisproposedtotheuserwiththeabilitytochangeitiftheydon’tlikeit.Moreover,becausewe’resavingaJSONfile,we’reforcingtheuserto

usethecorrectextensionbypassingthefiletypesargument.Itisalistwithanynumberof2-tuples(description,extension)thatrunsthelogicofthedialog.

Here’showthisdialogshouldlookonUbuntu:

Oncewehavechosenaplaceandafilename,wecanproceedwiththesavinglogic,whichisthesameasitwasinthepreviousscript.WecreateaJSONobjectfromaPythondictionary(data)thatwepopulatewithkey/valuepairsmadebytheimagesnameandBase64encodedcontent.

In_save_jsonaswell,wehavealittlecheckatthebeginningthatmakessurethatwedon’tproceedunlesswehaveafilenameandatleastoneimagetosave.

ThisensuresthatiftheuserpressestheCancelbutton,nothingbadhappens.

AlertingtheuserFinally,let’sseethealertinglogic.It’sextremelysimple.

def_sb(msg):

_status_msg.set(msg)

def_alert(msg):

messagebox.showinfo(message=msg)

That’sit!Tochangethestatusbarmessageallweneedtodoistoaccess_status_msgStringVar,asit’stiedtothe_statuslabel.

Ontheotherhand,ifwewanttoshowtheuseramorevisiblemessage,wecanfireupamessagebox.Here’showitshouldlookonUbuntu:

Themessageboxobjectcanalsobeusedtowarntheuser(messagebox.showwarning)ortosignalanerror(messagebox.showerror).Butitcanalsobeusedtoprovidedialogsthataskusifwe’resurethatwewanttoproceedorifwereallywanttodeletethatfile,andsoon.

Ifyouinspectmessageboxbysimplyprintingoutwhatdir(messagebox)returns,you’llfindmethodslikeaskokcancel,askquestion,askretrycancel,askyesno,andaskyesnocancel,aswellasasetofconstantstoverifytheresponseoftheuser,suchasCANCEL,NO,OK,OKCANCEL,YES,YESNOCANCEL,andsoon.Youcancomparethesetotheuser’schoicesothatyouknowwhatthenextactiontoexecutewhenthedialogcloses.

Howtoimprovetheapplication?Nowthatyou’reaccustomedtothefundamentalsofdesigningaGUIapplication,I’dliketogiveyousomesuggestionsonhowtomakeoursbetter.

Wecanstartfromthecodequality.Doyouthinkthiscodeisgoodenough,orwouldyouimproveit?Ifso,how?Iwouldtestit,andmakesureit’srobustandcatersforallthevariousscenariosthatausermightcreatebyclickingaroundontheapplication.IwouldalsomakesurethebehavioriswhatIwouldexpectwhenthewebsitewe’rescrapingisdownforanyreason.

Anotherthingthatwecouldimproveisthenaming.Ihaveprudentlynamedallthecomponentswithaleadingunderscore,bothtohighlighttheirsomewhat“private”nature,andtoavoidhavingnameclasheswiththeunderlyingobjectstheyarelinkedto.Butinretrospect,manyofthosecomponentscoulduseabettername,soit’sreallyuptoyoutorefactoruntilyoufindtheformthatsuitsyoubest.Youcouldstartbygivingabetternametothe_sbfunction!

Forwhatconcernstheuserinterface,youcouldtryandresizethemainapplication.Seewhathappens?Thewholecontentstaysexactlywhereitis.Emptyspaceisaddedifyouexpand,orthewholewidgetssetdisappearsgraduallyifyoushrink.Thisbehaviorisn’texactlynice,thereforeonequicksolutioncouldbetomaketherootwindowfixed(thatis,unabletoresize).

Anotherthingthatyoucoulddotoimprovetheapplicationistoaddthesamefunctionalitywehadinthescript,tosaveonlyPNGsorJPGs.Inordertodothis,youcouldplaceacomboboxsomewhere,withthreevalues:All,PNGs,JPGs,orsomethingsimilar.Theusershouldbeabletoselectoneofthoseoptionsbeforesavingthefiles.

Evenbetter,youcouldchangethedeclarationofListboxsothatit’spossibletoselectmultipleimagesatthesametime,andonlytheselectedoneswillbesaved.Ifyoumanagetodothis(it’snotashardasitseems,believeme),thenyoushouldconsiderpresentingtheListboxabitbetter,maybeprovidingalternatingbackgroundcolorsfortherows.

Anothernicethingyoucouldaddisabuttonthatopensupadialogtoselectafile.ThefilemustbeoneoftheJSONfilestheapplicationcanproduce.Onceselected,youcouldrunsomelogictoreconstructtheimagesfromtheirBase64-encodedversion.Thelogictodothisisverysimple,sohere’sanexample:

withopen('images.json','r')asf:

data=json.loads(f.read())

for(name,b64val)indata.items():

withopen(name,'wb')asf:

f.write(base64.b64decode(b64val))

Asyoucansee,weneedtoopenimages.jsoninreadmode,andgrabthedatadictionary.Oncewehaveit,wecanloopthroughitsitems,andsaveeachimagewiththeBase64decodedcontent.I’llleaveituptoyoutotiethislogictoabuttonintheapplication.

AnothercoolfeaturethatyoucouldaddistheabilitytoopenupapreviewpanethatshowsanyimageyouselectfromtheListbox,sothattheusercantakeapeekattheimagesbeforedecidingtosavethem.

Finally,onelastsuggestionforthisapplicationistoaddamenu.MaybeevenasimplemenuwithFileand?toprovidetheusualHelporAbout.Justforfun.Addingmenusisnotthatcomplicated;youcanaddtext,keyboardshortcuts,images,andsoon.

Wheredowegofromhere?IfyouareinterestedindiggingdeeperintotheworldofGUIs,thenI’dliketoofferyouthefollowingsuggestions.

Thetkinter.tixmoduleExploringtkinteranditsthemedwidgetset,tkinter.ttk,willtakeyousometime.There’smuchtolearnandplaywith.Anotherinterestingmoduletoexplore,whenyou’llbefamiliarwiththistechnology,istkinter.tix.

Thetkinter.tix(TkInterfaceExtension)moduleprovidesanadditionalveryrichsetofwidgets.TheneedforthemstemsfromthefactthatthewidgetsinthestandardTklibraryarefarfromcomplete.

Thetkinter.tixlibraryallowsustosolvethisproblembyprovidingwidgetslikeHList,ComboBox,Control(orSpinBox),andvariousscrollablewidgets.Altogether,therearemorethan40widgets.Theyallowyoutointroducedifferentinteractiontechniquesandparadigmsintoyourapplications,thusimprovingtheirqualityandusability.

TheturtlemoduleTheturtlemoduleisanextendedreimplementationoftheeponymousmodulefromthePythonstandarddistributionuptoversionPython2.5.It’saverypopularwaytointroducechildrentoprogramming.

It’sbasedontheideaofanimaginaryturtlestartingat(0,0)intheCartesianplane.Youcanprogrammaticallycommandtheturtletomoveforwardandbackwards,rotate,andsoon.andbycombiningtogetherallthepossiblemoves,allsortsofintricateshapesandimagescanbedrawn.

It’sdefinitelyworthcheckingout,ifonlytoseesomethingdifferent.

wxPython,PyQt,andPyGTKAfteryouhaveexploredthevastnessofthetkinterrealm,I’dsuggestyoutoexploreotherGUIlibraries:wxPython,PyQt,andPyGTK.Youmayfindoutoneoftheseworksbetterforyou,oritmakeseasierforyoutocodetheapplicationyouneed.

Ibelievethatcoderscanrealizetheirideasonlywhentheyareconsciousaboutwhattoolstheyhaveavailable.Ifyourtoolsetistoonarrow,yourideasmayseemimpossibleorextremelyhardtorealize,andtheyriskremainingexactlywhattheyare,justideas.

Ofcourse,thetechnologicalspectrumtodayishumongous,soknowingeverythingisnotpossible;therefore,whenyouareabouttolearnanewtechnologyoranewsubject,mysuggestionistogrowyourknowledgebyexploringbreadthfirst.

Investigateseveralthingsnottoodeeply,andthengodeepwiththeoneorthefewthatlookedmostpromising.Thiswayyou’llbeabletobeproductivewithatleastonetool,andwhenthetoolnolongerfitsyourneeds,you’llknowwheretodigdeeper,thankstoyourpreviousexploration.

TheprincipleofleastastonishmentWhendesigninganinterface,therearemanydifferentthingstobearinmind.Oneofthem,whichformeisthemostimportant,isthelaworprincipleofleastastonishment.Itbasicallystatesthatifinyourdesignanecessaryfeaturehasahighastonishingfactor,itmaybenecessarytoredesignyourapplication.Togiveyouoneexample,whenyou’reusedtoworkingwithWindows,wherethebuttonstominimize,maximizeandcloseawindowareonthetop-rightcorner,it’squitehardtoworkonLinux,wheretheyareatthetop-leftcorner.You’llfindyourselfconstantlygoingtothetop-rightcorneronlytodiscoveroncemorethatthebuttonsareontheotherside.

Ifacertainbuttonhasbecomesoimportantinapplicationsthatit’snowplacedinapreciselocationbydesigners,pleasedon’tinnovate.Justfollowtheconvention.Userswillonlybecomefrustratedwhentheyhavetowastetimelookingforabuttonthatisnotwhereit’ssupposedtobe.

ThedisregardforthisruleisthereasonwhyIcannotworkwithproductslikeJira.Ittakesmeminutestodosimplethingsthatshouldrequireseconds.

ThreadingconsiderationsThistopicisbeyondthescopeofanintroductorybooklikethis,butIdowanttomentionit.Inanutshell,athreadofexecutionisthesmallestsequenceofprogrammedinstructionsthatcanbemanagedindependentlybyascheduler.Thereasonwehavetheperceptionthatmoderncomputerscandomanythingsatthesametimeisnotonlyduetothefactthattheyhavemultipleprocessors.Theyalsosubdividetheworkindifferentthreads,whicharethenworkedoninsequence.Iftheirlifecycleissufficientlyshort,threadscanbeworkedoninonesinglego,buttypically,whathappensisthattheOSworksonathreadforalittletime,thenswitchestoanotherone,thentoanotherone,thenbacktothefirstone,andsoon.Theorderinwhichtheyareworkedondependsondifferentfactors.Theendresultisthat,becausecomputersareextremelyfastindoingthisswitching,weperceivemanythingshappeningatthesametime.

IfyouarecodingaGUIapplicationthatneedstoperformalongrunningoperationwhenabuttonisclicked,youwillseethatyourapplicationwillprobablyfreezeuntiltheoperationhasbeencarriedout.Inordertoavoidthis,andmaintaintheapplication’sresponsiveness,youmayneedtorunthattime-expensiveoperationinadifferentthreadsothattheOSwillbeabletodedicatealittlebitoftimetotheGUIeverynowandthen,tokeepitresponsive.

Threadsareanadvancedtopic,especiallyinPython.Gainagoodgraspofthefundamentalsfirst,andthenhavefunexploringthem!

SummaryInthischapter,weworkedonaprojecttogether.Wehavewrittenascriptthatscrapesaverysimplewebpageandacceptsoptionalcommandsthatalteritsbehaviorindoingso.WealsocodedaGUIapplicationtodothesamethingbyclickingbuttonsinsteadoftypingonaconsole.IhopeyouenjoyedreadingitandfollowingalongasmuchasIenjoyedwritingit.

Wesawmanydifferentconceptslikecontextmanagers,workingwithfiles,performingHTTPrequests,andwe’vetalkedaboutguidelinesforusabilityanddesign.

Ihaveonlybeenabletoscratchthesurface,buthopefully,youhaveagoodstartingpointfromwhichtoexpandyourexploration.

Throughoutthechapter,Ihavepointedyouinseveraldifferentwaysonhowtoimprovetheapplication,andIhavechallengedyouwithafewexercisesandquestions.Ihopeyouhavetakenthetimetoplaywiththoseideas.Onecanlearnalotjustbyplayingaroundwithfunapplicationsliketheonewe’vecodedtogether.

Inthenextchapter,we’regoingtotalkaboutdatascience,oratleastaboutthetoolsthataPythonprogrammerhaswhenitcomestofacingthissubject.

Chapter9.DataScience “Ifwehavedata,let’slookatdata.Ifallwehaveareopinions,let’sgowithmine.”

—JimBarksdale,formerNetscapeCEO

Datascienceisaverybroadterm,andcanassumeseveraldifferentmeaningsaccordingtocontext,understanding,tools,andsoon.Therearecountlessbooksaboutthissubject,whichisnotsuitableforthefaint-hearted.

Inordertodoproperdatascience,youneedtoknowmathematicsandstatisticsattheveryleast.Then,youmaywanttodigintoothersubjectssuchaspatternrecognitionandmachinelearningand,ofcourse,thereisaplethoraoflanguagesandtoolsyoucanchoosefrom.

UnlessItransformintoTheAmazingFabriziointhenextfewminutes,Iwon’tbeabletotalkabouteverything;Iwon’tevengetclosetoit.Therefore,inordertorenderthischaptermeaningful,we’regoingtoworkonacoolprojecttogether.

About3yearsago,Iwasworkingforatop-tiersocialmediacompanyinLondon.Istayedtherefor2years,andIwasprivilegedtoworkwithseveralpeoplewhosebrillianceIcanonlystarttodescribe.WewerethefirstintheworldtohaveaccesstotheTwitterAdsAPI,andwewerepartnerswithFacebookaswell.Thatmeansalotofdata.

Ouranalystsweredealingwithahugenumberofcampaignsandtheywerestrugglingwiththeamountofworktheyhadtodo,sothedevelopmentteamIwasapartoftriedtohelpbyintroducingthemtoPythonandtothetoolsPythongivesyoutodealwithdata.ItwasaveryinterestingjourneythatledmetomentorseveralpeopleinthecompanyandeventuallytoManilawhere,for2weeks,IgaveintensivetraininginPythonanddatasciencetoouranalyststhere.

Theprojectwe’regoingtodotogetherinthischapterisalightweightversionofthefinalexampleIpresentedtomyManilastudents.Ihaverewrittenittoasizethatwillfitthischapter,andmadeafewadjustmentshereandthereforteachingpurposes,butallthemainconceptsarethere,soitshouldbefunandinstructionalforyoutocodealong.

Onourjourney,we’regoingtomeetafewofthetoolsyoucanfindinthePythonecosystemwhenitcomestodealingwithdata,solet’sstartbytalkingaboutRomangods.

IPythonandJupyternotebookIn2001,FernandoPerezwasagraduatestudentinphysicsatCUBoulder,andwastryingtoimprovethePythonshellsothathecouldhavesomenicetieslikethosehewasusedtowhenhewasworkingwithtoolssuchasMathematicaandMaple.TheresultofthatefforttookthenameIPython.

Inanutshell,thatsmallscriptbeganasanenhancedversionofthePythonshelland,throughtheeffortofothercodersandeventuallyproperfundingfromseveraldifferentcompanies,itbecamethewonderfulandsuccessfulprojectitistoday.Some10yearsafteritsbirth,anotebookenvironmentwascreated,poweredbytechnologieslikeWebSockets,theTornadowebserver,jQuery,CodeMirror,andMathJax.TheZeroMQlibrarywasalsousedtohandlethemessagesbetweenthenotebookinterfaceandthePythoncorethatliesbehindit.

TheIPythonnotebookhasbecomesopopularandwidelyusedthateventually,allsortsofgoodieshavebeenaddedtoit.Itcanhandlewidgets,parallelcomputing,allsortsofmediaformats,andmuch,muchmore.Moreover,atsomepoint,itbecamepossibletocodeinlanguagesotherthanPythonfromwithinthenotebook.

Thishasledtoahugeprojectthatonlyrecentlyhasbeensplitintotwo:IPythonhasbeenstrippeddowntofocusmoreonthekernelpartandtheshell,whilethenotebookhasbecomeabrandnewprojectcalledJupyter.Jupyterallowsinteractivescientificcomputationstobedoneinmorethan40languages.

Thischapter’sprojectwillallbecodedandruninaJupyternotebook,soletmeexplaininafewwordswhatanotebookis.

AnotebookenvironmentisawebpagethatexposesasimplemenuandthecellsinwhichyoucanrunPythoncode.Eventhoughthecellsareseparateentitiesthatyoucanrunindividually,theyallsharethesamePythonkernel.Thismeansthatallthenamesthatyoudefineinacell(thevariables,functions,andsoon)willbeavailableinanyothercell.

NoteSimplyput,aPythonkernelisaprocessinwhichPythonisrunning.Thenotebookwebpageisthereforeaninterfaceexposedtotheuserfordrivingthiskernel.Thewebpagecommunicatestoitusingaveryfastmessagingsystem.

Apartfromallthegraphicaladvantages,thebeautytohavesuchanenvironmentconsistsintheabilityofrunningaPythonscriptinchunks,andthiscanbeatremendousadvantage.Takeascriptthatisconnectingtoadatabasetofetchdataandthenmanipulatethatdata.Ifyoudoitintheconventionalway,withaPythonscript,youhavetofetchthedataeverytimeyouwanttoexperimentwithit.Withinanotebookenvironment,youcanfetchthedatainacellandthenmanipulateandexperimentwithitinothercells,sofetchingiteverytimeisnotnecessary.

Thenotebookenvironmentisalsoextremelyhelpfulfordatasciencebecauseitallowsforstep-by-stepintrospection.Youdoonechunkofworkandthenverifyit.Youthendo

anotherchunkandverifyagain,andsoon.

It’salsoinvaluableforprototypingbecausetheresultsarethere,rightinfrontofyoureyes,immediatelyavailable.

Ifyouwanttoknowmoreaboutthesetools,pleasecheckouthttp://ipython.org/andhttp://jupyter.org/.

IhavecreatedaverysimpleexamplenotebookwithafibonaccifunctionthatgivesyouthelistofallFibonaccinumberssmallerthanagivenN.Inmybrowser,itlookslikethis:

EverycellhasanIn[]label.Ifthere’snothingbetweenthebraces,itmeansthatcellhasneverbeenexecuted.Ifthereisanumber,itmeansthatthecellhasbeenexecuted,andthenumberrepresentstheorderinwhichthecellwasexecuted.Finally,a*meansthatthecelliscurrentlybeingexecuted.

YoucanseeinthepicturethatinthefirstcellIhavedefinedthefibonaccifunction,andIhaveexecutedit.Thishastheeffectofplacingthefibonaccinameintheglobalframeassociatedwiththenotebook,thereforethefibonaccifunctionisnowavailabletotheothercellsaswell.Infact,inthesecondcell,Icanrunfibonacci(100)andseetheresultsinOut[2].Inthethirdcell,Ihaveshownyouoneoftheseveralmagicfunctionsyoucanfindinanotebookinthesecondcell.%timeitrunsthecodeseveraltimesandprovidesyouwithanicebenchmarkforit.AllthemeasurementsforthelistcomprehensionsandgeneratorsIdidinChapter5,SavingTimeandMemorywerecarriedoutwiththisnicefeature.

Youcanexecuteacellasmanytimesasyouwant,andchangetheorderinwhichyourunthem.Cellsareverymalleable,youcanalsoputinmarkdowntextorrenderthemas

headers.

NoteMarkdownisalightweightmarkuplanguagewithplaintextformattingsyntaxdesignedsothatitcanbeconvertedtoHTMLandmanyotherformats.

Also,whateveryouplaceinthelastrowofacellwillbeautomaticallyprintedforyou.Thisisveryhandybecauseyou’renotforcedtowriteprint(...)explicitly.

Feelfreetoexplorethenotebookenvironment;onceyou’refriendswithit,it’salong-lastingrelationship,Ipromise.

Inordertorunthenotebook,youhavetoinstallahandfuloflibraries,eachofwhichcollaborateswiththeotherstomakethewholethingwork.Alternatively,youcanjustinstallJupyteranditwilltakecareofeverythingforyou.Forthischapter,thereareafewotherdependenciesthatweneedtoinstall,sopleaserunthefollowingcommand:

$pipinstalljupyterpandasmatplotlibfake-factorydeloreanxlwt

Don’tworry,I’llintroduceyoutoeachofthesegradually.Now,whenyou’redoneinstallingtheselibraries(itmaytakeafewminutes),youcanstartthenotebook:

$jupyternotebook

Thiswillopenapageinyourbrowseratthisaddress:http://localhost:8888/.

Gotothatpageandcreateanewnotebookusingthemenu.Whenyouhaveitandyou’recomfortablewithit,we’rereadytogo.

TipIfyouexperienceanyissuessettingupthenotebookenvironment,pleasedon’tgetdiscouraged.Ifyougetanerror,it’susuallyjustamatterofsearchingalittlebitonthewebandyou’llenduponapagewheresomeoneelsehashadthesameissue,andtheyhaveexplainedhowtofixit.Tryyourbesttohavethenotebookenvironmentupandrunningbeforecontinuingwiththechapter.

Ourprojectwilltakeplaceinanotebook,thereforeIwilltageachcodesnippetwiththecellnumberitbelongsto,sothatyoucaneasilyreproducethecodeandfollowalong.

TipIfyoufamiliarizeyourselfwiththekeyboardshortcuts(lookinthenotebook’shelpsection),youwillbeabletomovebetweencellsandhandletheircontentwithouthavingtoreachforthemouse.Thiswillmakeyoumoreproficientandwayfasterwhenyouworkinanotebook.

DealingwithdataTypically,whenyoudealwithdata,thisisthepathyougothrough:youfetchit,youcleanandmanipulateit,thenyouinspectitandpresentresultsasvalues,spreadsheets,graphs,andsoon.Iwantyoutobeinchargeofallthreestepsoftheprocesswithouthavinganyexternaldependencyonadataprovider,sowe’regoingtodothefollowing:

1. We’regoingtocreatethedata,simulatingthefactthatitcomesinaformatwhichisnotperfectorreadytobeworkedon.

2. We’regoingtocleanitandfeedittothemaintoolwe’lluseintheproject:DataFrameofpandas.

3. We’regoingtomanipulatethedataintheDataFrame.4. We’regoingtosavetheDataFrametoafileindifferentformats.5. Finally,we’regoingtoinspectthedataandgetsomeresultsoutofit.

SettingupthenotebookFirstthingsfirst,weneedtosetupthenotebook.Thismeansimportsandabitofconfiguration.#1

importjson

importcalendar

importrandom

fromdatetimeimportdate,timedelta

importfaker

importnumpyasnp

frompandasimportDataFrame

fromdeloreanimportparse

importpandasaspd

#makethegraphsnicer

pd.set_option('display.mpl_style','default')

Cell#1takescareoftheimports.Therearequiteafewnewthingshere:thecalendar,randomanddatetimemodulesarepartofthestandardlibrary.Theirnamesareself-explanatory,solet’slookatfaker.Thefake-factorylibrarygivesyouthismodule,whichyoucanusetopreparefakedata.It’sveryusefulintests,whenyouprepareyourfixtures,togetallsortsofthingssuchasnames,e-mailaddresses,phonenumbers,creditcarddetails,andmuchmore.Itisallfake,ofcourse.

numpyistheNumPylibrary,thefundamentalpackageforscientificcomputingwithPython.I’llspendafewwordsonitlateroninthechapter.

pandasistheverycoreuponwhichthewholeprojectisbased.ItstandsforPythonDataAnalysisLibrary.Amongmanyothers,itprovidestheDataFrame,amatrix-likedatastructurewithadvancedprocessingcapabilities.It’scustomarytoimporttheDataFrameseparatelyandthendoimportpandasaspd.

deloreanisanicethird-partylibrarythatspeedsupdealingwithdatesdramatically.Technically,wecoulddoitwiththestandardlibrary,butIseenoreasonnottoexpandabittherangeoftheexampleandshowyousomethingdifferent.

Finally,wehaveaninstructiononthelastlinethatwillmakeourgraphsattheendalittlebitnicer,whichdoesn’thurt.

PreparingthedataWewanttoachievethefollowingdatastructure:we’regoingtohavealistofuserobjects.Eachuserobjectwillbelinkedtoanumberofcampaignobjects.

InPython,everythingisanobject,soI’musingthisterminagenericway.Theuserobjectmaybeastring,adict,orsomethingelse.

Acampaigninthesocialmediaworldisapromotionalcampaignthatamediaagencyrunsonsocialmedianetworksonbehalfofaclient.

Rememberthatwe’regoingtopreparethisdatasothatit’snotinperfectshape(butitwon’tbesobadeither…).#2

fake=faker.Faker()

Firstly,weinstantiatetheFakerthatwe’llusetocreatethedata.#3

usernames=set()

usernames_no=1000

#populatethesetwith1000uniqueusernames

whilelen(usernames)<usernames_no:

usernames.add(fake.user_name())

Thenweneedusernames.Iwant1,000uniqueusernames,soIloopoverthelengthoftheusernamessetuntilithas1,000elements.Asetdoesn’tallowduplicatedelements,thereforeuniquenessisguaranteed.#4

defget_random_name_and_gender():

skew=.6#60%ofuserswillbefemale

male=random.random()>skew

ifmale:

returnfake.name_male(),'M'

else:

returnfake.name_female(),'F'

defget_users(usernames):

users=[]

forusernameinusernames:

name,gender=get_random_name_and_gender()

user={

'username':username,

'name':name,

'gender':gender,

'email':fake.email(),

'age':fake.random_int(min=18,max=90),

'address':fake.address(),

}

users.append(json.dumps(user))

returnusers

users=get_users(usernames)

users[:3]

Here,wecreatealistofusers.Eachusernamehasnowbeenaugmentedtoafull-blownuserdict,withotherdetailssuchasname,gender,e-mail,andsoon.EachuserdictisthendumpedtoJSONandaddedtothelist.Thisdatastructureisnotoptimal,ofcourse,butwe’resimulatingascenariowhereuserscometouslikethat.

Notetheskeweduseofrandom.random()tomake60%ofusersfemale.Therestofthelogicshouldbeveryeasyforyoutounderstand.

Notealsothelastline.Eachcellautomaticallyprintswhat’sonthelastline;therefore,theoutputofthisisalistwiththefirstthreeusers:Out#4

['{"gender":"F","age":48,"email":"jovani.dickinson@gmail.com",

"address":"2006SawaynTrailApt.207\\nHyattview,MO27278","username":

"darcy00","name":"VirgiaHilpert"}',

'{"gender":"F","age":58,"email":"veum.javen@hotmail.com","address":

"5176AndresPlainsApt.040\\nLakinside,GA92446","username":

"renner.virgie","name":"MissClarabelleKertzmannMD"}',

'{"gender":"M","age":33,"email":"turner.felton@rippin.com",

"address":"1218JacobsonFort\\nNorthDoctor,OK04469","username":

"hettinger.alphonsus","name":"LudwigProsacco"}']

NoteIhopeyou’refollowingalongwithyourownnotebook.Ifyoudo,pleasenotethatalldataisgeneratedusingrandomfunctionsandvalues;therefore,youwillseedifferentresults.Theywillchangeeverytimeyouexecutethenotebook.#5

#campaignnameformat:

#InternalType_StartDate_EndDate_TargetAge_TargetGender_Currency

defget_type():

#justsomegibberishinternalcodes

types=['AKX','BYU','GRZ','KTR']

returnrandom.choice(types)

defget_start_end_dates():

duration=random.randint(1,2*365)

offset=random.randint(-365,365)

start=date.today()-timedelta(days=offset)

end=start+timedelta(days=duration)

def_format_date(date_):

returndate_.strftime("%Y%m%d")

return_format_date(start),_format_date(end)

defget_age():

age=random.randint(20,45)

age-=age%5

diff=random.randint(5,25)

diff-=diff%5

return'{}-{}'.format(age,age+diff)

defget_gender():

returnrandom.choice(('M','F','B'))

defget_currency():

returnrandom.choice(('GBP','EUR','USD'))

defget_campaign_name():

separator='_'

type_=get_type()

start_end=separator.join(get_start_end_dates())

age=get_age()

gender=get_gender()

currency=get_currency()

returnseparator.join(

(type_,start_end,age,gender,currency))

In#5,wedefinethelogictogenerateacampaignname.Analystsusespreadsheetsallthetimeandtheycomeupwithallsortsofcodingtechniquestocompressasmuchinformationaspossibleintothecampaignnames.TheformatIchoseisasimpleexampleofthattechnique:thereisacodethattellsthecampaigntype,thenstartandenddates,thenthetargetageandgender,andfinallythecurrency.Allvaluesareseparatedbyanunderscore.

Intheget_typefunction,Iuserandom.choice()togetonevaluerandomlyoutofacollection.Probablymoreinterestingisget_start_end_dates.First,Igetthedurationforthecampaign,whichgoesfrom1dayto2years(randomly),thenIgetarandomoffsetintimewhichIsubtractfromtoday’sdateinordertogetthestartdate.Giventhattheoffsetisarandomnumberbetween-365and365,wouldanythingbedifferentifIaddedittotoday’sdateinsteadofsubtractingit?

WhenIhaveboththestartandenddates,Ireturnastringifiedversionofthem,joinedbyanunderscore.

Then,wehaveabitofmodulartrickerygoingonwiththeagecalculation.Ihopeyourememberthemodulooperator(%)fromChapter2,Built-inDataTypes.

WhathappenshereisthatIwantadaterangethathasmultiplesof5asextremes.So,therearemanywaystodoit,butwhatIdoistogetarandomnumberbetween20and45fortheleftextreme,andremovetheremainderofthedivisionby5.So,if,forexample,Iget28,Iwillremove28%5=3toit,getting25.Icouldhavejustusedrandom.randrange(),butit’shardtoresistmodulardivision.

Therestofthefunctionsarejustsomeotherapplicationsofrandom.choice()andthelastone,get_campaign_name,isnothingmorethanacollectorforallthesepuzzlepiecesthatreturnsthefinalcampaignname.#6

defget_campaign_data():

name=get_campaign_name()

budget=random.randint(10**3,10**6)

spent=random.randint(10**2,budget)

clicks=int(random.triangular(10**2,10**5,0.2*10**5))

impressions=int(random.gauss(0.5*10**6,2))

return{

'cmp_name':name,

'cmp_bgt':budget,

'cmp_spent':spent,

'cmp_clicks':clicks,

'cmp_impr':impressions

}

In#6,wewriteafunctionthatcreatesacompletecampaignobject.Iusedafewdifferentfunctionsfromtherandommodule.random.randint()givesyouanintegerbetweentwoextremes.Theproblemwithitisthatitfollowsauniformprobabilitydistribution,whichmeansthatanynumberintheintervalhasthesameprobabilityofcomingup.

Therefore,whendealingwithalotofdata,ifyoudistributeyourfixturesusingauniformdistribution,theresultsyouwillgetwillalllooksimilar.Forthisreason,Ichosetousetriangularandgauss,forclicksandimpressions.Theyusedifferentprobabilitydistributionssothatwe’llhavesomethingmoreinterestingtoseeintheend.

Justtomakesurewe’reonthesamepagewiththeterminology:clicksrepresentsthenumberofclicksonacampaignadvertisement,budgetisthetotalamountofmoneyallocatedforthecampaign,spentishowmuchofthatmoneyhasalreadybeenspent,andimpressionsisthenumberoftimesthecampaignhasbeenfetched,asaresource,fromitssource,regardlessoftheamountofclicksthatwereperformedonthecampaign.Normally,theamountofimpressionsisgreaterthantheamountofclicks.

Nowthatwehavethedata,it’stimetoputitalltogether:#7

defget_data(users):

data=[]

foruserinusers:

campaigns=[get_campaign_data()

for_inrange(random.randint(2,8))]

data.append({'user':user,'campaigns':campaigns})

returndata

Asyoucansee,eachitemindataisadictwithauserandalistofcampaignsthatareassociatedwiththatuser.

CleaningthedataLet’sstartcleaningthedata:#8

rough_data=get_data(users)

rough_data[:2]#let'stakeapeek

Wesimulatefetchingthedatafromasourceandtheninspectit.Thenotebookistheperfecttooltoinspectyoursteps.Youcanvarythegranularitytoyourneeds.Thefirstiteminrough_datalookslikethis:

[{'campaigns':[{'cmp_bgt':130532,

'cmp_clicks':25576,

'cmp_impr':500001,

'cmp_name':'AKX_20150826_20170305_35-50_B_EUR',

'cmp_spent':57574},

...omit…

{'cmp_bgt':884396,

'cmp_clicks':10955,

'cmp_impr':499999,

'cmp_name':'KTR_20151227_20151231_45-55_B_GBP',

'cmp_spent':318887}],

'user':'{"age":44,"username":"jacob43",

"name":"HollandStrosin",

"email":"humberto.leuschke@brakus.com",

"address":"1038RunolfsdottirParks\\nElmapo…",

"gender":"M"}'}]

So,wenowstartworkingwithit.#9

data=[]

fordatuminrough_data:

forcampaignindatum['campaigns']:

campaign.update({'user':datum['user']})

data.append(campaign)

data[:2]#let'stakeanotherpeek

ThefirstthingweneedtodoinordertobeabletofeedaDataFramewiththisdataistodenormalizeit.Thismeanstransformingthedataintoalistwhoseitemsarecampaigndicts,augmentedwiththeirrelativeuserdict.Userswillbeduplicatedineachcampaigntheybelongto.Thefirstitemindatalookslikethis:

[{'cmp_bgt':130532,

'cmp_clicks':25576,

'cmp_impr':500001,

'cmp_name':'AKX_20150826_20170305_35-50_B_EUR',

'cmp_spent':57574,

'user':'{"age":44,"username":"jacob43",

"name":"HollandStrosin",

"email":"humberto.leuschke@brakus.com",

"address":"1038RunolfsdottirParks\\nElmaport…",

"gender":"M"}'}]

Youcanseethattheuserobjecthasbeenbroughtintothecampaigndictwhichwasrepeatedforeachcampaign.

CreatingtheDataFrameNowit’stimetocreatetheDataFrame:#10

df=DataFrame(data)

df.head()

Finally,wewillcreatetheDataFrameandinspectthefirstfiverowsusingtheheadmethod.Youshouldseesomethinglikethis:

Jupyterrenderstheoutputofthedf.head()callasHTMLautomatically.Inordertohaveatext-basedoutput,simplywrapdf.head()inaprintcall.

TheDataFramestructureisverypowerful.Itallowsustodoagreatdealofmanipulationonitscontents.Youcanfilterbyrows,columns,aggregateondata,andmanyotheroperations.YoucanoperatewithrowsorcolumnswithoutsufferingthetimepenaltyyouwouldhavetopayifyouwereworkingondatawithpurePython.Thishappensbecause,underthecovers,pandasisharnessingthepoweroftheNumPylibrary,whichitselfdrawsitsincrediblespeedfromthelow-levelimplementationofitscore.NumPystandsforNumericPython,anditisoneofthemostwidelyusedlibrariesinthedatascienceenvironment.

UsingaDataFrameallowsustocouplethepowerofNumPywithspreadsheet-likecapabilitiessothatwe’llbeabletoworkonourdatainafashionthatissimilartowhatananalystcoulddo.Only,wedoitwithcode.

Butlet’sgobacktoourproject.Let’sseetwowaystoquicklygetabird’seyeviewofthedata:#11

df.count()

countyieldsacountofallthenon-emptycellsineachcolumn.Thisisgoodtohelpyouunderstandhowsparseyourdatacanbe.Inourcase,wehavenomissingvalues,sotheoutputis:

cmp_bgt4974

cmp_clicks4974

cmp_impr4974

cmp_name4974

cmp_spent4974

user4974

dtype:int64

Nice!Wehave4,974rows,andthedatatypeisintegers(dtype:int64meanslongintegersbecausetheytake64bitseach).Giventhatwehave1,000usersandtheamountofcampaignsperuserisarandomnumberbetween2and8,we’reexactlyinlinewithwhatIwasexpecting.#12

df.describe()

describeisaniceandquickwaytointrospectabitfurther:

cmp_bgtcmp_clickscmp_imprcmp_spent

count4974.0000004974.0000004974.0000004974.000000

mean503272.70687640225.764978499999.495979251150.604343

std289393.74746521910.6319502.035355220347.594377

min1250.000000609.000000499992.000000142.000000

25%253647.50000022720.750000499998.00000067526.750000

50%508341.00000036561.500000500000.000000187833.000000

75%757078.25000055962.750000500001.000000385803.750000

max999631.00000098767.000000500006.000000982716.000000

Asyoucansee,itgivesusseveralmeasuressuchascount,mean,std(standarddeviation),min,max,andshowshowdataisdistributedinthevariousquadrants.Thankstothismethod,wecouldalreadyhavearoughideaofhowourdataisstructured.

Let’sseewhicharethethreecampaignswiththehighestandlowestbudgets:#13

df.sort_index(by=['cmp_bgt'],ascending=False).head(3)

Thisgivesthefollowingoutput(truncated):

cmp_bgtcmp_clickscmp_imprcmp_name

465599963115343499997AKX_20160814_20180226_40

370899960645367499997KTR_20150523_20150527_35

199599944512580499998AKX_20141102_20151009_30

And(#14)acallto.tail(3),showsustheoneswiththelowestbudget.

UnpackingthecampaignnameNowit’stimetoincreasethecomplexityupabit.Firstofall,wewanttogetridofthathorriblecampaignname(cmp_name).Weneedtoexplodeitintopartsandputeachpartinonededicatedcolumn.Inordertodothis,we’llusetheapplymethodoftheSeriesobject.

Thepandas.core.series.Seriesclassisbasicallyapowerfulwrapperaroundanarray(thinkofitasalistwithaugmentedcapabilities).WecanextrapolateaSeriesobjectfromaDataFramebyaccessingitinthesamewaywedowithakeyinadict,andwecancallapplyonthatSeriesobject,whichwillrunafunctionfeedingeachitemintheSeriestoit.WecomposetheresultintoanewDataFrame,andthenjointhatDataFramewithourbeloveddf.

#15

defunpack_campaign_name(name):

#veryoptimisticmethod,assumesdataincampaignname

#isalwaysingoodstate

type_,start,end,age,gender,currency=name.split('_')

start=parse(start).date

end=parse(end).date

returntype_,start,end,age,gender,currency

campaign_data=df['cmp_name'].apply(unpack_campaign_name)

campaign_cols=[

'Type','Start','End','Age','Gender','Currency']

campaign_df=DataFrame(

campaign_data.tolist(),columns=campaign_cols,index=df.index)

campaign_df.head(3)

Withinunpack_campaign_name,wesplitthecampaignnameinparts.Weusedelorean.parse()togetaproperdateobjectoutofthosestrings(deloreanmakesitreallyeasytodoit,doesn’tit?),andthenwereturntheobjects.Aquickpeekatthelastlinereveals:

TypeStartEndAgeGenderCurrency

0KTR2016-06-162017-01-2420-30MEUR

1BYU2014-10-252015-07-3135-50BUSD

2BYU2015-10-262016-03-1735-50MEUR

Nice!Oneimportantthing:evenifthedatesappearasstrings,theyarejusttherepresentationoftherealdateobjectsthatarehostedintheDataFrame.

Anotherveryimportantthing:whenjoiningtwoDataFrameinstances,it’simperativethattheyhavethesameindex,otherwisepandaswon’tbeabletoknowwhichrowsgowithwhich.Therefore,whenwecreatecampaign_df,wesetitsindextotheonefromdf.Thisenablesustojointhem.WhencreatingthisDataFrame,wealsopassthecolumnsnames.#16

df=df.join(campaign_df)

Andafterthejoin,wetakeapeek,hopingtoseematchingdata(outputtruncated):#17

df[['cmp_name']+campaign_cols].head(3)

Gives:

cmp_nameTypeStartEnd

0KTR_20160616_20170124_20-30_M_EURKTR2016-06-162017-01-24

1BYU_20141025_20150731_35-50_B_USDBYU2014-10-252015-07-31

2BYU_20151026_20160317_35-50_M_EURBYU2015-10-262016-03-17

Asyoucansee,thejoinwassuccessful;thecampaignnameandtheseparatecolumnsexposethesamedata.Didyouseewhatwedidthere?We’reaccessingtheDataFrameusingthesquarebracketssyntax,andwepassalistofcolumnnames.ThiswillproduceabrandnewDataFrame,withthosecolumns(inthesameorder),onwhichwethencall

head().

UnpackingtheuserdataWenowdotheexactsamethingforeachpieceofuserJSONdata.WecallapplyontheuserSeries,runningtheunpack_user_jsonfunction,whichtakesaJSONuserobjectandtransformsitintoalistofitsfields,whichwecantheninjectintoabrandnewDataFrameuser_df.Afterthat,we’lljoinuser_dfbackwithdf,likewedidwithcampaign_df.#18

defunpack_user_json(user):

#veryoptimisticaswell,expectsuserobjects

#tohaveallattributes

user=json.loads(user.strip())

return[

user['username'],

user['email'],

user['name'],

user['gender'],

user['age'],

user['address'],

]

user_data=df['user'].apply(unpack_user_json)

user_cols=[

'username','email','name','gender','age','address']

user_df=DataFrame(

user_data.tolist(),columns=user_cols,index=df.index)

Verysimilartothepreviousoperation,isn’tit?Weshouldalsonoteherethat,whencreatinguser_df,weneedtoinstructDataFrameaboutthecolumnnamesand,veryimportant,theindex.Let’sjoin(#19)andtakeaquickpeek(#20):

df=df.join(user_df)

df[['user']+user_cols].head(2)

Theoutputshowsusthateverythingwentwell.We’regood,butwe’renotdoneyet.

Ifyoucalldf.columnsinacell,you’llseethatwestillhaveuglynamesforourcolumns.Let’schangethat:#21

better_columns=[

'Budget','Clicks','Impressions',

'cmp_name','Spent','user',

'Type','Start','End',

'TargetAge','TargetGender','Currency',

'Username','Email','Name',

'Gender','Age','Address',

]

df.columns=better_columns

Good!Now,withtheexceptionof'cmp_name'and'user',weonlyhavenicenames.

CompletingthedatasetNextstepwillbetoaddsomeextracolumns.Foreachcampaign,

wehavetheamountofclicksandimpressions,andwehavethespent.Thisallowsustointroducethreemeasurementratios:CTR,CPC,andCPI.TheystandforClickThroughRate,CostPerClick,andCostPerImpression,respectively.

Thelasttwoareeasytounderstand,butCTRisnot.Sufficeittosaythatitistheratiobetweenclicksandimpressions.Itgivesyouameasureofhowmanyclickswereperformedonacampaignadvertisementperimpression:thehigherthisnumber,themoresuccessfultheadvertisementisinattractinguserstoclickonit.#22

defcalculate_extra_columns(df):

#ClickThroughRate

df['CTR']=df['Clicks']/df['Impressions']

#CostPerClick

df['CPC']=df['Spent']/df['Clicks']

#CostPerImpression

df['CPI']=df['Spent']/df['Impressions']

calculate_extra_columns(df)

Iwrotethisasafunction,butIcouldhavejustwrittenthecodeinthecell.It’snotimportant.WhatIwantyoutonoticehereisthatwe’readdingthosethreecolumnswithonelineofcodeeach,buttheDataFrameappliestheoperationautomatically(thedivision,inthiscase)toeachpairofcellsfromtheappropriatecolumns.So,eveniftheyaremaskedasthreedivisions,theseareactually4974*3divisions,becausetheyareperformedforeachrow.Pandasdoesalotofworkforus,andalsodoesaverygoodjobinhidingthecomplexityofit.

Thefunction,calculate_extra_columns,takesaDataFrame,andworksdirectlyonit.Thismodeofoperationiscalledin-place.Doyourememberhowlist.sort()wassortingthelist?Itisthesamedeal.

Wecantakealookattheresultsbyfilteringontherelevantcolumnsandcallinghead.#23

df[['Spent','Clicks','Impressions',

'CTR','CPC','CPI']].head(3)

Thisshowsusthatthecalculationswereperformedcorrectlyoneachrow:

SpentClicksImpressionsCTRCPCCPI

057574255765000010.0511522.2510950.115148

1226319612474999990.1224943.6951850.452639

24354155825000040.0311640.2794250.008708

Now,Iwanttoverifytheaccuracyoftheresultsmanuallyforthefirstrow:#24

clicks=df['Clicks'][0]

impressions=df['Impressions'][0]

spent=df['Spent'][0]

CTR=df['CTR'][0]

CPC=df['CPC'][0]

CPI=df['CPI'][0]

print('CTR:',CTR,clicks/impressions)

print('CPC:',CPC,spent/clicks)

print('CPI:',CPI,spent/impressions)

Ityieldsthefollowingoutput:

CTR:0.05115189769620.0511518976962

CPC:2.251094776352.25109477635

CPI:0.1151477697040.115147769704

Thisisexactlywhatwesawinthepreviousoutput.Ofcourse,Iwouldn’tnormallyneedtodothis,butIwantedtoshowyouhowcanyouperformcalculationsthisway.YoucanaccessaSeries(acolumn)bypassingitsnametotheDataFrame,insquarebrackets,andthenyouaccesseachrowbyitsposition,exactlyasyouwouldwitharegularlistortuple.

We’realmostdonewithourDataFrame.Allwearemissingnowisacolumnthattellsusthedurationofthecampaignandacolumnthattellsuswhichdayoftheweekcorrespondstothestartdateofeachcampaign.Thisallowsmetoexpandonhowtoplaywithdateobjects.#25

defget_day_of_the_week(day):

number_to_day=dict(enumerate(calendar.day_name,1))

returnnumber_to_day[day.isoweekday()]

defget_duration(row):

return(row['End']-row['Start']).days

df['DayofWeek']=df['Start'].apply(get_day_of_the_week)

df['Duration']=df.apply(get_duration,axis=1)

Weusedtwodifferenttechniquesherebutfirst,thecode.

get_day_of_the_weektakesadateobject.Ifyoucannotunderstandwhatitdoes,pleasetakeafewmomentstotryandunderstandforyourselfbeforereadingtheexplanation.Usetheinside-outtechniquelikewe’vedoneafewtimesbefore.

So,asI’msureyouknowbynow,ifyouputcalendar.day_nameinalistcall,youget['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday',

'Sunday'].Thismeansthat,ifweenumeratecalendar.day_namestartingfrom1,wegetpairssuchas(1,'Monday'),(2,'Tuesday'),andsoon.Ifwefeedthesepairstoadict,wegetamappingbetweenthedaysoftheweekasnumbers(1,2,3,…)andtheirnames.Whenthemappingiscreated,inordertogetthenameofaday,wejustneedtoknowitsnumber.Togetit,wecalldate.isoweekday(),whichtellsuswhichdayoftheweekthatdateis(asanumber).Youfeedthatintothemappingand,boom!Youhavethenameoftheday.

get_durationisinterestingaswell.First,noticeittakesanentirerow,notjustasinglevalue.Whathappensinitsbodyisthatweperformasubtractionbetweenacampaignendandstartdates.Whenyousubtractdateobjectstheresultisatimedeltaobject,whichrepresentsagivenamountoftime.Wetakethevalueofits.daysproperty.Itisassimpleasthat.

Now,wecanintroducethefunpart,theapplicationofthosetwofunctions.

ThefirstapplicationisperformedonaSeriesobject,likewedidbeforefor'user'and'cmp_name',thereisnothingnewhere.

ThesecondoneisappliedtothewholeDataFrameand,inordertoinstructPandastoperformthatoperationontherows,wepassaxis=1.

Wecanverifytheresultsveryeasily,asshownhere:#26

df[['Start','End','Duration','DayofWeek']].head(3)

Yields:

StartEndDurationDayofWeek

02015-08-262017-03-05557Wednesday

12014-10-152014-12-1965Wednesday

22015-02-222016-01-14326Sunday

So,wenowknowthatbetweenthe26thofAugust2015andthe5thofMarch2017thereare557days,andthatthe26thofAugust2015wasaWednesday.

Ifyou’rewonderingwhatthepurposeofthisis,I’llprovideanexample.ImaginethatyouhaveacampaignthatistiedtoasportseventthatusuallytakesplaceonaSunday.Youmaywanttoinspectyourdataaccordingtothedayssothatyoucancorrelatethemtothevariousmeasurementsyouhave.We’renotgoingtodoitinthisproject,butitwasusefultosee,ifonlyforthedifferentwayofcallingapply()onaDataFrame.

CleaningeverythingupNowthatwehaveeverythingwewant,it’stimetodothefinalcleaning:rememberwestillhavethe'cmp_name'and'user'columns.Thoseareuselessnow,sotheyhavetogo.Also,IwanttoreorderthecolumnsintheDataFramesothatitismorerelevanttothedataitnowcontains.Inordertodothis,wejustneedtofilterdfonthecolumnlistwewant.We’llgetbackabrandnewDataFramethatwecanreassigntodfitself.#27

final_columns=[

'Type','Start','End','Duration','DayofWeek','Budget',

'Currency','Clicks','Impressions','Spent','CTR','CPC',

'CPI','TargetAge','TargetGender','Username','Email',

'Name','Gender','Age'

]

df=df[final_columns]

Ihavegroupedthecampaigninformationatthebeginning,thenthemeasurements,andfinallytheuserdataattheend.NowourDataFrameiscleanandreadyforustoinspect.

Beforewestartgoingcrazywithgraphs,whatabouttakingasnapshotofourDataFramesothatwecaneasilyreconstructitfromafile,ratherthanhavingtoredoallthestepswedidtogethere.Someanalystsmaywanttohaveitinspreadsheetform,todoadifferentkindofanalysisthantheonewewanttodo,solet’sseehowtosaveaDataFrametoafile.

It’seasierdonethansaid.

SavingtheDataFrametoafileWecansaveaDataFrameinmanydifferentways.Youcantypedf.to_andthenpressTabtomakeauto-completionpopup,toseeallthepossibleoptions.

We’regoingtosaveourDataFrameinthreedifferentformats,justforfun:comma-separatedvalues(CSV),JSON,andExcelspreadsheet.#28

df.to_csv('df.csv')

#29

df.to_json('df.json')

#30

df.to_excel('df.xls')

TheCSVfilelookslikethis(outputtruncated):

Type,Start,End,Duration,DayofWeek,Budget,Currency,Clicks,Impres

0,GRZ,2015-03-15,2015-11-10,240,Sunday,622551,GBP,35018,500002,787

1,AKX,2016-06-19,2016-09-19,92,Sunday,148219,EUR,45185,499997,6588

2,BYU,2014-09-25,2016-07-03,647,Thursday,537760,GBP,55771,500001,3

AndtheJSONonelikethis(again,outputtruncated):

{

"Type":{

"0":"GRZ",

"1":"AKX",

"2":"BYU",

So,it’sextremelyeasytosaveaDataFrameinmanydifferentformats,andthegoodnewsisthattheoppositeisalsotrue:it’sveryeasytoloadaspreadsheetintoaDataFrame.TheprogrammersbehindPandaswentalongwaytoeaseourtasks,somethingtobegratefulfor.

VisualizingtheresultsFinally,thejuicybits.Inthissection,we’regoingtovisualizesomeresults.Fromadatascienceperspective,I’mnotveryinterestedingoingdeepintoanalysis,especiallybecausethedataiscompletelyrandom,butnonetheless,thiscodewillgetyoustartedwithgraphsandotherfeatures.

SomethingIlearnedinmylife—andthismaycomeasasurprisetoyou—isthatlooksalsocountssoit’sveryimportantthatwhenyoupresentyourresults,youdoyourbesttomakethempretty.

Iwon’ttrytoprovetoyouhowtruthfulthatlaststatementis,butIreallydobelieveinit.Ifyourecallthelastlineofcell#1:

#makethegraphsnicer

pd.set_option('display.mpl_style','default')

Itspurposeistomakethegraphswewilllookatinthissectionalittlebitprettier.

Okay,so,firstofallwehavetoinstructthenotebookthatwewanttousematplotlibinline.ThismeansthatwhenweaskPandastoplotsomething,wewillhavetheresultrenderedinthecelloutputframe.Inordertodothis,wejustneedonesimpleinstruction:#31

%matplotlibinline

Youcanalsoinstructthenotebooktodothiswhenyoustartitfromtheconsolebypassingaparameter,butIwantedtoshowyouthiswaytoo,sinceitcanbeannoyingtohavetorestartthenotebookjustbecauseyouwanttoplotsomething.Inthisway,youcandoitontheflyandthenkeepworking.

Next,we’regoingtosetsomeparametersonpylab.Thisisforplottingpurposesanditwillremoveawarningthatafonthasn’tbeenfound.Isuggestthatyoudonotexecutethislineandkeepgoing.Ifyougetawarningthatafontismissing,comebacktothiscellandrunit.#32

importpylab

pylab.rcParams.update({'font.family':'serif'})

ThisbasicallytellsPylabtousethefirstavailableseriffont.Itissimplebuteffective,andyoucanexperimentwithotherfontstoo.

NowthattheDataFrameiscomplete,let’srundf.describe()(#33)again.Theresultsshouldlooksomethinglikethis:

Thiskindofquickresultisperfecttosatisfythosemanagerswhohave20secondstodedicatetoyouandjustwantroughnumbers.

NoteOnceagain,pleasekeepinmindthatourcampaignshavedifferentcurrencies,sothesenumbersareactuallymeaningless.ThepointhereistodemonstratetheDataFramecapabilities,nottogettoacorrectordetailedanalysisofrealdata.

Alternatively,agraphisusuallymuchbetterthanatablewithnumbersbecauseit’smucheasiertoreaditanditgivesyouimmediatefeedback.So,let’sgraphoutthefourpiecesofinformationwehaveoneachcampaign:budget,spent,clicks,andimpressions.#34

df[['Budget','Spent','Clicks','Impressions']].hist(

bins=16,figsize=(16,6));

Weextrapolatethosefourcolumns(thiswillgiveusanotherDataFramemadewithonlythosecolumns)andcallthehistogramhist()methodonit.Wegivesomemeasurementsonthebinsandfiguresizes,butbasicallyeverythingisdoneautomatically.

Oneimportantthing:sincethisinstructionistheonlyoneinthiscell(whichalsomeans,it’sthelastone),thenotebookwillprintitsresultbeforedrawingthegraph.Tosuppressthisbehaviorandhaveonlythegraphdrawnwithnoprinting,justaddasemicolonattheend(youthoughtIwasreminiscingaboutJava,didn’tyou?).Herearethegraphs:

Theyarebeautiful,aren’tthey?Didyounoticetheseriffont?Howaboutthemeaningofthosefigures?Ifyougobackto#6andtakealookatthewaywegeneratethedata,youwillseethatallthesegraphsmakeperfectsense.

Budgetissimplyarandomintegerinaninterval,thereforewewereexpectingauniformdistribution,andtherewehaveit;it’spracticallyaconstantline.

Spentisauniformdistributionaswell,butthehighendofitsintervalisthebudget,whichismoving,thismeansweshouldexpectsomethinglikeaquadratichyperbolethatdecreasestotheright.Andthereitisaswell.

Clickswasgeneratedwithatriangulardistributionwithmeanroughly20%oftheintervalsize,andyoucanseethatthepeakisrightthere,atabout20%totheleft.

Finally,ImpressionswasaGaussiandistribution,whichistheonethatassumesthefamousbellshape.Themeanwasexactlyinthemiddleandwehadstandarddeviationof2.Youcanseethatthegraphmatchesthoseparameters.

Good!Let’splotoutthemeasureswecalculated:#35

df[['CTR','CPC','CPI']].hist(

bins=20,figsize=(16,6));

Wecanseethatthecostperclickishighlyskewedtotheleft,meaningthatmostoftheCPCvaluesareverylow.Thecostperimpressionhasasimilarshape,butlessextreme.

Now,allthisisnice,butifyouwantedtoanalyzeonlyaparticularsegmentofthedata,howwouldyoudoit?WecanapplyamasktoaDataFrame,sothatwegetanotheronewithonlytherowsthatsatisfythemaskcondition.It’slikeapplyingaglobalrow-wiseifclause.#36

mask=(df.Spent>0.75*df.Budget)

df[mask][['Budget','Spent','Clicks','Impressions']].hist(

bins=15,figsize=(16,6),color='g');

Inthiscase,Ipreparedamasktofilteroutalltherowsforwhichthespentislessthanorequalto75%ofthebudget.Inotherwords,we’llincludeonlythosecampaignsforwhichwehavespentatleastthreequartersofthebudget.NoticethatinthemaskIamshowingyouanalternativewayofaskingforaDataFramecolumn,byusingdirectpropertyaccess(object.property_name),insteadofdict-likeaccess(object['property_name']).Ifproperty_nameisavalidPythonname,youcanusebothwaysinterchangeably(JavaScriptworkslikethisaswell).

Themaskisappliedinthesamewaythatweaccessadictwithakey.WhenyouapplyamasktoaDataFrame,yougetbackanotheroneandweselectonlytherelevantcolumnsonthis,andcallhist()again.Thistime,justforfun,wewanttheresultstobepaintedgreen:

Notethattheshapesofthegraphshaven’tchangedmuch,apartfromthespent,whichisquitedifferent.Thereasonforthisisthatwe’veaskedonlyfortherowswherespentisatleast75%ofthebudget.Thismeansthatwe’reincludingonlytherowswherespentisclosetothebudget.Thebudgetnumberscomefromauniformdistribution.Therefore,itisquiteobviousthatthespentisnowassumingthatkindofshape.Ifyoumaketheboundaryeventighter,andaskfor85%ormore,you’llseespentbecomemoreandmorelikebudget.

Let’snowaskforsomethingdifferent.Howaboutthemeasureofspent,click,andimpressionsgroupedbydayoftheweek?#37

df_weekday=df.groupby(['DayofWeek']).sum()

df_weekday[['Impressions','Spent','Clicks']].plot(

figsize=(16,6),subplots=True);

ThefirstlinecreatesanewDataFrame,df_weekday,byaskingforagroupingby'DayofWeek'ondf.Thefunctionusedtoaggregatethedataisaddition.

Thesecondlinegetsasliceofdf_weekdayusingalistofcolumnnames,somethingwe’reaccustomedtobynow.Ontheresultwecallplot(),whichisabitdifferenttohist().Theoptionsubplots=Truemakesplotdrawthreeindependentgraphs:

Interestinglyenough,wecanseethatmostoftheactionhappensonThursdays.Ifthisweremeaningfuldata,thiswouldpotentiallybeimportantinformationtogivetoourclients,andthisisthereasonI’mshowingyouthisexample.

Notethatthedaysaresortedalphabetically,whichscramblesthemupabit.Canyouthinkofaquicksolutionthatwouldfixtheissue?I’llleaveittoyouasanexercisetocomeupwithsomething.

Let’sfinishthispresentationsectionwithacouplemorethings.First,asimpleaggregation.Wewanttoaggregateon'TargetGender'and'TargetAge',andshow'Impressions'and'Spent'.Forboth,wewanttoseethemeanandthestandarddeviation.#38

agg_config={

'Impressions':{

'MeanImpr':'mean',

'StdImpr':'std',

},

'Spent':['mean','std'],

}

df.groupby(['TargetGender','TargetAge']).agg(agg_config)

It’sveryeasytodoit.Wewillprepareadictionarythatwe’lluseasaconfiguration.I’mshowingyoutwooptionstodoit.Weuseanicerformatfor'Impressions',wherewepassanesteddictwithdescription/functionaskey/valuepairs.Ontheotherhand,for'Spent',wejustuseasimplerlistwithjustthefunctionnames.

Then,weperformagroupingonthe'TargetGender'and'TargetAge'columns,andwepassourconfigurationdicttotheagg()method.Theresultistruncatedandrearrangedalittlebittomakeitfit,andshownhere:

ImpressionsSpent

MeanImprStdImprmeanstd

TargetTarget

GenderAge

B20-255000002.189102239882209442.168488

20-305000002.245317271285236854.155720

20-355000001.886396243725174268.898935

20-404999992.100786247740211540.133771

20-455000001.772811148712118603.932051

...............

M20-255000002.022023212520215857.323228

20-305000002.111882292577231663.713956

20-354999991.965177255651222790.960907

20-404999991.932473282515250023.393334

20-454999991.905746271077219901.462405

Thisisthetextualrepresentation,ofcourse,butyoucanalsohavetheHTMLone.YoucanseethatSpenthasthemeanandstdcolumnswhoselabelsaresimplythefunctionnames,whileImpressionsfeaturesthenicetitlesweaddedtotheconfigurationdict.

Let’sdoonemorethingbeforewewrapthischapterup.Iwanttoshowyousomethingcalledapivottable.It’skindofabuzzwordinthedataenvironment,soanexamplesuchasthisone,albeitverysimple,isamust.#39

pivot=df.pivot_table(

values=['Impressions','Clicks','Spent'],

index=['TargetAge'],

columns=['TargetGender'],

aggfunc=np.sum

)

pivot

Wecreateapivottablethatshowsusthecorrelationbetweenthetargetageandimpressions,clicks,andspent.Theselastthreewillbesubdividedaccordingtothetargetgender.Theaggregationfunctionusedtocalculatetheresultsisthenumpy.sumfunction(numpy.meanwouldbethedefault,hadInotspecifiedanything).

Aftercreatingthepivottable,wesimplyprintitwiththelastlineinthecell,andhere’sacropoftheresult:

It’sprettyclearandprovidesveryusefulinformationwhenthedataismeaningful.

That’sit!I’llleaveyoutodiscovermoreaboutthewonderfulworldofIPython,Jupyter,anddatascience.Istronglyencourageyoutogetcomfortablewiththenotebookenvironment.It’smuchbetterthanaconsole,it’sextremelypracticalandfuntouse,andyoucanevendoslidesanddocumentswithit.

Wheredowegofromhere?Datascienceisindeedafascinatingsubject.AsIsaidintheintroduction,thosewhowanttodelveintoitsmeandersneedtobewelltrainedinmathematicsandstatistics.Workingwithdatathathasbeeninterpolatedincorrectlyrendersanyresultaboutituseless.Thesamegoesfordatathathasbeenextrapolatedincorrectlyorsampledwiththewrongfrequency.Togiveyouanexample,imagineapopulationofindividualsthatarealignedinaqueue.If,forsomereason,thegenderofthatpopulationalternatedbetweenmaleandfemale,thequeuewouldbesomethinglikethis:F-M-F-M-F-M-F-M-F…

Ifyousampledittakingonlytheevenelements,youwoulddrawtheconclusionthatthepopulationwasmadeuponlyofmales,whilesamplingtheoddoneswouldtellyouexactlytheopposite.

Ofcourse,thiswasjustasillyexample,Iknow,butbelievemeit’sveryeasytomakemistakesinthisfield,especiallywhendealingwithbigdatawheresamplingismandatoryandtherefore,thequalityoftheintrospectionyoumakedepends,firstandforemost,onthequalityofthesamplingitself.

WhenitcomestodatascienceandPython,thesearethemaintoolsyouwanttolookat:

NumPy(http://www.numpy.org/):ThisisthefundamentalpackageforscientificcomputingwithPython.ItcontainsapowerfulN-dimensionalarrayobject,sophisticated(broadcasting)functions,toolsforintegratingC/C++andFortrancode,usefullinearalgebra,Fouriertransform,randomnumbercapabilities,andmuchmore.Scikit-Learn(http://scikit-learn.org/stable/):ThisisprobablythemostpopularmachinelearninglibraryinPython.Ithassimpleandefficienttoolsfordatamininganddataanalysis,accessibletoeverybody,andreusableinvariouscontexts.It’sbuiltonNumPy,SciPy,andMatplotlib.Pandas(http://pandas.pydata.org/):Thisisanopensource,BSD-licensedlibraryprovidinghigh-performance,easy-to-usedatastructures,anddataanalysistools.We’veuseditthroughoutthiswholechapter.IPython(http://ipython.org/)/Jupyter(http://jupyter.org/):Theseprovidearicharchitectureforinteractivecomputing.Matplotlib(http://matplotlib.org/):ThisisaPython2Dplottinglibrarythatproducespublication-qualityfiguresinavarietyofhardcopyformatsandinteractiveenvironmentsacrossplatforms.MatplotlibcanbeusedinPythonscripts,thePythonandIPythonshellandnotebook,webapplicationservers,andsixgraphicaluserinterfacetoolkits.Numba(http://numba.pydata.org/):ThisgivesyouthepowertospeedupyourapplicationswithhighperformancefunctionswrittendirectlyinPython.Withafewannotations,array-orientedandmath-heavyPythoncodecanbejust-in-timecompiledtonativemachineinstructions,similarinperformancetoC,C++,andFortran,withouthavingtoswitchlanguagesorPythoninterpreters.Bokeh(http://bokeh.pydata.org/en/latest/):It’saPython-interactivevisualization

librarythattargetsmodernwebbrowsersforpresentation.Itsgoalistoprovideelegant,conciseconstructionofnovelgraphicsinthestyleofD3.js,butalsodeliverthiscapabilitywithhigh-performanceinteractivityoververylargeorstreamingdatasets.

Otherthanthesesinglelibraries,youcanalsofindecosystemssuchasSciPy(http://scipy.org/)andAnaconda(https://www.continuum.io/),whichbundleseveraldifferentpackagesinordertogiveyousomethingthatjustworksinan“out-of-the-box”fashion.

Installingallthesetoolsandtheirseveraldependenciesishardonsomesystems,soIsuggestthatyoutryoutecosystemsaswellandseeifyouarecomfortablewiththem.Itmaybeworthit.

SummaryInthischapter,wetalkedaboutdatascience.Ratherthanattemptingtoexplainanythingaboutthisextremelywidesubject,wedelvedintoaproject.WefamiliarizedourselveswiththeJupyternotebook,andwithdifferentlibrariessuchasPandas,Matplotlib,NumPy.

Ofcourse,havingtocompressallthisinformationintoonesinglechaptermeansIcouldonlytouchbrieflyonthesubjectsIpresented.Ihopetheprojectwe’vegonethroughtogetherhasbeencomprehensiveenoughtogiveyouagoodideaaboutwhatcouldpotentiallybetheworkflowyoumightfollowwhenworkinginthisfield.

Thenextchapterisdedicatedtowebdevelopment.So,makesureyouhaveabrowserreadyandlet’sgo!

Chapter10.WebDevelopmentDoneRight “Don’tbelieveeverythingyoureadontheWeb.”

—Confucius

Inthischapter,we’regoingtoworkonawebsitetogether.Byworkingonasmallproject,myaimistoopenawindowforyoutotakeapeekonwhatwebdevelopmentis,alongwiththemainconceptsandtoolsyoushouldknowifyouwanttobesuccessfulwithit.

WhatistheWeb?TheWorldWideWeb,orsimplyWeb,isawayofaccessinginformationthroughtheuseofamediumcalledtheInternet.TheInternetisahugenetworkofnetworks,anetworkinginfrastructure.Itspurposeistoconnectbillionsofdevicestogether,allaroundtheglobe,sothattheycancommunicatewithoneanother.InformationtravelsthroughtheInternetinarichvarietyoflanguagescalledprotocols,whichallowdifferentdevicestospeakthesametongueinordertosharecontent.

TheWebisaninformation-sharingmodel,builtontopoftheInternet,whichemploystheHypertextTransferProtocol(HTTP)asabasisfordatacommunication.TheWeb,therefore,isjustoneoftheseveraldifferentwaysinformationcanbeexchangedovertheInternet:e-mail,instantmessaging,newsgroups,andsoon,theyallrelyondifferentprotocols.

HowdoestheWebwork?Inanutshell,HTTPisanasymmetricrequest-responseclient-serverprotocol.AnHTTPclientsendsarequestmessagetoanHTTPserver.Theserver,inturn,returnsaresponsemessage.Inotherwords,HTTPisapullprotocolinwhichtheclientpullsinformationfromtheserver(asopposedtoapushprotocolinwhichtheserverpushesinformationdowntotheclient).Takealookatthefollowingimage:

HTTPisbasedonTCP/IP(TransmissionControlProtocol/InternetProtocol),whichprovidesthetoolsforareliablecommunicationexchange.

AnimportantfeatureoftheHTTPprotocolisthatit’sstateless.Thismeansthatthecurrentrequesthasnoknowledgeaboutwhathappenedinpreviousrequests.Thisisalimitation,butyoucanbrowseawebsitewiththeillusionofbeingloggedin.Underthecoversthough,whathappensisthat,onlogin,atokenofuserinformationissaved(mostoftenontheclientside,inspecialfilescalledcookies)sothateachrequesttheusermakescarriesthemeansfortheservertorecognizetheuserandprovideacustominterfacebyshowingtheirname,keepingtheirbasketpopulated,andsoon.

Eventhoughit’sveryinteresting,we’renotgoingtodelveintotherichdetailsofHTTPandhowitworks.However,we’regoingtowriteasmallwebsite,whichmeanswe’llhavetowritethecodetohandleHTTPrequestsandreturnHTTPresponses.Iwon’tkeepprependingHTTPtothetermsrequestandresponsefromnowon,asItrusttherewon’tbeanyconfusion.

TheDjangowebframeworkForourproject,we’regoingtouseoneofthemostpopularwebframeworksyoucanfindinthePythonecosystem:Django.

Awebframeworkisasetoftools(libraries,functions,classes,andsoon)thatwecanusetocodeawebsite.Weneedtodecidewhatkindofrequestswewanttoallowtobeissuedagainstourwebserverandhowwerespondtothem.Awebframeworkistheperfecttooltodothatbecauseittakescareofmanythingsforussothatwecanconcentrateonlyontheimportantbitswithouthavingtoreinventthewheel.

NoteTherearedifferenttypesofframeworks.Notallofthemaredesignedforwritingcodefortheweb.Ingeneral,aframeworkisatoolthatprovidesfunctionalitiestofacilitatethedevelopmentofsoftwareapplications,productsandsolutions.

DjangodesignphilosophyDjangoisdesignedaccordingtothefollowingprinciples:

DRY:Asin,Don’tRepeatYourself.Don’trepeatcode,andcodeinawaythatmakestheframeworkdeduceasmuchaspossiblefromaslittleaspossible.Loosecoupling:Thevariouslayersoftheframeworkshouldn’tknowabouteachother(unlessabsolutelynecessaryforwhateverreason).Loosecouplingworksbestwhenparalleledwithhighcohesion.ToquoteRobertMartin:puttingtogetherthingswhichchangeforthesamereason,andspreadingapartthosewhichchangefordifferentreasons.Lesscode:Applicationsshouldusetheleastpossibleamountofcode,andbewritteninawaythatfavorsreuseasmuchaspossible.Consistency:WhenusingtheDjangoframework,regardlessofwhichlayeryou’recodingagainst,yourexperiencewillbeveryconsistentwiththedesignpatternsandparadigmsthatwerechosentolayouttheproject.

Theframeworkitselfisdesignedaroundthemodel-template-view(MTV)pattern,whichisavariantofmodel-view-controller(MVC),whichiswidelyemployedbyotherframeworks.Thepurposeofsuchpatternsistoseparateconcernsandpromotecodereuseandquality.

ThemodellayerOfthethreelayers,thisistheonethatdefinesthestructureofthedatathatishandledbytheapplication,anddealswithdatasources.Amodelisaclassthatrepresentsadatastructure.ThroughsomeDjangomagic,modelsaremappedtodatabasetablessothatyoucanstoreyourdatainarelationaldatabase.

NoteArelationaldatabasestoresdataintablesinwhicheachcolumnisapropertyofthedataandeachrowrepresentsasingleitemorentryinthecollectionrepresentedbythattable.Throughtheprimarykeyofeachtable,whichisthatpartofthedatathatallowstouniquelyidentifyeachitem,itispossibletoestablishrelationshipsbetweenitemsbelongingtodifferenttables,thatis,toputthemintorelation.

Thebeautyofthissystemisthatyoudon’thavetowritedatabase-specificcodeinordertohandleyourdata.Youwilljusthavetoconfigureyourmodelscorrectlyandsimplyusethem.TheworkonthedatabaseisdoneforyoubytheDjangoobject-relationalmapping(ORM),whichtakescareoftranslatingoperationsdoneonPythonobjectsintoalanguagethatarelationaldatabasecanunderstand:SQL(StructuredQueryLanguage).

OnebenefitofthisapproachisthatyouwillbeabletochangedatabaseswithoutrewritingyourcodesinceallthedatabasespecificcodeisproducedbyDjangoonthefly,accordingtowhichdatabaseit’sconnectedto.RelationaldatabasesspeakSQL,buteachofthemhasitsownuniqueflavorofit;therefore,nothavingtohardcodeanySQLinourapplicationisatremendousadvantage.

Djangoallowsyoutomodifyyourmodelsatanytime.Whenyoudo,youcanrunacommandthatcreatesamigration,whichisthesetofinstructionsneededtoportthedatabaseinastatethatrepresentsthecurrentdefinitionofyourmodels.

Tosummarize,thislayerdealswithdefiningthedatastructuresyouneedtohandleinyourwebsiteandgivesyouthemeanstosaveandloadthemfromandtothedatabasebysimplyaccessingthemodels,whicharePythonobjects.

TheviewlayerThefunctionofaviewishandlingarequest,performingwhateveractionneedstobecarriedout,andeventuallyreturningaresponse.Forexample,ifyouopenyourbrowserandrequestapagecorrespondingtoacategoryofproductsinane-commerceshop,theviewwilllikelytalktothedatabase,askingforallthecategoriesthatarechildrenoftheselectedcategory(forexample,todisplaytheminanavigationsidebar)andforalltheproductsthatbelongtotheselectedcategory,inordertodisplaythemonthepage.

Therefore,theviewisthemechanismthroughwhichwecanfulfillarequest.Itsresult,theresponseobject,canassumeseveraldifferentforms:aJSONpayload,text,anHTMLpage,andsoon.Whenyoucodeawebsite,yourresponsesusuallyconsistofHTMLorJSON.

NoteTheHypertextMarkupLanguage,orHTML,isthestandardmarkuplanguageusedtocreatewebpages.WebbrowsersrunenginesthatarecapableofinterpretingHTMLcodeandrenderitintowhatweseewhenweopenapageofawebsite.

ThetemplatelayerThisisthelayerthatprovidesthebridgebetweenbackendandfrontenddevelopment.WhenaviewhastoreturnHTML,itusuallydoesitbypreparingacontextobject(adict)withsomedata,andthenitfeedsthiscontexttoatemplate,whichisrendered(thatistosay,transformedintoHTML)andreturnedtothecallerintheformofaresponse(moreprecisely,thebodyoftheresponse).Thismechanismallowsformaximumcodereuse.Ifyougobacktothecategoryexample,it’seasytoseethat,ifyoubrowseawebsitethatsellsproducts,itdoesn’treallymatterwhichcategoryyouclickonorwhattypeofsearchyouperform,thelayoutoftheproductspagedoesn’tchange.Whatdoeschangeisthedatawithwhichthatpageispopulated.

Therefore,thelayoutofthepageisdefinedbyatemplate,whichiswritteninamixtureofHTMLandDjangotemplatelanguage.Theviewthatservesthatpagecollectsalltheproductstobedisplayedinthecontextdict,andfeedsittothetemplatewhichwillberenderedintoanHTMLpagebytheDjangotemplateengine.

TheDjangoURLdispatcherThewayDjangoassociatesaUniformResourceLocator(URL)withaviewisthroughmatchingtherequestedURLwiththepatternsthatareregisteredinaspecialfile.AURLrepresentsapageinawebsiteso,forexample,http://mysite.com/categories?id=123wouldprobablypointtothepageforthecategorywithID123onmywebsite,whilehttps://mysite.com/loginwouldprobablybetheuserloginpage.

TipThedifferencebetweenHTTPandHTTPSisthatthelatteraddsencryptiontotheprotocolsothatthedatathatyouexchangewiththewebsiteissecured.Whenyouputyourcreditcarddetailsonawebsite,orloginanywhere,ordoanythingaroundsensitivedata,youwanttomakesurethatyou’reusingHTTPS.

RegularexpressionsThewayDjangomatchesURLstopatternsisthrougharegularexpression.Aregularexpressionisasequenceofcharactersthatdefinesasearchpatternwithwhichwecancarryoutoperationssuchaspatternandstringmatching,find/replace,andsoon.

Regularexpressionshaveaspecialsyntaxtoindicatethingslikedigits,letters,spaces,andsoon,aswellashowmanytimesweexpectacharactertoappear,andmuchmore.Acompleteexplanationofthistopicisbeyondofthescopeofthebook.However,itisaveryimportanttopic,sotheprojectwe’regoingtoworkontogetherwillevolvearoundit,inthehopethatyouwillbestimulatedtofindthetimetoexploreitabitmoreonyourown.

Togiveyouaquickexample,imaginethatyouwantedtospecifyapatterntomatchadatesuchas"26-12-1947".Thisstringconsistsoftwodigits,onedash,twodigits,onedash,andfinallyfourdigits.Therefore,wecouldwriteitlikethis:r'[0-9]{2}-[0-9]{2}-[0-9]{4}'.Wecreatedaclassbyusingsquarebrackets,andwedefinedarangeofdigitsinside,from0to9,henceallthepossibledigits.Then,betweencurlybraces,wesaythatweexpecttwoofthem.Thenadash,thenwerepeatthispatternonceasitis,andoncemore,bychanginghowmanydigitsweexpect,andwithoutthefinaldash.Havingaclasslike[0-9]issuchacommonpatternthataspecialnotationhasbeencreatedasashortcut:'\d'.Therefore,wecanrewritethepatternlikethis:r'\d{2}-\d{2}-\d{4}'anditwillworkexactlythesame.Thatrinfrontofthestringstandsforraw,anditspurposeistoalterthewayeverybackslash'\'isinterpretedbytheregularexpressionengine.

AregexwebsiteSo,hereweare.We’llcodeawebsitethatstoresregularexpressionssothatwe’llbeabletoplaywiththemalittlebit.

NoteBeforeweproceedcreatingtheproject,I’dliketospendawordaboutCSS.CSS(CascadingStyleSheets)arefilesinwhichwespecifyhowthevariouselementsonanHTMLpagelook.Youcansetallsortsofpropertiessuchasshape,size,color,margins,borders,fonts,andsoon.Inthisproject,Ihavetriedmybesttoachieveadecentresultonthepages,butI’mneitherafrontenddevelopernoradesigner,sopleasedon’tpaytoomuchattentiontohowthingslook.Tryandfocusonhowtheywork.

SettingupDjangoOntheDjangowebsite(https://www.djangoproject.com/),youcanfollowthetutorial,whichgivesyouaprettygoodideaofDjango’scapabilities.Ifyouwant,youcanfollowthattutorialfirstandthencomebacktothisexample.So,firstthingsfirst;let’sinstallDjangoinyourvirtualenvironment:

$pipinstalldjango

Whenthiscommandisdone,youcantestitwithinaconsole(trydoingitwithbpython,itgivesyouashellsimilartoIPythonbutwithniceintrospectioncapabilities):

>>>importdjango

>>>django.VERSION

(1,8,4,'final',0)

NowthatDjangoisinstalled,we’regoodtogo.We’llhavetodosomescaffolding,soI’llquicklyguideyouthroughthat.

StartingtheprojectChooseafolderinthebook’senvironmentandchangeintothat.I’llusech10.Fromthere,westartaDjangoprojectwiththefollowingcommand:

$django-adminstartprojectregex

ThiswillpreparetheskeletonforaDjangoprojectcalledregex.Changeintotheregexfolderandrunthefollowing:

$pythonmanage.pyrunserver

Youshouldbeabletogotohttp://127.0.0.1:8000/withyourbrowserandseetheItworked!defaultDjangopage.Thismeansthattheprojectiscorrectlysetup.Whenyou’veseenthepage,killtheserverwithCtrl+C(orwhateveritsaysintheconsole).I’llpastethefinalstructurefortheprojectnowsothatyoucanuseitasareference:

$tree-Aregex#fromthech10folder

regex

├──db.sqlite3

├──entries

│├──admin.py

│├──forms.py

│├──__init__.py

│├──migrations

││├──0001_initial.py

││└──__init__.py

│├──models.py

│├──static

││└──entries

││└──css

││└──main.css

│├──templates

││└──entries

││├──base.html

││├──footer.html

││├──home.html

││├──insert.html

││└──list.html

│└──views.py

├──manage.py

└──regex

├──__init__.py

├──settings.py

├──urls.py

└──wsgi.py

Don’tworryifyou’remissingfiles,we’llgetthere.ADjangoprojectistypicallyacollectionofseveraldifferentapplications.Eachapplicationismeanttoprovideafunctionalityinaself-contained,reusablefashion.We’llcreatejustone,calledentries:

$pythonmanage.pystartappentries

Withintheentriesfolderthathasbeencreated,youcangetridofthetests.pymodule.

Now,let’sfixtheregex/settings.pyfileintheregexfolder.WeneedtoaddourapplicationtotheINSTALLED_APPStuplesothatwecanuseit(additatthebottomofthetuple):

INSTALLED_APPS=(

...djangoapps…

'entries',

)

Then,youmaywanttofixthelanguageandtimezoneaccordingtoyourpersonalpreference.IliveinLondon,soIsetthemlikethis:

LANGUAGE_CODE='en-gb'

TIME_ZONE='Europe/London'

Thereisnothingelsetodointhisfile,soyoucansaveandcloseit.

Nowit’stimetoapplythemigrationstothedatabase.Djangoneedsdatabasesupporttohandleusers,sessions,andthingslikethat,soweneedtocreateadatabaseandpopulateitwiththenecessarydata.Luckily,thisisveryeasilydonewiththefollowingcommand:

$pythonmanage.pymigrate

NoteForthisproject,weuseaSQLitedatabase,whichisbasicallyjustafile.Onarealproject,youwouldprobablyuseadifferentdatabaseenginelikeMySQLorPostgreSQL.

CreatingusersNowthatwehaveadatabase,wecancreateasuperuserusingtheconsole.

$pythonmanage.pycreatesuperuser

Afterenteringusernameandotherdetails,wehaveauserwithadminprivileges.ThisisenoughtoaccesstheDjangoadminsection,sotryandstarttheserver:

$pythonmanage.pyrunserver

ThiswillstarttheDjangodevelopmentserver,whichisaveryusefulbuilt-inwebserverthatyoucanusewhileworkingwithDjango.Nowthattheserverisrunning,wecanaccesstheadminpageathttp://localhost:8000/admin/.Iwillshowyouascreenshotofthissectionlater.IfyouloginwiththecredentialsoftheuseryoujustcreatedandheadtotheAuthenticationandAuthorizationsection,you’llfindUsers.Openthatandyouwillbeabletoseethelistofusers.Youcaneditthedetailsofanyuseryouwantasanadmin.Inourcase,makesureyoucreateadifferentonesothatthereareatleasttwousersinthesystem(we’llneedthemlater).I’llcallthefirstuserFabrizio(username:fab)andthesecondoneAdriano(username:adri)inhonorofmyfather.

Bytheway,youshouldseethattheDjangoadminpanelcomesforfreeautomatically.Youdefineyourmodels,hookthemup,andthat’sit.ThisisanincredibletoolthatshowshowadvancedDjango’sintrospectioncapabilitiesare.Moreover,itiscompletelycustomizableandextendable.It’strulyanexcellentpieceofwork.

AddingtheEntrymodelNowthattheboilerplateisoutoftheway,andwehaveacoupleofusers,we’rereadytocode.WestartbyaddingtheEntrymodeltoourapplicationsothatwecanstoreobjectsinthedatabase.Here’sthecodeyou’llneedtoadd(remembertousetheprojecttreeforreference):entries/models.py

fromdjango.dbimportmodels

fromdjango.contrib.auth.modelsimportUser

fromdjango.utilsimporttimezone

classEntry(models.Model):

user=models.ForeignKey(User)

pattern=models.CharField(max_length=255)

test_string=models.CharField(max_length=255)

date_added=models.DateTimeField(default=timezone.now)

classMeta:

verbose_name_plural='entries'

Thisisthemodelwe’llusetostoreregularexpressionsinoursystem.We’llstoreapattern,ateststring,areferencetotheuserwhocreatedtheentry,andthemomentofcreation.Youcanseethatcreatingamodelisactuallyquiteeasy,butnonetheless,let’sgothroughitlinebyline.

Firstweneedtoimportthemodelsmodulefromdjango.db.ThiswillgiveusthebaseclassforourEntrymodel.Djangomodelsarespecialclassesandmuchisdoneforusbehindthesceneswhenweinheritfrommodels.Model.

Wewantareferencetotheuserwhocreatedtheentry,soweneedtoimporttheUsermodelfromDjango’sauthorizationapplicationandwealsoneedtoimportthetimezonemodeltogetaccesstothetimezone.now()function,whichprovidesuswithatimezone-awareversionofdatetime.now().Thebeautyofthisisthatit’shookedupwiththeTIME_ZONEsettingsIshowedyoubefore.

Asfortheprimarykeyforthisclass,ifwedon’tsetoneexplicitly,Djangowilladdoneforus.AprimarykeyisakeythatallowsustouniquelyidentifyanEntryobjectinthedatabase(inthiscase,Djangowilladdanauto-incrementingintegerID).

So,wedefineourclass,andwesetupfourclassattributes.WehaveaForeignKeyattributethatisourreferencetotheUsermodel.WealsohavetwoCharFieldattributesthatholdthepatternandteststringsforourregularexpressions.WealsohaveaDateTimeField,whosedefaultvalueissettotimezone.now.Notethatwedon’tcalltimezone.nowrightthere,it’snow,notnow().So,we’renotpassingaDateTimeinstance(setatthemomentintimewhenthatlineisparsed)rather,we’repassingacallable,afunctionthatiscalledwhenwesaveanentryinthedatabase.ThisissimilartothecallbackmechanismweusedinChapter8,TheEdges–GUIsandScripts,whenwewereassigningcommandstobuttonclicks.

Thelasttwolinesareveryinteresting.WedefineaclassMetawithintheEntryclassitself.TheMetaclassisusedbyDjangotoprovideallsortsofextrainformationforamodel.DjangohasagreatdealoflogicunderthehoodtoadaptitsbehavioraccordingtotheinformationweputintheMetaclass.Inthiscase,intheadminpanel,thepluralizedversionofEntrywouldbeEntrys,whichiswrong,thereforeweneedtomanuallysetit.Wespecifythepluralalllowercase,asDjangotakescareofcapitalizingitforuswhenneeded.

Nowthatwehaveanewmodel,weneedtoupdatethedatabasetoreflectthenewstateofthecode.Inordertodothis,weneedtoinstructDjangothatitneedstocreatethecodetoupdatethedatabase.Thiscodeiscalledmigration.Let’screateitandexecuteit:

$pythonmanage.pymakemigrationsentries

$pythonmanage.pymigrate

Afterthesetwoinstructions,thedatabasewillbereadytostoreEntryobjects.

NoteTherearetwodifferentkindsofmigrations:dataandschemamigration.Datamigrationsportdatafromonestatetoanotherwithoutalteringitsstructure.Forexample,adatamigrationcouldsetallproductsforacategoryasoutofstockbyswitchingaflagtoFalseor0.Aschemamigrationisasetofinstructionsthatalterthestructureofthedatabaseschema.Forexample,thatcouldbeaddinganagecolumntoaPersontable,orincreasingthemaximumlengthofafieldtoaccountforverylongaddresses.WhendevelopingwithDjango,it’squitecommontohavetoperformbothkindsofmigrationsoverthecourseofdevelopment.Dataevolvescontinuously,especiallyifyoucodeinanagileenvironment.

CustomizingtheadminpanelThenextstepistohooktheEntrymodelupwiththeadminpanel.Youcandoitwithonelineofcode,butinthiscase,Iwanttoaddsomeoptionstocustomizeabitthewaytheadminpanelshowstheentries,bothinthelistviewofallentryitemsinthedatabaseandintheformviewthatallowsustocreateandmodifythem.

Allweneedtodoistoaddthefollowingcode:entries/admin.py

fromdjango.contribimportadmin

from.modelsimportEntry

@admin.register(Entry)

classEntryAdmin(admin.ModelAdmin):

fieldsets=[

('RegularExpression',

{'fields':['pattern','test_string']}),

('OtherInformation',

{'fields':['user','date_added']}),

]

list_display=('pattern','test_string','user')

list_filter=['user']

search_fields=['test_string']

Thisissimplybeautiful.Myguessisthatyouprobablyalreadyunderstandmostofit,evenifyou’renewtoDjango.

So,westartbyimportingtheadminmoduleandtheEntrymodel.Becausewewanttofostercodereuse,weimporttheEntrymodelusingarelativeimport(there’sadotbeforemodels).Thiswillallowustomoveorrenametheappwithouttoomuchtrouble.Then,wedefinetheEntryAdminclass,whichinheritsfromadmin.ModelAdmin.ThedecorationontheclasstellsDjangotodisplaytheEntrymodelintheadminpanel,andwhatweputintheEntryAdminclasstellsDjangohowtocustomizethewayithandlesthismodel.

Firstly,wespecifythefieldsetsforthecreate/editpage.Thiswilldividethepageintotwosectionssothatwegetabettervisualizationofthecontent(patternandteststring)andtheotherdetails(userandtimestamp)separately.

Then,wecustomizethewaythelistpagedisplaystheresults.Wewanttoseeallthefields,butnotthedate.Wealsowanttobeabletofilterontheusersothatwecanhavealistofalltheentriesbyjustoneuser,andwewanttobeabletosearchontest_string.

Iwillgoaheadandaddthreeentries,oneformyselfandtwoonbehalfofmyfather.Theresultisshowninthenexttwoimages.Afterinsertingthem,thelistpagelookslikethis:

IhavehighlightedthethreepartsofthisviewthatwecustomizedintheEntryAdminclass.Wecanfilterbyuser,wecansearchandwehaveallthefieldsdisplayed.Ifyouclickonapattern,theeditviewopensup.

Afterourcustomization,itlookslikethis:

Noticehowwehavetwosections:RegularExpressionandOtherInformation,thankstoourcustomEntryAdminclass.Haveagowithit,addsomeentriestoacoupleofdifferentusers,getfamiliarwiththeinterface.Isn’titnicetohaveallthisforfree?

CreatingtheformEverytimeyoufillinyourdetailsonawebpage,you’reinsertingdatainformfields.AformisapartoftheHTMLDocumentObjectModel(DOM)tree.InHTML,youcreateaformbyusingtheformtag.Whenyouclickonthesubmitbutton,yourbrowsernormallypackstheformdatatogetherandputsitinthebodyofaPOSTrequest.AsopposedtoGETrequests,whichareusedtoaskthewebserverforaresource,aPOSTrequestnormallysendsdatatothewebserverwiththeaimofcreatingorupdatingaresource.Forthisreason,handlingPOSTrequestsusuallyrequiresmorecarethanGETrequests.

WhentheserverreceivesdatafromaPOSTrequest,thatdataneedstobevalidated.Moreover,theserverneedstoemploysecuritymechanismstoprotectagainstvarioustypesofattacks.Oneattackthatisverydangerousisthecross-siterequestforgery(CSRF)attack,whichhappenswhendataissentfromadomainthatisnottheonetheuserisauthenticatedon.Djangoallowsyoutohandlethisissueinaveryelegantway.

So,insteadofbeinglazyandusingtheDjangoadmintocreatetheentries,I’mgoingtoshowyouhowtodoitusingaDjangoform.Byusingthetoolstheframeworkgivesyou,yougetaverygooddegreeofvalidationworkalreadydone(infact,wewon’tneedtoaddanycustomvalidationourselves).

TherearetwokindsofformclassesinDjango:FormandModelForm.Youusetheformertocreateaformwhoseshapeandbehaviordependsonhowyoucodetheclass,whatfieldsyouadd,andsoon.Ontheotherhand,thelatterisatypeofformthat,albeitstillcustomizable,infersfieldsandbehaviorfromamodel.SinceweneedaformfortheEntrymodel,we’llusethatone.entries/forms.py

fromdjango.formsimportModelForm

from.modelsimportEntry

classEntryForm(ModelForm):

classMeta:

model=Entry

fields=['pattern','test_string']

Amazinglyenough,thisisallwehavetodotohaveaformthatwecanputonapage.Theonlynotablethinghereisthatwerestrictthefieldstoonlypatternandtest_string.Onlylogged-inuserswillbeallowedaccesstotheinsertpage,andthereforewedon’tneedtoaskwhotheuseris:weknowthat.Asforthedate,whenwesaveanEntry,thedate_addedfieldwillbesetaccordingtoitsdefault,thereforewedon’tneedtospecifythataswell.We’llseeintheviewhowtofeedtheuserinformationtotheformbeforesaving.So,allthebackgroundworkisdone,allweneedistheviewsandthetemplates.Let’sstartwiththeviews.

WritingtheviewsWeneedtowritethreeviews.Weneedoneforthehomepage,onetodisplaythelistofallentriesforauser,andonetocreateanewentry.Wealsoneedviewstologinandlogout.ButthankstoDjango,wedon’tneedtowritethem.I’llpasteallthecode,andthenwe’llgothroughittogether,stepbystep.entries/views.py

importre

fromdjango.contrib.auth.decoratorsimportlogin_required

fromdjango.contrib.messages.viewsimportSuccessMessageMixin

fromdjango.core.urlresolversimportreverse_lazy

fromdjango.utils.decoratorsimportmethod_decorator

fromdjango.views.genericimportFormView,TemplateView

from.formsimportEntryForm

from.modelsimportEntry

classHomeView(TemplateView):

template_name='entries/home.html'

@method_decorator(

login_required(login_url=reverse_lazy('login')))

defget(self,request,*args,**kwargs):

context=self.get_context_data(**kwargs)

returnself.render_to_response(context)

classEntryListView(TemplateView):

template_name='entries/list.html'

@method_decorator(

login_required(login_url=reverse_lazy('login')))

defget(self,request,*args,**kwargs):

context=self.get_context_data(**kwargs)

entries=Entry.objects.filter(

user=request.user).order_by('-date_added')

matches=(self._parse_entry(entry)forentryinentries)

context['entries']=list(zip(entries,matches))

returnself.render_to_response(context)

def_parse_entry(self,entry):

match=re.search(entry.pattern,entry.test_string)

ifmatchisnotNone:

return(

match.group(),

match.groups()orNone,

match.groupdict()orNone

)

returnNone

classEntryFormView(SuccessMessageMixin,FormView):

template_name='entries/insert.html'

form_class=EntryForm

success_url=reverse_lazy('insert')

success_message="Entrywascreatedsuccessfully"

@method_decorator(

login_required(login_url=reverse_lazy('login')))

defget(self,request,*args,**kwargs):

returnsuper(EntryFormView,self).get(

request,*args,**kwargs)

@method_decorator(

login_required(login_url=reverse_lazy('login')))

defpost(self,request,*args,**kwargs):

returnsuper(EntryFormView,self).post(

request,*args,**kwargs)

defform_valid(self,form):

self._save_with_user(form)

returnsuper(EntryFormView,self).form_valid(form)

def_save_with_user(self,form):

self.object=form.save(commit=False)

self.object.user=self.request.user

self.object.save()

Let’sstartwiththeimports.Weneedtheremoduletohandleregularexpressions,thenweneedafewclassesandfunctionsfromDjango,andfinally,weneedtheEntrymodelandtheEntryFormform.

ThehomeviewThefirstviewisHomeView.ItinheritsfromTemplateView,whichmeansthattheresponsewillbecreatedbyrenderingatemplatewiththecontextwe’llcreateintheview.Allwehavetodoisspecifythetemplate_nameclassattributetopointtothecorrecttemplate.Djangopromotescodereusetoapointthatifwedidn’tneedtomakethisviewaccessibleonlytologged-inusers,thefirsttwolineswouldhavebeenallweneeded.

However,wewantthisviewtobeaccessibleonlytologged-inusers;therefore,weneedtodecorateitwithlogin_required.Now,historicallyviewsinDjangousedtobefunctions;therefore,thisdecoratorwasdesignedtoacceptafunctionnotamethodlikewehaveinthisclass.We’reusingDjangoclass-basedviewsinthisprojectso,inordertomakethingswork,weneedtotransformlogin_requiredsothatitacceptsamethod(thedifferencebeinginthefirstargument:self).Wedothisbypassinglogin_requiredtomethod_decorator.

Wealsoneedtofeedthelogin_requireddecoratorwithlogin_urlinformation,andherecomesanotherwonderfulfeatureofDjango.Asyou’llseeafterwe’redonewiththeviews,inDjango,youtieaviewtoaURLthroughapattern,consistingofaregularexpressionandotherinformation.Youcangiveanametoeachentryintheurls.pyfilesothatwhenyouwanttorefertoaURL,youdon’thavetohardcodeitsvalueintoyourcode.AllyouhavetodoisgetDjangotoreverse-engineerthatURLfromthenamewegavetotheentryinurls.pydefiningtheURLandtheviewthatistiedtoit.Thismechanismwillbecomeclearerlater.Fornow,justthinkofreverse('...')asawayofgettingaURLfromanidentifier.Inthisway,youonlywritetheactualURLonce,intheurls.pyfile,

whichisbrilliant.Intheviews.pycode,weneedtousereverse_lazy,whichworksexactlylikereversewithonemajordifference:itonlyfindstheURLwhenweactuallyneedit(inalazyfashion).Thisisneededwhentheurls.pyfilehasn’tbeenloadedyetwhenthereversefunctionisused.

Thegetmethod,whichwejustdecorated,simplycallsthegetmethodoftheparentclass.Ofcourse,thegetmethodisthemethodthatDjangocallswhenaGETrequestisperformedagainsttheURLtiedtothisview.

TheentrylistviewThisviewismuchmoreinterestingthanthepreviousone.Firstofall,wedecoratethegetmethodaswedidbefore.Insideofit,weneedtopreparealistofEntryobjectsandfeedittothetemplate,whichshowsittotheuser.Inordertodoso,westartbygettingthecontextdictlikewe’resupposedtodo,bycallingtheget_context_datamethodoftheTemplateViewclass.Then,weusetheORMtogetalistoftheentries.Wedothisbyaccessingtheobjectsmanager,andcallingafilteronit.Wefiltertheentriesaccordingtowhichuserisloggedin,andweaskforthemtobesortedinadescendingorder(that'-'infrontofthenamespecifiesthedescendingorder).TheobjectsmanageristhedefaultmanagereveryDjangomodelisaugmentedwithoncreation,itallowsustointeractwiththedatabasethroughitsmethods.

Weparseeachentrytogetalistofmatches(actually,Icodeditsothatmatchesisageneratorexpression).Finally,weaddtothecontextan'entries'keywhosevalueisthecouplingofentriesandmatches,sothateachEntryinstanceispairedwiththeresultingmatchofitspatternandteststring.

Onthelastline,wesimplyaskDjangotorenderthetemplateusingthecontextwecreated.

Takealookatthe_parse_entrymethod.Allitdoesisperformasearchontheentry.test_stringwiththeentry.pattern.IftheresultingmatchobjectisnotNone,itmeansthatwefoundsomething.Ifso,wereturnatuplewiththreeelements:theoverallgroup,thesubgroups,andthegroupdictionary.Ifyou’renotfamiliarwiththeseterms,don’tworry,you’llseeascreenshotsoonwithanexample.WereturnNoneifthereisnomatch.

TheformviewFinally,let’sexamineEntryFormView.Thisisparticularlyinterestingforafewreasons.Firstly,itshowsusaniceexampleofPython’smultipleinheritance.Wewanttodisplayamessageonthepage,afterhavinginsertedanEntry,soweinheritfromSuccessMessageMixin.Butwewanttohandleaformaswell,sowealsoinheritfromFormView.

NoteNotethat,whenyoudealwithmixinsandinheritance,youmayhavetoconsidertheorderinwhichyouspecifythebaseclassesintheclassdeclaration.

Inordertosetupthisviewcorrectly,weneedtospecifyafewattributesatthebeginning:thetemplatetoberendered,theformclasstobeusedtohandlethedatafromthePOSTrequest,theURLweneedtoredirecttheusertointhecaseofsuccess,andthesuccessmessage.

AnotherinterestingfeatureisthatthisviewneedstohandlebothGETandPOSTrequests.Whenwelandontheformpageforthefirsttime,theformisempty,andthatistheGETrequest.Ontheotherhand,whenwefillintheformandwanttosubmittheEntry,wemakeaPOSTrequest.YoucanseethatthebodyofgetisconceptuallyidenticaltoHomeView.Djangodoeseverythingforus.

Thepostmethodisjustlikeget.Theonlyreasonweneedtocodethesetwomethodsissothatwecandecoratethemtorequirelogin.

WithintheDjangoformhandlingprocess(intheFormViewclass),thereareafewmethodsthatwecanoverrideinordertocustomizetheoverallbehavior.Weneedtodoitwiththeform_validmethod.Thismethodwillbecalledwhentheformvalidationissuccessful.ItspurposeistosavetheformsothatanEntryobjectiscreatedoutofit,andthenstoredinthedatabase.

Theonlyproblemisthatourformismissingtheuser.Weneedtointerceptthatmomentinthechainofcallsandputtheuserinformationinourselves.Thisisdonebycallingthe_save_with_usermethod,whichisverysimple.

Firstly,weaskDjangotosavetheformwiththecommitargumentsettoFalse.ThiscreatesanEntryinstancewithoutattemptingtosaveittothedatabase.Savingitimmediatelywouldfailbecausetheuserinformationisnotthere.

ThenextlineupdatestheEntryinstance(self.object),addingtheuserinformationand,onthelastline,wecansafelysaveit.ThereasonIcalleditobjectandsetitontheinstancelikethatwastofollowwhattheoriginalFormViewclassdoes.

We’refiddlingwiththeDjangomechanismhere,soifwewantthewholethingtowork,weneedtopayattentiontowhenandhowwemodifyitsbehavior,andmakesurewedon’talteritincorrectly.Forthisreason,it’sveryimportanttoremembertocalltheform_validmethodofthebaseclass(weusesuperforthat)attheendofourowncustomizedversion,tomakesurethateveryotheractionthatmethodusuallyperformsiscarriedoutcorrectly.

Notehowtherequestistiedtoeachviewinstance(self.request)sothatwedon’tneedtopassitthroughwhenwerefactorourlogicintomethods.NotealsothattheuserinformationhasbeenaddedtotherequestautomaticallybyDjango.Finally,notethatthereasonwhyalltheprocessissplitintoverysmallmethodsliketheseissothatwecanonlyoverridethosethatweneedtocustomize.Allthisremovestheneedtowritealotofcode.

Nowthatwehavetheviewscovered,let’sseehowwecouplethemtotheURLs.

TyingupURLsandviewsIntheurls.pymodule,wetieeachviewtoaURL.Therearemanywaysofdoingthis.Ichosethesimplestone,whichworksperfectlyfortheextentofthisexercise,butyoumaywanttoexplorethisargumentmoredeeplyifyouintendtoworkwithDjango.Thisisthecorearoundwhichthewholewebsitelogicwillrevolve;therefore,youshouldtrytogetitdowncorrectly.Notethattheurls.pymodulebelongstotheprojectfolder.regex/urls.py

fromdjango.conf.urlsimportinclude,url

fromdjango.contribimportadmin

fromdjango.contrib.authimportviewsasauth_views

fromdjango.core.urlresolversimportreverse_lazy

fromentries.viewsimportHomeView,EntryListView,EntryFormView

urlpatterns=[

url(r'^admin/',include(admin.site.urls)),

url(r'^entries/$',EntryListView.as_view(),name='entries'),

url(r'^entries/insert$',

EntryFormView.as_view(),

name='insert'),

url(r'^login/$',

auth_views.login,

kwargs={'template_name':'admin/login.html'},

name='login'),

url(r'^logout/$',

auth_views.logout,

kwargs={'next_page':reverse_lazy('home')},

name='logout'),

url(r'^$',HomeView.as_view(),name='home'),

]

Asyoucansee,themagiccomesfromtheurlfunction.Firstly,wepassitaregularexpression;thentheview;andfinally,aname,whichiswhatwewilluseinthereverseandreverse_lazyfunctionstorecovertheURL.

Notethat,whenusingclass-basedviews,wehavetotransformthemintofunctions,whichiswhaturlisexpecting.Todothat,wecalltheas_view()methodonthem.

Notealsothatthefirsturlentry,fortheadmin,isspecial.InsteadofspecifyingaURLandaview,itspecifiesaURLprefixandanotherurls.pymodule(fromtheadmin.sitepackage).Inthisway,DjangowillcompletealltheURLsfortheadminsectionbyprepending'admin/'toalltheURLsspecifiedinadmin.site.urls.Wecouldhavedonethesameforourentriesapp(andweshouldhave),butIfeelitwouldhavebeenabittoomuchforthissimpleproject.

Intheregularexpressionlanguage,the'^'and'$'symbolsrepresentthestartandendofastring.Notethatifyouusetheinclusiontechnique,asfortheadmin,the'$'ismissing.Ofcourse,thisisbecause'admin/'isjustaprefix,whichneedstobecompletedbyallthedefinitionsintheincludedurlsmodule.

Somethingelseworthnoticingisthatwecanalsoincludethestringifiedversionofapathtoaview,whichwedofortheloginandlogoutviews.Wealsoaddinformationaboutwhichtemplatestousewiththekwargsargument.Theseviewscomestraightfromthedjango.contrib.authpackage,bytheway,sothatwedon’tneedtowriteasinglelineofcodetohandleauthentication.Thisisbrilliantandsavesusalotoftime.

Eachurldeclarationmustbedonewithintheurlpatternslistandonthismatter,it’simportanttoconsiderthat,whenDjangoistryingtofindaviewforaURLthathasbeenrequested,thepatternsareexercisedinorder,fromtoptobottom.Thefirstonethatmatchesistheonethatwillprovidetheviewforitso,ingeneral,youhavetoputspecificpatternsbeforegenericones,otherwisetheywillnevergetachancetobecaught.Forexample,'^shop/categories/$'needstocomebefore'^shop'(notetheabsenceofthe'$'inthelatter),otherwiseitwouldneverbecalled.OurexamplefortheentriesworksfinebecauseIthoroughlyspecifiedURLsusingthe'$'attheend.

So,models,forms,admin,viewsandURLsarealldone.Allthatislefttodoistakecareofthetemplates.I’llhavetobeverybriefonthispartbecauseHTMLcanbeveryverbose.

WritingthetemplatesAlltemplatesinheritfromabaseone,whichprovidestheHTMLstructureforallothers,inaveryOOPtypeoffashion.Italsospecifiesafewblocks,whichareareasthatcanbeoverriddenbychildrensothattheycanprovidecustomcontentforthoseareas.Let’sstartwiththebasetemplate:entries/templates/entries/base.html

{%loadstaticfromstaticfiles%}

<!DOCTYPEhtml>

<htmllang="en">

<head>

{%blockmeta%}

<metacharset="utf-8">

<metaname="viewport"

content="width=device-width,initial-scale=1.0">

{%endblockmeta%}

{%blockstyles%}

<linkhref="{%static"entries/css/main.css"%}"

rel="stylesheet">

{%endblockstyles%}

<title>{%blocktitle%}Title{%endblocktitle%}</title>

</head>

<body>

<divid="page-content">

{%blockpage-content%}

{%endblockpage-content%}

</div>

<divid="footer">

{%blockfooter%}

{%endblockfooter%}

</div>

</body>

</html>

Thereisagoodreasontorepeattheentriesfolderfromthetemplatesone.WhenyoudeployaDjangowebsite,youcollectallthetemplatefilesunderonefolder.Ifyoudon’tspecifythepathslikeIdid,youmaygetabase.htmltemplateintheentriesapp,andabase.htmltemplateinanotherapp.Thelastonetobecollectedwilloverrideanyotherfilewiththesamename.Forthisreason,byputtingtheminatemplates/entriesfolderandusingthistechniqueforeachDjangoappyouwrite,youavoidtheriskofnamecollisions(thesamegoesforanyotherstaticfile).

Thereisnotmuchtosayaboutthistemplate,really,apartfromthefactthatitloadsthestatictagsothatwecangeteasyaccesstothestaticpathwithouthardcodingitinthetemplatebyusing{%static…%}.Thecodeinthespecial{%...%}sectionsiscodethatdefineslogic.Thecodeinthespecial{{...}}representsvariablesthatwillberenderedonthepage.

Wedefinethreeblocks:title,page-content,andfooter,whosepurposeistoholdthetitle,thecontentofthepage,andthefooter.Blockscanbeoptionallyoverriddenbychildtemplatesinordertoprovidedifferentcontentwithinthem.

Here’sthefooter:entries/templates/entries/footer.html

<divclass="footer">

Goback<ahref="{%url"home"%}">home</a>.

</div>

Itgivesusanicelinktothehomepage.

Thehomepagetemplateisthefollowing:entries/templates/entries/home.html

{%extends"entries/base.html"%}

{%blocktitle%}WelcometotheEntrywebsite.{%endblocktitle%}

{%blockpage-content%}

<h1>Welcome{{user.first_name}}!</h1>

<divclass="home-option">Toseethelistofyourentries

pleaseclick<ahref="{%url"entries"%}">here.</a>

</div>

<divclass="home-option">Toinsertanewentrypleaseclick

<ahref="{%url"insert"%}">here.</a>

</div>

<divclass="home-option">Tologinasanotheruserpleaseclick

<ahref="{%url"logout"%}">here.</a>

</div>

<divclass="home-option">Togototheadminpanel

pleaseclick<ahref="{%url"admin:index"%}">here.</a>

</div>

{%endblockpage-content%}

Itextendsthebase.htmltemplate,andoverridestitleandpage-content.Youcanseethatbasicallyallitdoesisprovidefourlinkstotheuser.Thesearethelistofentries,theinsertpage,thelogoutpage,andtheadminpage.AllofthisisdonewithouthardcodingasingleURL,throughtheuseofthe{%url…%}tag,whichisthetemplateequivalentofthereversefunction.

ThetemplateforinsertinganEntryisasfollows:entries/templates/entries/insert.html

{%extends"entries/base.html"%}

{%blocktitle%}InsertanewEntry{%endblocktitle%}

{%blockpage-content%}

{%ifmessages%}

{%formessageinmessages%}

<pclass="{{message.tags}}">{{message}}</p>

{%endfor%}

{%endif%}

<h1>InsertanewEntry</h1>

<formaction="{%url"insert"%}"method="post">

{%csrf_token%}{{form.as_p}}

<inputtype="submit"value="Insert">

</form><br>

{%endblockpage-content%}

{%blockfooter%}

<div><ahref="{%url"entries"%}">Seeyourentries.</a></div>

{%include"entries/footer.html"%}

{%endblockfooter%}

Thereissomeconditionallogicatthebeginningtodisplaymessages,ifany,andthenwedefinetheform.Djangogivesustheabilitytorenderaformbysimplycalling{{form.as_p}}(alternatively,form.as_ulorform.as_table).Thiscreatesallthenecessaryfieldsandlabelsforus.Thedifferencebetweenthethreecommandsisinthewaytheformislaidout:asaparagraph,asanunorderedlistorasatable.Weonlyneedtowrapitinformtagsandaddasubmitbutton.Thisbehaviorwasdesignedforourconvenience;weneedthefreedomtoshapethat<form>tagaswewant,soDjangoisn’tintrusiveonthat.Also,notethat{%csrf_token%}.ItwillberenderedintoatokenbyDjangoandwillbecomepartofthedatasenttotheserveronsubmission.ThiswayDjangowillbeabletoverifythattherequestwasfromanallowedsource,thusavoidingtheaforementionedcross-siterequestforgeryissue.DidyouseehowwehandledthetokenwhenwewrotetheviewfortheEntryinsertion?Exactly.Wedidn’twriteasinglelineofcodeforit.Djangotakescareofitautomaticallythankstoamiddlewareclass(CsrfViewMiddleware).PleaserefertotheofficialDjangodocumentationtoexplorethissubjectfurther.

Forthispage,wealsousethefooterblocktodisplayalinktothehomepage.Finally,wehavethelisttemplate,whichisthemostinterestingone.entries/templates/entries/list.html

{%extends"entries/base.html"%}

{%blocktitle%}Entrieslist{%endblocktitle%}

{%blockpage-content%}

{%ifentries%}

<h1>Yourentries({{entries|length}}found)</h1>

<div><ahref="{%url"insert"%}">Insertnewentry.</a></div>

<tableclass="entries-table">

<thead>

<tr><th>Entry</th><th>Matches</th></tr>

</thead>

<tbody>

{%forentry,matchinentries%}

<trclass="entries-list{%cycle'light-gray''white'%}">

<td>

Pattern:<codeclass="code">

"{{entry.pattern}}"</code><br>

TestString:<codeclass="code">

"{{entry.test_string}}"</code><br>

Added:{{entry.date_added}}

</td>

<td>

{%ifmatch%}

Group:{{match.0}}<br>

Subgroups:

{{match.1|default_if_none:"none"}}<br>

GroupDict:{{match.2|default_if_none:"none"}}

{%else%}

Nomatchesfound.

{%endif%}

</td>

</tr>

{%endfor%}

</tbody>

</table>

{%else%}

<h1>Youhavenoentries</h1>

<div><ahref="{%url"insert"%}">Insertnewentry.</a></div>

{%endif%}

{%endblockpage-content%}

{%blockfooter%}

{%include"entries/footer.html"%}

{%endblockfooter%}

Itmaytakeyouawhiletogetusedtothetemplatelanguage,butreally,allthereistoitisthecreationofatableusingaforloop.Westartbycheckingifthereareanyentriesand,ifso,wecreateatable.Therearetwocolumns,onefortheEntry,andtheotherforthematch.

IntheEntrycolumn,wedisplaytheEntryobject(apartfromtheuser)andintheMatchescolumn,wedisplaythat3-tuplewecreatedintheEntryListView.Notethattoaccesstheattributesofanobject,weusethesamedotsyntaxweuseinPython,forexample{{entry.pattern}}or{{entry.test_string}},andsoon.

Whendealingwithlistsandtuples,wecannotaccessitemsusingthesquarebracketssyntax,soweusethedotoneaswell({{match.0}}isequivalenttomatch[0],andsoon.).Wealsouseafilter,throughthepipe(|)operatortodisplayacustomvalueifamatchisNone.

TheDjangotemplatelanguage(whichisnotproperlyPython)iskeptsimpleforaprecisereason.Ifyoufindyourselflimitedbythelanguage,itmeansyou’reprobablytryingtodosomethinginthetemplatethatshouldactuallybedoneintheview,wherethatlogicismorerelevant.

Allowmetoshowyouacoupleofscreenshotsofthelistandinserttemplates.Thisiswhatthelistofentrieslookslikeformyfather:

Notehowtheuseofthecycletagalternatesthebackgroundcoloroftherowsfromwhitetolightgray.Thoseclassesaredefinedinthemain.cssfile.

TheEntryinsertionpageissmartenoughtoprovideafewdifferentscenarios.Whenyoulandonitatfirst,itpresentsyouwithjustanemptyform.Ifyoufillitincorrectly,itwilldisplayanicemessageforyou(seethefollowingpicture).However,ifyoufailtofillinbothfields,itwilldisplayanerrormessagebeforethem,alertingyouthatthosefieldsarerequired.

Notealsothecustomfooter,whichincludesbothalinktotheentrieslistandalinktothehomepage:

Andthat’sit!YoucanplayaroundwiththeCSSstylesifyouwish.Downloadthecodeforthebookandhavefunexploringandextendingthisproject.Addsomethingelsetothemodel,createandapplyamigration,playwiththetemplates,there’slotstodo!

Djangoisaverypowerfulframework,andofferssomuchmorethanwhatI’vebeenabletoshowyouinthischapter,soyoudefinitelywanttocheckitout.Thebeautyofitisthatit’sPython,soreadingitssourcecodeisaveryusefulexercise.

ThefutureofwebdevelopmentComputerscienceisaveryyoungsubject,comparedtootherbranchesofsciencethathaveexistedalongsidehumankindforcenturiesormore.Oneofitsmaincharacteristicsisthatitmovesextremelyfast.Itleapsforwardwithsuchspeedthat,injustafewyears,youcanseechangesthatarecomparabletorealworldchangesthattookacenturytohappen.Therefore,asacoder,youmustpayattentiontowhathappensinthisworld,allthetime.

Somethingthatishappeningnowisthatbecausepowerfulcomputersarenowquitecheapandalmosteveryonehasaccesstothem,thetrendistotryandavoidputtingtoomuchworkloadonthebackend,andletthefrontendhandlepartofit.Therefore,inthelastfewyears,JavaScriptframeworksandlibrarieslikejQueryandBackbonehavebecomeverypopularandwebdevelopmenthasshiftedfromaparadigmwherethebackendtakescareofhandlingdata,preparingit,andservingittothefrontendtodisplayit,toaparadigmwherethebackendissometimesjustusedasanAPI,asheerdataprovider.ThefrontendfetchesthedatafromthebackendwithanAPIcall,andthenittakescareoftherest.ThisshiftfacilitatestheexistenceofparadigmslikeSingle-PageApplication(SPA),where,ideally,thewholepageisloadedonceandthenevolves,basedonthecontentthatusuallycomesfromthebackend.E-commercewebsitesthatloadtheresultsofasearchinapagethatdoesn’trefreshthesurroundingstructure,aremadewithsimilartechniques.Browserscanperformasynchronouscalls(AJAX)thatcanreturndatawhichcanberead,manipulatedandinjectedbackintothepagewithJavaScriptcode.

So,ifyou’replanningtoworkonwebdevelopment,IstronglysuggestyoutogetacquaintedwithJavaScript(ifyou’renotalready),andalsowithAPIs.Inthelastfewpagesofthischapter,I’llgiveyouanexampleofhowtomakeasimpleAPIusingtwodifferentPythonmicroframeworks:FlaskandFalcon.

WritingaFlaskviewFlask(http://flask.pocoo.org/)isaPythonmicroframework.ItprovidesfewerfeaturesthanDjango,butit’ssupposedlyfasterandquickertogetupandrunning.Tobehonest,gettingDjangoupandrunningnowadaysisalsoveryquicklydone,butFlaskissopopularthatit’sgoodtoseeanexampleofit,nonetheless.

Inyourch10folder,createaflaskfolderwiththefollowingstructure:

$tree-Aflask#fromthech10folder

flask

├──main.py

└──templates

└──main.html

Basically,we’regoingtocodetwosimplefiles:aFlaskapplicationandanHTMLtemplate.FlaskusesJinja2astemplateengine.It’sextremelypopularandveryfast,andjustrecentlyevenDjangohasstartedtooffernativesupportforit,whichissomethingthatPythoncodershavelongedfor,foralongtime.flask/templates/main.html

<!doctypehtml>

<title>HellofromFlask</title>

<h1>

{%ifname%}

Hello{{name}}!

{%else%}

Helloshyperson!

{%endif%}

</h1>

Thetemplateisalmostoffensivelysimple;allitdoesistochangethegreetingaccordingtothepresenceofthenamevariable.AbitmoreinterestingistheFlaskapplicationthatrendersit:flask/main.py

fromflaskimportFlask,render_template

app=Flask(__name__)

@app.route('/')

@app.route('/<name>')

defhello(name=None):

returnrender_template('main.html',name=name)

if__name__=='__main__':

app.run()

Wecreateanappobject,whichisaFlaskapplication.Weonlyfeedthefully-qualifiednameofthemodule,whichisstoredin__name__.

Then,wewriteasimplehelloview,whichtakesanoptionalnameargument.Inthebodyoftheview,wesimplyrenderthemain.htmltemplate,passingtoitthenameargument,

regardlessofitsvalue.

What’sinterestingistherouting.DifferentlyfromDjango’swayoftyingupviewsandURLs(theurls.pymodule),inFlaskyoudecorateyourviewwithoneormore@app.routedecorators.Inthiscase,weacceptboththerootURLwithoutanythingelse,orwithnameinformation.

Changeintotheflaskfolderandtype(makesureyouhaveFlaskinstalledwith$pipinstallflask):

$pythonmain.py

Youcanopenabrowserandgotohttp://127.0.0.1:5000/.ThisURLhasnonameinformation;therefore,youwillseeHelloshyperson!Itiswrittenallniceandbig.TrytoaddsomethingtothatURLlikehttp://127.0.0.1:5000/Adriano.HitEnterandthepagewillchangetoHelloAdriano!.

Ofcourse,Flaskoffersyoumuchmorethanthisbutwedon’thavetheroomtoseeamorecomplexexample.It’sdefinitelyworthexploring,though.Severalprojectsuseitsuccessfullyandit’sfunanditisnicetocreatewebsitesorAPIswithit.Flask’sauthor,ArminRonacher,isasuccessfulandveryprolificcoder.HealsocreatedorcollaboratedonseveralotherinterestingprojectslikeWerkzeug,Jinja2,Click,andSphinx.

BuildingaJSONquoteserverinFalconFalcon(http://falconframework.org/)isanothermicroframeworkwritteninPython,whichwasdesignedtobelight,fastandflexible.Ithinkthisrelativelyyoungprojectwillevolvetobecomesomethingreallypopularduetoitsspeed,whichisimpressive,soI’mhappytoshowyouatinyexampleusingit.

We’regoingtobuildaviewthatreturnsarandomlychosenquotefromtheBuddha.

Inyourch10folder,createanewonecalledfalcon.We’llhavetwofiles:quotes.pyandmain.py.Torunthisexample,installFalconandGunicorn($pipinstallfalcongunicorn).Falconistheframework,andGunicorn(GreenUnicorn)isaPythonWSGIHTTPServerforUnix(which,inlaymanterms,meansthetechnologythatisusedtoruntheserver).Whenyou’reallsetup,startbycreatingthequotes.pyfile.falcon/quotes.py

quotes=[

"Thousandsofcandlescanbelightedfromasinglecandle,"

"andthelifeofthecandlewillnotbeshortened."

"Happinessneverdecreasesbybeingshared.",

...

"Peacecomesfromwithin.Donotseekitwithout.",

]

Youwillfindthecompletelistofquotesinthesourcecodeforthisbook.Ifyoudon’thaveit,youcanalsofillinyourfavoritequotes.Notethatnoteverylinehasacommaattheend.InPython,it’spossibletoconcatenatestringslikethat,aslongastheyareinbrackets(orbraces).It’scalledimplicitconcatenation.

Thecodeforthemainappisnotlong,butitisinteresting:falcon/main.py

importjson

importrandom

importfalcon

fromquotesimportquotes

classQuoteResource:

defon_get(self,req,resp):

quote={

'quote':random.choice(quotes),

'author':'TheBuddha'

}

resp.body=json.dumps(quote)

api=falcon.API()

api.add_route('/quote',QuoteResource())

Let’sstartwiththeclass.InDjangowehadagetmethod,inFlaskwedefinedafunction,andherewewriteanon_getmethod,anamingstylethatremindsmeofC#eventhandlers.Ittakesarequestandaresponseargument,bothautomaticallyfedbytheframework.Initsbody,wedefineadictwitharandomlychosenquote,andtheauthor

information.ThenwedumpthatdicttoaJSONstringandsettheresponsebodytoitsvalue.Wedon’tneedtoreturnanything,Falconwilltakecareofitforus.

Attheendofthefile,wecreatetheFalconapplication,andwecalladd_routeonittotiethehandlerwehavejustwrittentotheURLwewant.

Whenyou’reallsetup,changetothefalconfolderandtype:

$gunicornmain:api

Then,makearequest(orsimplyopenthepagewithyourbrowser)tohttp://127.0.0.1:8000/quote.WhenIdidit,IgotthisJSONinresponse:

{

quote:"Themindiseverything.Whatyouthinkyoubecome.",

author:"TheBuddha"

}

Whatevertheframeworkyouendupusingforyourwebdevelopment,tryandkeepyourselfinformedaboutotherchoicestoo.Sometimesyoumaybeinsituationswhereadifferentframeworkistherightwaytogo,andhavingaworkingknowledgeofdifferenttoolswillgiveyouanadvantage.

SummaryInthischapter,wecaughtaglimpseofwebdevelopment.WetalkedaboutimportantconceptsliketheDRYphilosophyandtheconceptofaframeworkasatoolthatprovidesuswithmanythingsweneedinordertowritecodetoserverequests.WealsotalkedabouttheMTVpattern,andhownicelythesethreelayersplaytogethertorealizearequest-responsepath.

Lateron,webrieflyintroducedregularexpressions,whichisasubjectofparamountimportance,andit’sthelayerwhichprovidesthetoolsforURLrouting.

Therearemanydifferentframeworksoutthere,andDjangoisdefinitelyoneofthebestandmostwidelyused,soit’sdefinitelyworthexploring,especiallyitssourcecode,whichisverywellwritten.

Thereareotherveryinterestingandimportantframeworkstoo,likeFlask.Theyprovidefewerfeaturesbut,ingeneral,theyarefaster,bothinexecutiontimeandtosetup.OnethatisextremelyfastistherelativelyyoungFalconproject,whosebenchmarksareoutstanding.

It’simportanttogetasolidunderstandingofhowtherequest-responsemechanismworks,andhowtheWebingeneralworks,sothateventuallyitwon’tmattertoomuchwhichframeworkyouhavetouse.Youwillbeabletopickitupquicklybecauseitwillonlybeamatterofgettingfamiliarwithawayofdoingsomethingyoualreadyknowalotabout.

Exploreatleastthreeframeworksandtrytocomeupwithdifferentusecasestodecidewhichoneofthemcouldbetheidealchoice.Whenyouareabletomakethatchoice,youwillknowyouhaveagoodenoughunderstandingofthem.

Thenextchapterisaboutdebuggingandtroubleshooting.We’lllearnhowtodealwitherrorsandissuessothatifyougetintroublewhencoding(don’tworry,normallyitonlyhappensaboutallthetime),youwillbeabletoquicklyfindthesolutiontoyourproblemandmoveon.

Chapter11.DebuggingandTroubleshooting “Ifdebuggingistheprocessofremovingsoftwarebugs,thenprogrammingmustbetheprocessofputtingthemin.”

—EdsgerW.Dijkstra

Inthelifeofaprofessionalcoder,debuggingandtroubleshootingtakeupasignificantamountoftime.Evenifyouworkonthemostbeautifulcodebaseeverwrittenbyman,therewillstillbebugsinit,thatisguaranteed.

Wespendanawfullotoftimereadingotherpeople’scodeand,inmyopinion,agoodsoftwaredeveloperissomeonewhokeepstheirattentionhigh,evenwhenthey’rereadingcodethatisnotreportedtobewrongorbuggy.

Beingabletodebugcodeefficientlyandquicklyisaskillthatanycoderneedstokeepimproving.Somethinkthatbecausetheyhavereadthemanual,they’refine,buttherealityis,thenumberofvariablesinthegameissobigthatthereisnomanual.Thereareguidelinesthatonecanfollow,butthereisnomagicbookthatwillteachyoueverythingyouneedtoknowinordertobecomegoodatthis.

Ifeelthatonthisparticularsubject,Ihavelearnedthemostfrommycolleagues.Itamazesmetoobservesomeoneveryskilledattackingaproblem.Ienjoyseeingthestepstheytake,thethingstheyverifytoexcludepossiblecauses,andthewaytheyconsiderthesuspectsthateventuallyleadthemtothesolutiontotheproblem.

Everycolleagueweworkwithcanteachussomething,orsurpriseuswithafantasticguessthatturnsouttobetherightone.Whenthathappens,don’tjustremaininwonderment(orworse,inenvy),butseizethemomentandaskthemhowtheygottothatguessandwhy.Theanswerwillallowyoutoseeifthereissomethingyoucanstudyindeeplateronsothat,maybenexttime,you’llbetheonewhowillcatchthebug.

Somebugsareveryeasytospot.Theycomeoutofcoarsemistakesand,onceyouseetheeffectsofthosemistakes,it’seasytofindasolutionthatfixestheproblem.

Butthereareotherbugswhicharemuchmoresubtle,muchmoreslippery,andrequiretrueexpertise,andagreatdealofcreativityandout-of-the-boxthinking,tobedealtwith.

Theworstofall,atleastforme,arethenondeterministicones.Thesesometimeshappen,andsometimesdon’t.SomehappenonlyinenvironmentAbutnotinenvironmentB,eventhoughAandBaresupposedtobeexactlythesame.Thosebugsarethetrueevilones,andtheycandriveyoucrazy.

Andofcourse,bugsdon’tjusthappeninthesandbox,right?Withyourbosstellingyou“don’tworry!takeyourtimetofixthis,havelunchfirst!“.Nope.TheyhappenonaFridayathalfpastfive,whenyourbrainiscookedandyoujustwanttogohome.It’sinthosemoments,wheneveryoneisgettingupsetinasplitsecond,whenyourbossisbreathingonyourneck,thatyouhavetobeabletokeepcalm.AndIdomeanit.That’sthemost

importantskilltohaveifyouwanttobeabletofightbugseffectively.Ifyouallowyourmindtogetstressed,saygoodbyetocreativethinking,tologicdeduction,andtoeverythingyouneedatthatmoment.Sotakeadeepbreath,sitproperly,andfocus.

Inthischapter,Iwilltrytodemonstratesomeusefultechniquesthatyoucanemployaccordingtotheseverityofthebug,andafewsuggestionsthatwillhopefullyboostyourweaponsagainstbugsandissues.

DebuggingtechniquesInthispart,I’llpresentyouwiththemostcommontechniques,theonesIusemostoften,however,pleasedon’tconsiderthislisttobeexhaustive.

DebuggingwithprintThisisprobablytheeasiesttechniqueofall.It’snotveryeffective,itcannotbeusedeverywhereanditrequiresaccesstoboththesourcecodeandaterminalthatwillrunit(andthereforeshowtheresultsoftheprintfunctioncalls).

However,inmanysituations,thisisstillaquickandusefulwaytodebug.Forexample,ifyouaredevelopingaDjangowebsiteandwhathappensinapageisnotwhatwouldyouexpect,youcanfilltheviewwithprintsandkeepaneyeontheconsolewhileyoureloadthepage.I’veprobablydoneitamilliontimes.

Whenyouscattercallstoprintinyourcode,younormallyendupinasituationwhereyouduplicatealotofdebuggingcode,eitherbecauseyou’reprintingatimestamp(likewedidwhenweweremeasuringhowfastlistcomprehensionsandgeneratorswere),orbecauseyouhavetosomehowbuildastringofsomesortthatyouwanttodisplay.

Anotherissueisthatit’sextremelyeasytoforgetcallstoprintinyourcode.

So,forthesereasons,ratherthanusingabarecalltoprint,Isometimesprefertocodeacustomfunction.Let’sseehow.

DebuggingwithacustomfunctionHavingacustomfunctioninasnippetthatyoucanquicklygrabandpasteintothecode,andthenusetodebug,canbeveryuseful.Ifyou’refast,youcanalwayscodeoneonthefly.Theimportantthingistocodeitinawaythatitwon’tleavestuffaroundwhenyoueventuallyremovethecallsanditsdefinition,thereforeit’simportanttocodeitinawaythatiscompletelyself-contained.Anothergoodreasonforthisrequirementisthatitwillavoidpotentialnameclasheswiththerestofthecode.

Let’sseeanexampleofsuchafunction.custom.py

defdebug(*msg,print_separator=True):

print(*msg)

ifprint_separator:

print('-'*40)

debug('Datais…')

debug('Different','Strings','Arenotaproblem')

debug('Afterwhileloop',print_separator=False)

Inthiscase,Iamusingakeyword-onlyargumenttobeabletoprintaseparator,whichisalineof40dashes.

Thefunctionisverysimple,Ijustredirectwhateverisinmsgtoacalltoprintand,ifprint_separatorisTrue,Iprintalineseparator.Runningthecodewillshow:

$pythoncustom.py

Datais…

----------------------------------------

DifferentStringsArenotaproblem

----------------------------------------

Afterwhileloop

Asyoucansee,thereisnoseparatorafterthelastline.

Thisisjustoneeasywaytosomehowaugmentasimplecalltotheprintfunction.Let’sseehowwecancalculateatimedifferencebetweencalls,usingoneofPython’strickyfeaturestoouradvantage.custom_timestamp.py

fromtimeimportsleep

defdebug(*msg,timestamp=[None]):

print(*msg)

fromtimeimporttime#localimport

iftimestamp[0]isNone:

timestamp[0]=time()#1

else:

now=time()

print('Timeelapsed:{:.3f}s'.format(

now-timestamp[0]))

timestamp[0]=now#2

debug('Enteringnastypieceofcode…')

sleep(.3)

debug('Firststepdone.')

sleep(.5)

debug('Secondstepdone.')

Thisisabittrickier,butstillquitesimple.Firstnoticeweimportthetimefunctionfromthetimemodulefromthedebugfunction.Thisallowsustoavoidhavingtoaddthatimportoutsideofthefunction,andmaybeforgetitthere.

TakealookathowIdefinedtimestamp.It’salist,ofcourse,butwhat’simportanthereisthatitisamutableobject.ThismeansthatitwillbesetupwhenPythonparsesthefunctionanditwillretainitsvaluethroughoutdifferentcalls.Therefore,ifweputatimestampinitaftereachcall,wecankeeptrackoftimewithouthavingtouseanexternalglobalvariable.Iborrowedthistrickfrommystudiesonclosures,atechniquethatIencourageyoutoreadaboutbecauseit’sveryinteresting.

Right,so,afterhavingprintedwhatevermessagewehadtoprintandimportingtime,wetheninspectthecontentoftheonlyitemintimestamp.IfitisNone,wehavenopreviousreference,thereforewesetthevaluetothecurrenttime(#1).

Ontheotherhand,ifwehaveapreviousreference,wecancalculateadifference(whichwenicelyformattothreedecimaldigits)andthenwefinallyputthecurrenttimeagainintimestamp(#2).It’sanicetrick,isn’tit?

Runningthiscodeshowsthisresult:

$pythoncustom_timestamp.py

Enteringnastypieceofcode…

Firststepdone.

Timeelapsed:0.300s

Secondstepdone.

Timeelapsed:0.501s

Whateverisyoursituation,havingaselfcontainedfunctionlikethiscanbeveryuseful.

InspectingthetracebackWebrieflytalkedaboutthetracebackinChapter7,Testing,Profiling,andDealingwithExceptionswhenwesawseveraldifferentkindsofexceptions.Thetracebackgivesyouinformationaboutwhathappenedinyourapplicationthatwentwrong.Yougetagreathelpfromreadingit.Let’sseeaverysmallexample:traceback_simple.py

d={'some':'key'}

key='some-other'

print(d[key])

Wehaveadictandwehavetriedtoaccessakeywhichisn’tinit.YoushouldrememberthatthiswillraiseaKeyErrorexception.Let’srunthecode:

$pythontraceback_simple.py

Traceback(mostrecentcalllast):

File"traceback_simple.py",line3,in<module>

print(d[key])

KeyError:'some-other'

Youcanseethatwegetalltheinformationweneed:themodulename,thelinethatcausedtheerror(boththenumberandtheinstruction),andtheerroritself.Withthisinformation,youcangobacktothesourcecodeandtryandunderstandwhat’sgoingwrong.

Let’snowcreateamoreinterestingexamplethatbuildsonthis,andexercisesafeaturethatisonlyavailableinPython3.Imaginethatwe’revalidatingadict,workingonmandatoryfields,thereforeweexpectthemtobethere.Ifnot,weneedtoraiseacustomValidationError,thatwewilltrapfurtherupstreamintheprocessthatrunsthevalidator(whichisnotshownhere,itcouldbeanything,really).Itshouldbesomethinglikethis:traceback_validator.py

classValidatorError(Exception):

"""RaisedwhenaccessingadictresultsinKeyError."""

d={'some':'key'}

mandatory_key='some-other'

try:

print(d[mandatory_key])

exceptKeyError:

raiseValidatorError(

'`{}`notfoundind.'.format(mandatory_key))

Wedefineacustomexceptionthatisraisedwhenthemandatorykeyisn’tthere.Notethatitsbodyconsistsofitsdocumentationstringsowedon’tneedtoaddanyotherstatements.

Verysimply,wedefineadummydictandtrytoaccessitusingmandatory_key.WetraptheKeyErrorandraiseValidatorErrorwhenthathappens.ThepurposeofdoingthisisthatwemayalsowanttoraiseValidatorErrorinothercircumstances,notnecessarilyasaconsequenceofamandatorykeybeingmissing.Thistechniqueallowsustorunthevalidationinasimpletry/exceptthatonlycaresaboutValidatorError.

Thethingis,inPython2,thiscodewouldjustdisplaythelastexception(ValidatorError),whichmeanswewouldlosetheinformationabouttheKeyErrorthatprecedesit.InPython3,thisbehaviorhaschangedandexceptionsarenowchainedsothatyouhaveamuchbetterinformationreportwhensomethinghappens.Thecodeproducesthisresult:

$pythontraceback_validator.py

Traceback(mostrecentcalllast):

File"traceback_validator.py",line7,in<module>

print(d[mandatory_key])

KeyError:'some-other'

Duringhandlingoftheaboveexception,anotherexceptionoccurred:

Traceback(mostrecentcalllast):

File"traceback_validator.py",line10,in<module>

'`{}`notfoundind.'.format(mandatory_key))

__main__.ValidatorError:`some-other`notfoundind.

Thisisbrilliant,becausewecanseethetracebackoftheexceptionthatledustoraiseValidationError,aswellasthetracebackfortheValidationErroritself.

Ihadanicediscussionwithoneofmyreviewersaboutthetracebackyougetfromthepipinstaller.HewashavingtroublesettingeverythingupinordertoreviewthecodeforChapter9,DataScience.HisfreshUbuntuinstallationwasmissingafewlibrariesthatwereneededbythepippackagesinordertoruncorrectly.

Thereasonhewasblockedwasthathewastryingtofixtheerrorsdisplayedinthetracebackstartingfromthetopone.Isuggestedthathestartedfromthebottomoneinstead,andfixthat.Thereasonwasthat,iftheinstallerhadgottentothatlastline,Iguessthatbeforethat,whatevererrormayhaveoccurred,itwasstillpossibletorecoverfromit.Onlyafterthelastline,pipdecideditwasn’tpossibletocontinueanyfurther,andthereforeIstartedfixingthatone.Oncethelibrariesrequiredtofixthaterrorhadbeeninstalled,everythingelsewentsmoothly.

Readingatracebackcanbetricky,andmyfriendwaslackingthenecessaryexperiencetoaddressthisproblemcorrectly,therefore,ifyouendupinthesamesituation,don’tbediscouraged,andtrytoshakethingsupabit,don’ttakeanythingforgranted.

Pythonhasahugeandwonderfulcommunityandit’sveryunlikelythat,whenyouencounteraproblem,you’rethefirstonetoseeit,soopenabrowserandsearch.Bydoingso,yoursearchingskillswillalsoimprovebecauseyouwillhavetotrimtheerrordowntotheminimumbutessentialsetofdetailsthatwillmakeyoursearcheffective.

Ifyouwanttoplayandunderstandthetracebackabitbetter,inthestandardlibrarythereisamodulecalled,surprisesurprise,tracebackthatyoucanuse.Itprovidesastandardinterfacetoextract,format,andprintstacktracesofPythonprograms,mimickingexactlythebehaviorofthePythoninterpreterwhenitprintsastacktrace.

UsingthePythondebuggerAnotherveryeffectivewayofdebuggingPythonistousethePythondebugger:pdb.IfyouareaddictedtotheIPythonconsole,likeme,youshoulddefinitelycheckouttheipdblibrary.ipdbaugmentsthestandardpdbinterfacelikeIPythondoeswiththePythonconsole.

Thereareseveraldifferentwaysofusingthisdebugger(whicheverversion,itisnotimportant),butthemostcommononeconsistsofsimplysettingabreakpointandrunningthecode.WhenPythonreachesthebreakpoint,executionissuspendedandyougetconsoleaccesstothatpointsothatyoucaninspectallthenames,andsoon.Youcanalsoalterdataontheflytochangetheflowoftheprogram.

Asatoyexample,let’spretendwehaveaparserthatisraisingaKeyErrorbecauseakeyismissinginadict.ThedictisfromaJSONpayloadthatwecannotcontrol,andwejustwant,forthetimebeing,tocheatandpassthatcontrol,sincewe’reinterestedinwhatcomesafterwards.Let’sseehowwecouldinterceptthismoment,inspectthedata,fixitandgettothebottom,withipdb.ipdebugger.py

#dcomesfromaJSONpayloadwedon'tcontrol

d={'first':'v1','second':'v2','fourth':'v4'}

#keysalsocomesfromaJSONpayloadwedon'tcontrol

keys=('first','second','third','fourth')

defdo_something_with_value(value):

print(value)

forkeyinkeys:

do_something_with_value(d[key])

print('Validationdone.')

Asyoucansee,thiscodewillbreakwhenkeygetsthevalue'third',whichismissinginthedict.Remember,we’repretendingthatbothdandkeyscomedynamicallyfromaJSONpayloadwedon’tcontrol,soweneedtoinspecttheminordertofixdandpasstheforloop.Ifwerunthecodeasitis,wegetthefollowing:

$pythonipdebugger.py

v1

v2

Traceback(mostrecentcalllast):

File"ipdebugger.py",line10,in<module>

do_something_with_value(d[key])

KeyError:'third'

Soweseethatthatkeyismissingfromthedict,butsinceeverytimewerunthiscodewemaygetadifferentdictorkeystuple,thisinformationdoesn’treallyhelpus.Let’sinjectacalltoipdb.ipdebugger_ipdb.py

#dcomesfromaJSONpayloadwedon'tcontrol

d={'first':'v1','second':'v2','fourth':'v4'}

#keysalsocomesfromaJSONpayloadwedon'tcontrol

keys=('first','second','third','fourth')

defdo_something_with_value(value):

print(value)

importipdb

ipdb.set_trace()#weplaceabreakpointhere

forkeyinkeys:

do_something_with_value(d[key])

print('Validationdone.')

Ifwenowrunthiscode,thingsgetinteresting(notethatyouroutputmayvaryalittleandthatallthecommentsinthisoutputwereaddedbyme):

$pythonipdebugger_ipdb.py

>/home/fab/srv/l.p/ch11/ipdebugger_ipdb.py(12)<module>()

11

--->12forkeyinkeys:#thisiswherethebreakpointcomes

13do_something_with_value(d[key])

ipdb>keys#let'sinspectthekeystuple

('first','second','third','fourth')

ipdb>!d.keys()#nowthekeysofd

dict_keys(['first','fourth','second'])#wemiss'third'

ipdb>!d['third']='somethingdarkside…'#let'sputitin

ipdb>c#...andcontinue

v1

v2

somethingdarkside…

v4

Validationdone.

Thisisveryinteresting.First,notethat,whenyoureachabreakpoint,you’reservedaconsolethattellsyouwhereyouare(thePythonmodule)andwhichlineisthenextonetobeexecuted.Youcan,atthispoint,performabunchofexploratoryactions,suchasinspectingthecodebeforeandafterthenextline,printingastacktrace,interactingwiththeobjects,andsoon.PleaseconsulttheofficialPythondocumentationonpdbtolearnmoreaboutthis.Inourcase,wefirstinspectthekeystuple.Afterthat,weinspectthekeysofd.

HaveyounoticedthatexclamationmarkIprependedtod?It’sneededbecausedisacommandinthepdbinterfacethatmovestheframe(d)own.

NoteIindicatecommandswithintheipdbshellwiththisnotation:eachcommandisactivatedbyoneletter,whichtypicallyisthefirstletterofthecommandname.So,dfordown,nfornext,andsforstepbecome,moreconcisely,(d)own,(n)extand(s)tep.

Iguessthisisagoodenoughreasontohavebetternames,right?Indeed,butIneededto

showyouthis,soIchosetoused.Inordertotellpdbthatwe’renotyieldinga(d)owncommand,weput“!”infrontofdandwe’refine.

Afterseeingthekeysofd,weseethat'third'ismissing,soweputitinourselves(couldthisbedangerous?thinkaboutit).Finally,nowthatallthekeysarein,wetypec,whichmeans(c)ontinue.

pdbalsogivesyoutheabilitytoproceedwithyourcodeonelineatatimeusing(n)ext,to(s)tepintoafunctionfordeeperanalysis,orhandlingbreakswith(b)reak.Foracompletelistofcommands,pleaserefertothedocumentationortype(h)elpintheconsole.

Youcanseefromtheoutputthatwecouldfinallygettotheendofthevalidation.

pdb(oripdb)areinvaluabletoolsthatIuseeveryday,Icouldn’tlivewithoutthem.So,goandhavefun,setabreakpointsomewhereandtryandinspect,followtheofficialdocumentationandtrythecommandsinyourcodetoseetheireffectandlearnthemwell.

InspectinglogfilesAnotherwayofdebuggingamisbehavingapplicationistoinspectitslogfiles.Logfilesarespecialfilesinwhichanapplicationwritesdownallsortsofthings,normallyrelatedtowhat’sgoingoninsideofit.Ifanimportantprocedureisstarted,Iwouldtypicallyexpectalineforthatinthelogs.Itisthesamewhenitfinishes,andpossiblyforwhathappensinsideofit.

Errorsneedtobeloggedsothatwhenaproblemhappenswecaninspectwhatwentwrongbytakingalookattheinformationinthelogfiles.

TherearemanydifferentwaystosetupaloggerinPython.Loggingisverymalleableandyoucanconfigureit.Inanutshell,therearenormallyfourplayersinthegame:loggers,handlers,filters,andformatters:

LoggersexposetheinterfacethattheapplicationcodeusesdirectlyHandlerssendthelogrecords(createdbyloggers)totheappropriatedestinationFiltersprovideafinergrainedfacilityfordeterminingwhichlogrecordstooutputFormattersspecifythelayoutofthelogrecordsinthefinaloutput

LoggingisperformedbycallingmethodsoninstancesoftheLoggerclass.Eachlineyouloghasalevel.Thelevelsnormallyusedare:DEBUG,INFO,WARNING,ERROR,andCRITICAL.Youcanimportthemfromtheloggingmodule.Theyareinorderofseverityandit’sveryimportanttousethemproperlybecausetheywillhelpyoufilterthecontentsofalogfilebasedonwhatyou’researchingfor.Logfilesusuallybecomeextremelybigsoit’sveryimportanttohavetheinformationinthemwrittenproperlysothatyoucanfinditquicklywhenitmatters.

Youcanlogtoafilebutyoucanalsologtoanetworklocation,toaqueue,toaconsole,andsoon.Ingeneral,ifyouhaveanarchitecturethatisdeployedononemachine,loggingtoafileisacceptable,butwhenyourarchitecturespansovermultiplemachines(suchasinthecaseofservice-orientedarchitectures),it’sveryusefultoimplementacentralizedsolutionforloggingsothatalllogmessagescomingfromeachservicecanbestoredandinvestigatedinasingleplace.Ithelpsalot,otherwiseyoucanreallygocrazytryingtocorrelategiantfilesfromseveraldifferentsourcestofigureoutwhatwentwrong.

NoteAservice-orientedarchitecture(SOA)isanarchitecturalpatterninsoftwaredesigninwhichapplicationcomponentsprovideservicestoothercomponentsviaacommunicationsprotocol,typicallyoveranetwork.Thebeautyofthissystemisthat,whencodedproperly,eachservicecanbewritteninthemostappropriatelanguagetoserveitspurpose.Theonlythingthatmattersisthecommunicationwiththeotherservices,whichneedstohappenviaacommonformatsothatdataexchangecanbedone.

Here,Iwillpresentyouwithaverysimpleloggingexample.Wewilllogafewmessagestoafile:log.py

importlogging

logging.basicConfig(

filename='ch11.log',

level=logging.DEBUG,#minimumlevelcaptureinthefile

format='[%(asctime)s]%(levelname)s:%(message)s',

datefmt='%m/%d/%Y%I:%M:%S%p')

mylist=[1,2,3]

logging.info('Startingtoprocess`mylist`...')

forpositioninrange(4):

try:

logging.debug('Valueatposition{}is{}'.format(

position,mylist[position]))

exceptIndexError:

logging.exception('Faultyposition:{}'.format(position))

logging.info('Doneparsing`mylist`.')

Let’sgothroughitlinebyline.First,weimporttheloggingmodule,thenwesetupabasicconfiguration.Ingeneral,aproductionloggingconfigurationismuchmorecomplicatedthanthis,butIwantedtokeepthingsaseasyaspossible.Wespecifyafilename,theminimumlogginglevelwewanttocaptureinthefile,andthemessageformat.We’lllogthedateandtimeinformation,thelevel,andthemessage.

Iwillstartbylogginganinfomessagethattellsmewe’reabouttoprocessourlist.Then,Iwilllog(thistimeusingtheDEBUGlevel,byusingthedebugfunction)whichisthevalueatsomeposition.I’musingdebugherebecauseIwanttobeabletofilterouttheselogsinthefuture(bysettingtheminimumleveltologging.INFOormore),becauseImighthavetohandleverybiglistsandIdon’twanttologallthevalues.

IfwegetanIndexError(andwedo,sinceI’mloopingoverrange(4)),wecalllogging.exception(),whichisthesameaslogging.error(),butitalsoprintsthetraceback.

Attheendofthecode,Iloganotherinfomessagesayingwe’redone.Theresultisthis:

[10/08/201504:17:06PM]INFO:Startingtoprocess`mylist`...

[10/08/201504:17:06PM]DEBUG:Valueatposition0is1

[10/08/201504:17:06PM]DEBUG:Valueatposition1is2

[10/08/201504:17:06PM]DEBUG:Valueatposition2is3

[10/08/201504:17:06PM]ERROR:Faultyposition:3

Traceback(mostrecentcalllast):

File"log.py",line15,in<module>

position,mylist[position]))

IndexError:listindexoutofrange

[10/08/201504:17:06PM]INFO:Doneparsing`mylist`.

Thisisexactlywhatweneedtobeabletodebuganapplicationthatisrunningonabox,andnotonourconsole.Wecanseewhatwenton,thetracebackofanyexceptionraised,andsoon.

Note

Theexamplepresentedhereonlyscratchesthesurfaceoflogging.Foramorein-depthexplanation,youcanfindaveryniceintroductioninthehowto(https://docs.python.org/3.4/howto/logging.html)sectionoftheofficialPythondocumentation.

Loggingisanart,youneedtofindagoodbalancebetweenloggingeverythingandloggingnothing.Ideally,youshouldloganythingthatyouneedtomakesureyourapplicationisworkingcorrectly,andpossiblyallerrorsorexceptions.

OthertechniquesInthisfinalsection,I’dliketodemonstratebrieflyacoupleoftechniquesthatyoumayfinduseful.

ProfilingWetalkedaboutprofilinginChapter7,Testing,Profiling,andDealingwithExceptions,andI’monlymentioningitherebecauseprofilingcansometimesexplainweirderrorsthatareduetoacomponentbeingtooslow.Especiallywhennetworkingisinvolved,havinganideaofthetimingsandlatenciesyourapplicationhastogothroughisveryimportantinordertounderstandwhatmaybegoingonwhenproblemsarise,thereforeIsuggestyougetacquaintedwithprofilingtechniquesalsoforatroubleshootingperspective.

AssertionsAssertionsareanicewaytomakeyourcodeensureyourassumptionsareverified.Iftheyare,allproceedsregularlybut,iftheyarenot,yougetaniceexceptionthatyoucanworkwith.Sometimes,insteadofinspecting,it’squickertodropacoupleofassertionsinthecodejusttoexcludepossibilities.Let’sseeanexample:assertions.py

mylist=[1,2,3]#thisideallycomesfromsomeplace

assert4==len(mylist)#thiswillbreak

forpositioninrange(4):

print(mylist[position])

Thiscodesimulatesasituationinwhichmylistisn’tdefinedbyuslikethat,ofcourse,butwe’reassumingithasfourelements.Soweputanassertionthere,andtheresultisthis:

$pythonassertions.py

Traceback(mostrecentcalllast):

File"assertions.py",line3,in<module>

assert4==len(mylist)

AssertionError

Thistellsusexactlywheretheproblemis.

WheretofindinformationInthePythonofficialdocumentation,thereisasectiondedicatedtodebuggingandprofiling,whereyoucanreadupaboutthebdbdebuggerframework,andaboutmodulessuchasfaulthandler,timeit,trace,tracemallock,andofcoursepdb.Justheadtothestandardlibrarysectioninthedocumentationandyou’llfindallthisinformationveryeasily.

TroubleshootingguidelinesInthisshortsection,I’llliketogiveyouafewtipsthatcomefrommytroubleshootingexperience.

UsingconsoleeditorsFirst,getcomfortableusingvimornanoasaneditor,andlearnthebasicsoftheconsole.Whenthingsbreakbadyoudon’thavetheluxuryofyoureditorwithallthebellsandwhistlesthere.Youhavetoconnecttoaboxandworkfromthere.Soit’saverygoodideatobecomfortablebrowsingyourproductionenvironmentwithconsolecommands,andbeabletoeditfilesusingconsole-basededitorssuchasvi,vim,ornano.Don’tletyourusualdevelopmentenvironmentspoilyou,becauseyou’llhavetopayapriceifyoudo.

WheretoinspectMysecondsuggestionisonwheretoplaceyourdebuggingbreakpoints.Itdoesn’tmatterifyouareusingprint,acustomfunction,oripdb,youstillhavetochoosewheretoplacethecallsthatprovideyouwiththeinformation,right?

Well,someplacesarebetterthanothers,andtherearewaystohandlethedebuggingprogressionthatarebetterthanothers.

Inormallyavoidplacingabreakpointinanifclausebecause,ifthatclauseisnotexercised,IlosethechanceofgettingtheinformationIwanted.Sometimesit’snoteasyorquicktogettothebreakpoint,sothinkcarefullybeforeplacingthem.

Anotherimportantthingiswheretostart.Imaginethatyouhave100linesofcodethathandleyourdata.Datacomesinatline1,andsomehowit’swrongatline100.Youdon’tknowwherethebugis,sowhatdoyoudo?Youcanplaceabreakpointatline1andpatientlygothroughallthelines,checkingyourdata.Intheworstcasescenario,99lineslater(andmanycoffeecups)youspotthebug.So,considerusingadifferentapproach.

Youstartatline50,andinspect.Ifthedataisgood,itmeansthebughappenslater,inwhichcaseyouplaceyournextbreakpointatline75.Ifthedataatline50isalreadybad,yougoonbyplacingabreakpointatline25.Then,yourepeat.Eachtime,youmoveeitherbackwardsorforwards,byhalfthejumpyoudidlasttime.

Inourworstcasescenario,yourdebuggingwouldgofrom1,2,3,…,99to50,75,87,93,96,…,99whichiswayfaster.Infact,it’slogarithmic.Thissearchingtechniqueiscalledbinarysearch,it’sbasedonadivideandconquerapproachandit’sveryeffective,sotrytomasterit.

UsingteststodebugDoyourememberChapter7,Testing,Profiling,andDealingwithExceptions,abouttests?Well,ifwehaveabugandalltestsarepassing,itmeanssomethingiswrongormissinginourtestcodebase.So,oneapproachistomodifythetestsinsuchawaythattheycaterforthenewedgecasethathasbeenspotted,andthenworkyourwaythroughthecode.Thisapproachcanbeverybeneficial,becauseitmakessurethatyourbugwillbecoveredbyatestwhenit’sfixed.

MonitoringMonitoringisalsoveryimportant.Softwareapplicationscangocompletelycrazyandhavenon-deterministichiccupswhentheyencounteredgecasesituationssuchasthenetworkbeingdown,aqueuebeingfull,anexternalcomponentbeingunresponsive,andsoon.Inthesecases,it’simportanttohaveanideaofwhatwasthebigpicturewhentheproblemhappenedandbeabletocorrelateittosomethingrelatedtoitinasubtle,perhapsmysteriousway.

YoucanmonitorAPIendpoints,processes,webpagesavailabilityandloadtime,andbasicallyalmosteverythingthatyoucancode.Ingeneral,whenstartinganapplicationfromscratch,itcanbeveryusefultodesignitkeepinginmindhowyouwanttomonitorit.

SummaryInthisshortchapter,wesawdifferenttechniquesandsuggestionstodebugandtroubleshootourcode.Debuggingisanactivitythatisalwayspartofasoftwaredeveloper’swork,soit’simportanttobegoodatit.

Ifapproachedwiththecorrectattitude,itcanbefunandrewarding.

Wesawtechniquestoinspectourcodebaseonfunctions,logging,debuggers,tracebackinformation,profiling,andassertions.Wesawsimpleexamplesofmostofthemandwealsotalkedaboutasetofguidelinesthatwillhelpwhenitcomestofacethefire.

Justremembertoalwaysstaycalmandfocused,anddebuggingwillbeeasieralready.Thistoo,isaskillthatneedstobelearnedandit’sthemostimportant.Anagitatedandstressedmindcannotworkproperly,logicallyandcreatively,therefore,ifyoudon’tstrengthenit,itwillbehardforyoutoputallofyourknowledgetogooduse.

Inthenextchapter,wewillendthebookwithanothersmallprojectwhosegoalistoleaveyoumorethirstythanyouwerewhenyoustartedthisjourneywithme.

Ready?

Chapter12.SummingUp–ACompleteExample “Donotdwellinthepast,donotdreamofthefuture,concentratethemindonthepresentmoment.”

—TheShakyamuniBuddha

Inthischapter,Iwillshowyouonelastproject.Ifyou’veworkedwellintherestofthebook,thisexampleshouldbeeasy.Itriedmybesttocraftitinawaythatitwillneitherbetoohardforthosewhohaveonlyreadthebook,nortoosimpleforthosewhoalsotookthetimetoworkontheexamples,andmaybehavereaduponthelinksandtopicsIsuggested.

ThechallengeOneproblemthatweallhavethesedaysisrememberingpasswords.Wehavepasswordsforeverything:websites,phones,cards,bankaccounts,andsoon.Theamountofinformationwehavetomemorizeisjusttoomuch,somanypeopleendupusingthesamepasswordoverandoveragain.Thisisverybad,ofcourse,soatsomepoint,toolswereinventedtoalleviatethisproblem.OneofthesetoolsiscalledKeepassX,andbasicallyitworkslikethis:youstartthesoftwarebysettingupaspecialpasswordcalledmasterpassword.Onceinside,youstorearecordforeachpasswordyouneedtomemorize,forexample,youre-mailaccount,thebankwebsite,creditcardinformation,andsoon.Whenyouclosethesoftware,itencryptsthedatabaseusedtostoreallthatinformation,sothatthedatacanonlybeaccessedbytheownerofthemasterpassword.Therefore,kindofinaLordofTheRingsfashion,byjustowningonepassword,yourulethemall.

OurimplementationOurgoalinthischapteristocreatesomethingsimilarbutweb-based,andthewayIwanttoimplementitisbywritingtwoapplications.

OnewillbeanAPIwritteninFalcon.Itspurposewillbetwofold,itwillbeabletobothgenerateandvalidatepasswords.Itwillprovidethecallerwithinformationaboutthevalidityandascorewhichshouldindicatehowstrongthepasswordis.

ThesecondapplicationisaDjangowebsite,whichwillprovidetheinterfacetohandlerecords.Eachrecordwillretaininformationsuchastheusername,e-mail,password,URL,andsoon.Itwillshowalistofalltherecords,anditwillallowtheusertocreate,updateanddeletethem.Passwordswillbeencryptedbeforebeingstoredinthedatabase.

Thepurposeofthewholeprojectis,therefore,tomimicthewayKeepassXworks,eventhoughitisinamuchsimplerfashion.Itwillbeuptoyou,ifyoulikethisidea,todevelopitfurtherinordertoaddotherfeaturesandmakeitmoresecure.Iwillmakesuretogiveyousomesuggestionsonhowtoextendit,towardstheend.

Thischapterwillthereforebequitedense,code-wise.It’sthepriceIhavetopayforgivingyouaninterestingexampleinarestrictedamountofspace.

Beforewestart,pleasemakesureyouarecomfortablewiththeprojectspresentedinChapter10,WebDevelopmentDoneRightsothatyou’refamiliarwiththebasicsofwebdevelopment.Makesurealsothatyouhaveinstalledallthepippackagesneededforthisproject:django,falcon,cryptography,andnose-parameterized.Ifyoudownloadthesourcecodeforthebook,you’llfindeverythingyouneedtoinstallintherequirementsfolder,whilethecodeforthischapterwillbeinch12.

ImplementingtheDjangointerfaceIhopeyou’recomfortablewiththeconceptspresentedinChapter10,WebDevelopmentDoneRightwhichwasmostlyaboutDjango.Ifyouhaven’treadit,thisisprobablyagoodtime,beforereadingonhere.

ThesetupInyourrootfolder(ch12,forme),whichwillcontaintherootfortheinterfaceandtherootfortheAPI,startbyrunningthiscommand:

$django-adminstartprojectpwdweb

ThiswillcreatethestructureforaDjangoproject,whichweknowwellbynow.I’llshowyouthefinalstructureoftheinterfaceprojecthere:

$tree-Apwdweb

pwdweb

├──db.sqlite3

├──manage.py

├──pwdweb

│├──__init__.py

│├──settings.py

│├──urls.py

│└──wsgi.py

└──records

├──admin.py

├──forms.py

├──__init__.py

├──migrations

│├──0001_initial.py

│└──__init__.py

├──models.py

├──static

│└──records

│├──css

││└──main.css

│└──js

│├──api.js

│└──jquery-2.1.4.min.js

├──templates

│└──records

│├──base.html

│├──footer.html

│├──home.html

│├──list.html

│├──messages.html

│├──record_add_edit.html

│└──record_confirm_delete.html

├──templatetags

│└──record_extras.py

├──urls.py

└──views.py

Asusual,don’tworryifyoudon’thaveallthefiles,we’lladdthemgradually.Changetothepwdwebfolder,andmakesureDjangoiscorrectlysetup:$pythonmanage.pyrunserver(ignorethewarningaboutunappliedmigrations).

Shutdowntheserverandcreateanapp:$pythonmanage.pystartapprecords.Thatisexcellent,nowwecanstartcoding.Firstthingsfirst,let’sopenpwdweb/settings.pyand

startbyadding'records',attheendoftheINSTALLED_APPtuple(notethatthecommaisincludedinthecode).Then,goaheadandfixtheLANGUAGE_CODEandTIME_ZONEsettingsaccordingtoyourpreferenceandfinally,addthefollowinglineatthebottom:

ENCRYPTION_KEY=b'qMhPGx-ROWUDr4veh0ybPRL6viIUNe0vcPDmy67x6CQ='

ThisisacustomencryptionkeythathasnothingtodowithDjangosettings,butwewillneeditlateron,andthisisthebestplaceforittobe.Don’tworryfornow,we’llgetbacktoit.

ThemodellayerWeneedtoaddjustonemodelfortherecordsapplication:Record.Thismodelwillrepresenteachrecordwewanttostoreinthedatabase:records/models.py

fromcryptography.fernetimportFernet

fromdjango.confimportsettings

fromdjango.dbimportmodels

classRecord(models.Model):

DEFAULT_ENCODING='utf-8'

title=models.CharField(max_length=64,unique=True)

username=models.CharField(max_length=64)

email=models.EmailField(null=True,blank=True)

url=models.URLField(max_length=255,null=True,blank=True)

password=models.CharField(max_length=2048)

notes=models.TextField(null=True,blank=True)

created=models.DateTimeField(auto_now_add=True)

last_modified=models.DateTimeField(auto_now=True)

defencrypt_password(self):

self.password=self.encrypt(self.password)

defdecrypt_password(self):

self.password=self.decrypt(self.password)

defencrypt(self,plaintext):

returnself.cypher('encrypt',plaintext)

defdecrypt(self,cyphertext):

returnself.cypher('decrypt',cyphertext)

defcypher(self,cypher_func,text):

fernet=Fernet(settings.ENCRYPTION_KEY)

result=getattr(fernet,cypher_func)(

self._to_bytes(text))

returnself._to_str(result)

def_to_str(self,bytes_str):

returnbytes_str.decode(self.DEFAULT_ENCODING)

def_to_bytes(self,s):

returns.encode(self.DEFAULT_ENCODING)

Firstly,wesettheDEFAULT_ENCODINGclassattributeto'utf-8',whichisthemostpopulartypeofencodingfortheweb(andnotonlytheweb).Wesetthisattributeontheclasstoavoidhardcodingastringinmorethanoneplace.

Then,weproceedtosetupallthemodel’sfields.Asyoucansee,Djangoallowsyoutospecifyveryspecificfields,suchasEmailFieldandURLField.Thereasonwhyit’sbettertousethesespecificfieldsinsteadofaplainandsimpleCharFieldiswe’llgete-mailandURLvalidationforfreewhenwecreateaformforthismodel,whichisbrilliant.

Alltheoptionsarequitestandard,andwesawtheminChapter10,WebDevelopmentDoneRightbutIwanttopointoutafewthingsanyway.Firstly,titleneedstobeuniquesothateachRecordobjecthasauniquetitleandwedon’twanttoriskhavingdoubles.Eachdatabasetreatsstringsalittlebitdifferently,accordingtohowitissetup,whichengineitruns,andsoon,soIhaven’tmadethetitlefieldtheprimarykeyforthismodel,whichwouldhavebeenthenaturalthingtodo.IprefertoavoidthepainofhavingtodealwithweirdstringerrorsandIamhappywithlettingDjangoaddaprimarykeytothemodelautomatically.

Anotheroptionyoushouldunderstandisthenull=True,blank=Truecouple.TheformerallowsthefieldtobeNULL,whichmakesitnon-mandatory,whilethesecondallowsittobeblank(thatistosay,anemptystring).TheiruseisquitepeculiarinDjango,soIsuggestyoutotakealookattheofficialdocumentationtounderstandexactlyhowtousethem.

Finally,thedates:createdneedstohaveauto_add_now=True,whichwillsetthecurrentmomentintimeontheobjectwhenit’screated.Ontheotherhand,last_modifiedneedstobeupdatedeverytimewesavethemodel,hencewesetauto_now=True.

Afterthefielddefinitions,thereareafewmethodsforencryptinganddecryptingthepassword.Itisalwaysaverybadideatosavepasswordsastheyareinadatabase,thereforeyoushouldalwaysencryptthembeforesavingthem.

Normally,whensavingapassword,youencryptitusingaonewayencryptionalgorithm(alsoknownasaonewayhashfunction).Thismeansthat,onceyouhavecreatedthehash,thereisnowayforyoutorevertitbacktotheoriginalpassword.

Thiskindofencryptionisnormallyusedforauthentication:theuserputstheirusernameandpasswordinaformand,onsubmission,thecodefetchesthehashfromtheuserrecordinthedatabaseandcomparesitwiththehashofthepasswordtheuserhasjustputintheform.Ifthetwohashesmatch,itmeansthattheywereproducedbythesamepassword,thereforeauthenticationisgranted.

Inthiscasethough,weneedtobeabletorecoverthepasswords,otherwisethiswholeapplicationwouldn’tbeveryuseful.Therefore,wewilluseaso-calledsymmetricencryptionalgorithmtoencryptthem.Thewaythisworksisverysimple:thepassword(calledplaintext)ispassedtoanencryptfunction,alongwithasecretkey.Thealgorithmproducesanencryptedstring(calledcyphertext)outofthem,whichiswhatyoustoreinthedatabase.Whenyouwanttorecoverthepassword,youwillneedthecyphertextandthesecretkey.Youfeedthemtoadecryptfunction,andyougetbackyouroriginalpassword.Thisisexactlywhatweneed.

Inordertoperformsymmetricencryption,weneedthecryptographypackage,whichiswhyIinstructedyoutoinstallit.

AllthemethodsintheRecordclassareverysimple.encrypt_passwordanddecrypt_passwordareshortcutstoencryptanddecryptthepasswordfieldandreassigntheresulttoitself.

Theencryptanddecryptmethodsaredispatchersforthecyphermethod,and_to_str

and_to_bytesarejustacoupleofhelpers.Thecryptographylibraryworkswithbytesobjects,soweneedthosehelperstogobackandforthbetweenbytesandstrings,usingacommonencoding.

Theonlyinterestinglogicisinthecyphermethod.Icouldhavecodeditdirectlyintheencryptanddecryptones,butthatwouldhaveresultedinabitofredundancy,andIwouldn’thavehadthechancetoshowyouadifferentwayofaccessinganobject’sattribute,solet’sanalyzethebodyofcypher.

WestartbycreatinganinstanceoftheFernetclass,whichprovidesuswiththesymmetricencryptionfunctionalityweneed.Wesettheinstanceupbypassingthesecretkeyinthesettings(ENCRYPTION_KEY).Aftercreatingfernet,weneedtouseit.Wecanuseittoeitherencryptordecrypt,accordingtowhatvalueisgiventothecypher_funcparameter.Weusegetattrtogetanattributefromanobjectgiventheobjectitselfandthenameoftheattribute.Thistechniqueallowsustofetchanyattributefromanobjectdynamically.

Theresultofgetattr(fernet,cypher_func),withcyper_funcbeing'encrypt',forexample,isthesameasfernet.encrypt.Thegetattrfunctionreturnsamethod,whichwethencallwiththebytesrepresentationofthetextargument.Wethenreturntheresult,instringformat.

Here’swhatthisfunctionisequivalenttowhenit’scalledbytheencryptdispatcher:

defcypher_encrypt(self,text):

fernet=Fernet(settings.ENCRYPTION_KEY)

result=fernet.encrypt(

self._to_bytes(text))

returnself._to_str(result)

Whenyoutakethetimetounderstanditproperly,you’llseeit’snotasdifficultasitsounds.

So,wehaveourmodel,henceit’stimetomigrate(Ihopeyourememberthatthiswillcreatethetablesinthedatabaseforyourapplication):

$pythonmanage.pymakemigrations

$pythonmanage.pymigrate

Nowyoushouldhaveanicedatabasewithallthetablesyouneedtoruntheinterfaceapplication.Goaheadandcreateasuperuser($pythonmanage.pycreatesuperuser).

Bytheway,ifyouwanttogenerateyourownencryptionkey,itisaseasyasthis:

>>>fromcryptography.fernetimportFernet

>>>Fernet.generate_key()

AsimpleformWeneedaformfortheRecordmodel,sowe’llusetheModelFormtechniquewesawinChapter10,WebDevelopmentDoneRight.records/forms.py

fromdjango.formsimportModelForm,Textarea

from.modelsimportRecord

classRecordForm(ModelForm):

classMeta:

model=Record

fields=['title','username','email','url',

'password','notes']

widgets={'notes':Textarea(

attrs={'cols':40,'rows':4})}

WecreateaRecordFormclassthatinheritsfromModelForm,sothattheformiscreatedautomaticallythankstotheintrospectioncapabilitiesofDjango.Weonlyspecifywhichmodeltouse,whichfieldstodisplay(weexcludethedates,whicharehandledautomatically)andweprovideminimalstylingforthedimensionsofthenotesfield,whichwillbedisplayedusingaTextarea(whichisamultilinetextfieldinHTML).

TheviewlayerThereareatotaloffivepagesintheinterfaceapplication:home,recordlist,recordcreation,recordupdate,andrecorddeleteconfirmation.Hence,therearefiveviewsthatwehavetowrite.Asyou’llseeinamoment,Djangohelpsusalotbygivingusviewswecanreusewithminimumcustomization.Allthecodethatfollowsbelongstotherecords/views.pyfile.

ImportsandhomeviewJusttobreaktheice,herearetheimportsandtheviewforthehomepage:

fromdjango.contribimportmessages

fromdjango.contrib.messages.viewsimportSuccessMessageMixin

fromdjango.core.urlresolversimportreverse_lazy

fromdjango.views.genericimportTemplateView

fromdjango.views.generic.editimport(

CreateView,UpdateView,DeleteView)

from.formsimportRecordForm

from.modelsimportRecord

classHomeView(TemplateView):

template_name='records/home.html'

WeimportafewtoolsfromDjango.Thereareacoupleofmessaging-relatedobjects,aURLlazyreverser,andfourdifferenttypesofview.WealsoimportourRecordmodelandRecordForm.Asyoucansee,theHomeViewclassconsistsofonlytwolinessinceweonlyneedtospecifywhichtemplatewewanttouse,therestjustreusesthecodefromTemplateView,asitis.It’ssoeasy,italmostfeelslikecheating.

ListingallrecordsAfterthehomeview,wecanwriteaviewtolistalltheRecordinstancesthatwehaveinthedatabase.

classRecordListView(TemplateView):

template_name='records/list.html'

defget(self,request,*args,**kwargs):

context=self.get_context_data(**kwargs)

records=Record.objects.all().order_by('title')#1

forrecordinrecords:

record.plaintext=record.decrypt(record.password)#2

context['records']=records

returnself.render_to_response(context)

Allweneedtodoissub-classTemplateViewagain,andoverridethegetmethod.Weneedtodoacoupleofthings:wefetchalltherecordsfromthedatabaseandsortthembytitle(#1)andthenparsealltherecordsinordertoaddtheattributeplaintext(#2)ontoeachofthem,toshowtheoriginalpasswordonthepage.Anotherwayofdoingthiswouldbetoaddaread-onlypropertytotheRecordmodel,todothedecryptiononthefly.I’llleaveittoyou,asafunexercise,toamendthecodetodoit.

Afterrecoveringandaugmentingtherecords,weputtheminthecontextdictandfinishasusualbyinvokingrender_to_response.

CreatingrecordsHere’sthecodeforthecreationview:

classEncryptionMixin:

defform_valid(self,form):

self.encrypt_password(form)

returnsuper(EncryptionMixin,self).form_valid(form)

defencrypt_password(self,form):

self.object=form.save(commit=False)

self.object.encrypt_password()

self.object.save()

classRecordCreateView(

EncryptionMixin,SuccessMessageMixin,CreateView):

template_name='records/record_add_edit.html'

form_class=RecordForm

success_url=reverse_lazy('records:add')

success_message='Recordwascreatedsuccessfully'

Apartofitslogichasbeenfactoredoutinordertobereusedlateronintheupdateview.Let’sstartwithEncryptionMixin.Allitdoesisoverridetheform_validmethodsothat,priortosavinganewRecordinstancetothedatabase,wemakesurewecallencrypt_passwordontheobjectthatresultsfromsavingtheform.Inotherwords,whentheusersubmitstheformtocreateanewRecord,iftheformvalidatessuccessfully,thentheform_validmethodisinvoked.WithinthismethodwhatusuallyhappensisthatanobjectiscreatedoutoftheModelForminstance,likethis:

self.object=form.save()

Weneedtointerferewiththisbehaviorbecauserunningthiscodeasitiswouldsavetherecordwiththeoriginalpassword,whichisn’tencrypted.Sowechangethistocallsaveontheformpassingcommit=False,whichcreatestheRecordinstanceoutoftheform,butdoesn’tattempttosaveitinthedatabase.Immediatelyafterwards,weencryptthepasswordonthatinstanceandthenwecanfinallycallsaveonit,actuallycommittingittothedatabase.

Sinceweneedthisbehaviorbothforcreatingandupdatingrecords,Ihavefactoreditoutinamixin.

NotePerhaps,abettersolutionforthispasswordencryptionlogicistocreateacustomField(inheritingfromCharFieldistheeasiestwaytodoit)andaddthenecessarylogictoit,sothatwhenwehandleRecordinstancesfromandtothedatabase,theencryptionanddecryptionlogicisperformedautomaticallyforus.Thoughmoreelegant,thissolutionneedsmetodigressandexplainalotmoreaboutDjangointernals,whichistoomuchfortheextentofthisexample.Asusual,youcantrytodoityourself,ifyoufeellikea

challenge.

AftercreatingtheEncryptionMixinclass,wecanuseitintheRecordCreateViewclass.Wealsoinheritfromtwootherclasses:SuccessMessageMixinandCreateView.Themessagemixinprovidesuswiththelogictoquicklysetupamessagewhencreationissuccessful,andtheCreateViewgivesusthenecessarylogictocreateanobjectfromaform.

Youcanseethatallwehavetocodeissomecustomization:thetemplatename,theformclass,andthesuccessmessageandURL.EverythingelseisgracefullyhandledforusbyDjango.

UpdatingrecordsThecodetoupdateaRecordinstanceisonlyatinybitmorecomplicated.Wejustneedtoaddsomelogictodecryptthepasswordbeforewepopulatetheformwiththerecorddata.

classRecordUpdateView(

EncryptionMixin,SuccessMessageMixin,UpdateView):

template_name='records/record_add_edit.html'

form_class=RecordForm

model=Record

success_message='Recordwasupdatedsuccessfully'

defget_context_data(self,**kwargs):

kwargs['update']=True

returnsuper(

RecordUpdateView,self).get_context_data(**kwargs)

defform_valid(self,form):

self.success_url=reverse_lazy(

'records:edit',

kwargs={'pk':self.object.pk}

)

returnsuper(RecordUpdateView,self).form_valid(form)

defget_form_kwargs(self):

kwargs=super(RecordUpdateView,self).get_form_kwargs()

kwargs['instance'].decrypt_password()

returnkwargs

Inthisview,westillinheritfrombothEncryptionMixinandSuccessMessageMixin,buttheviewclassweuseisUpdateView.

Thefirstfourlinesarecustomizationasbefore,wesetthetemplatename,theformclass,theRecordmodel,andthesuccessmessage.Wecannotsetthesuccess_urlasaclassattributebecausewewanttoredirectasuccessfuledittothesameeditpageforthatrecordand,inordertodothis,weneedtheIDoftheinstancewe’reediting.Noworries,we’lldoitanotherway.

First,weoverrideget_context_datainordertoset'update'toTrueinthekwargsargument,whichmeansthatakey'update'willendupinthecontextdictthatispassedtothetemplateforrenderingthepage.Wedothisbecausewewanttousethesame

templateforcreatingandupdatingarecord,thereforewewillusethisvariableinthecontexttobeabletounderstandinwhichsituationweare.ThereareotherwaystodothisbutthisoneisquickandeasyandIlikeitbecauseit’sexplicit.

Afteroverridingget_context_data,weneedtotakecareoftheURLredirection.Wedothisintheform_validmethodsinceweknowthat,ifwegetthere,itmeanstheRecordinstancehasbeensuccessfullyupdated.Wereversethe'records:edit'view,whichisexactlytheviewwe’reworkingon,passingtheprimarykeyoftheobjectinquestion.Wetakethatinformationfromself.object.pk.

Oneofthereasonsit’shelpfultohavetheobjectsavedontheviewinstanceisthatwecanuseitwhenneededwithouthavingtoalterthesignatureofthemanymethodsintheviewinordertopasstheobjectaround.Thisdesignisveryhelpfulandallowsustoachievealotwithveryfewlinesofcode.

Thelastthingweneedtodoistodecryptthepasswordontheinstancebeforepopulatingtheformfortheuser.It’ssimpleenoughtodoitintheget_form_kwargsmethod,whereyoucanaccesstheRecordinstanceinthekwargsdict,andcalldecrypt_passwordonit.

Thisisallweneedtodotoupdatearecord.Ifyouthinkaboutit,theamountofcodewehadtowriteisreallyverylittle,thankstoDjangoclass-basedviews.

TipAgoodwayofunderstandingwhichisthebestmethodtooverride,istotakealookattheDjangoofficialdocumentationor,evenbetterinthiscase,checkoutthesourcecodeandlookattheclass-basedviewssection.You’llbeabletoappreciatehowmuchworkhasbeendonetherebyDjangodeveloperssothatyouonlyhavetotouchthesmallestamountsofcodetocustomizeyourviews.

DeletingrecordsOfthethreeactions,deletingarecordisdefinitelytheeasiestone.Allweneedisthefollowingcode:

classRecordDeleteView(SuccessMessageMixin,DeleteView):

model=Record

success_url=reverse_lazy('records:list')

defdelete(self,request,*args,**kwargs):

messages.success(

request,'Recordwasdeletedsuccessfully')

returnsuper(RecordDeleteView,self).delete(

request,*args,**kwargs)

WeonlyneedtoinheritfromSuccessMessageMixinandDeleteView,whichgivesusallweneed.WesetupthemodelandthesuccessURLasclassattributes,andthenweoverridethedeletemethodonlytoaddanicemessagethatwillbedisplayedinthelistview(whichiswhereweredirecttoafterdeletion).

Wedon’tneedtospecifythetemplatename,sincewe’lluseanamethatDjangoinfersbydefault:record_confirm_delete.html.

Withthisfinalview,we’reallsettohaveaniceinterfacethatwecanusetohandleRecordinstances.

SettinguptheURLsBeforewemoveontothetemplatelayer,let’ssetuptheURLs.Thistime,IwanttoshowyoutheinclusiontechniqueItalkedaboutinChapter10,WebDevelopmentDoneRight.pwdweb/urls.py

fromdjango.conf.urlsimportinclude,url

fromdjango.contribimportadmin

fromrecordsimporturlsasrecords_url

fromrecords.viewsimportHomeView

urlpatterns=[

url(r'^admin/',include(admin.site.urls)),

url(r'^records/',include(records_url,namespace='records')),

url(r'^$',HomeView.as_view(),name='home'),

]

ThesearetheURLsforthemainproject.Wehavetheusualadmin,ahomepage,andthenfortherecordssection,weincludeanotherurls.pyfile,whichwedefineintherecordsapplication.Thistechniqueallowsforappstobereusableandself-contained.Notethat,whenincludinganotherurls.pyfile,youcanpassnamespaceinformation,whichyoucanthenuseinfunctionssuchasreverse,ortheurltemplatetag.Forexample,we’veseenthatthepathtotheRecordUpdateViewwas'records:edit'.Thefirstpartofthatstringisthenamespace,andthesecondisthenamethatwehavegiventotheview,asyoucanseeinthefollowingcode:records/urls.py

fromdjango.conf.urlsimportinclude,url

fromdjango.contribimportadmin

from.viewsimport(RecordCreateView,RecordUpdateView,

RecordDeleteView,RecordListView)

urlpatterns=[

url(r'^add/$',RecordCreateView.as_view(),name='add'),

url(r'^edit/(?P<pk>[0-9]+)/$',RecordUpdateView.as_view(),

name='edit'),

url(r'^delete/(?P<pk>[0-9]+)/$',RecordDeleteView.as_view(),

name='delete'),

url(r'^$',RecordListView.as_view(),name='list'),

]

Wedefinefourdifferenturlinstances.Thereisoneforaddingarecord,whichdoesn’tneedprimarykeyinformationsincetheobjectdoesn’texistyet.Thenwehavetwourlinstancesforupdatinganddeletingarecord,andforthoseweneedtoalsospecifyprimarykeyinformationtobepassedtotheview.SinceRecordinstanceshaveintegerIDs,wecansafelypassthemontheURL,followinggoodURLdesignpractice.Finally,wedefineoneurlinstanceforthelistofrecords.

Allurlinstanceshavenameinformationwhichisusedinviewsandtemplates.

ThetemplatelayerLet’sstartwiththetemplatewe’lluseasthebasisfortherest:records/templates/records/base.html

{%loadstaticfromstaticfiles%}

<!DOCTYPEhtml>

<htmllang="en">

<head>

<metacharset="utf-8">

<metaname="viewport"

content="width=device-width,initial-scale=1.0">

<linkhref="{%static"records/css/main.css"%}"

rel="stylesheet">

<title>{%blocktitle%}Title{%endblocktitle%}</title>

</head>

<body>

<divid="page-content">

{%blockpage-content%}{%endblockpage-content%}

</div>

<divid="footer">{%blockfooter%}{%endblockfooter%}</div>

{%blockscripts%}

<script

src="{%static"records/js/jquery-2.1.4.min.js"%}">

</script>

{%endblockscripts%}

</body>

</html>

It’sverysimilartotheoneIusedinChapter10,WebDevelopmentDoneRightalthoughitisabitmorecompressedandwithonemajordifference.WewillimportjQueryineverypage.

NotejQueryisthemostpopularJavaScriptlibraryoutthere.Itallowsyoutowritecodethatworksonallthemainbrowsersanditgivesyoumanyextratoolssuchastheabilitytoperformasynchronouscalls(AJAX)fromthebrowseritself.We’llusethislibrarytoperformthecallstotheAPI,bothtogenerateandvalidateourpasswords.Youcandownloaditathttps://jquery.com/,andputitinthepwdweb/records/static/records/js/folder(youmayhavetoamendtheimportinthetemplate).

Ihighlightedtheonlyinterestingpartofthistemplateforyou.NotethatweloadtheJavaScriptlibraryattheend.Thisiscommonpractice,asJavaScriptisusedtomanipulatethepage,soloadinglibrariesattheendhelpsinavoidingsituationssuchasJavaScriptcodefailingbecausetheelementneededhadn’tbeenrenderedonthepageyet.

HomeandfootertemplatesThehometemplateisverysimple:records/templates/records/home.html

{%extends"records/base.html"%}

{%blocktitle%}WelcometotheRecordswebsite.{%endblock%}

{%blockpage-content%}

<h1>Welcome{{user.first_name}}!</h1>

<divclass="home-option">Tocreatearecordclick

<ahref="{%url"records:add"%}">here.</a>

</div>

<divclass="home-option">Toseeallrecordsclick

<ahref="{%url"records:list"%}">here.</a>

</div>

{%endblockpage-content%}

Thereisnothingnewherewhencomparedtothehome.htmltemplatewesawinChapter10,WebDevelopmentDoneRight.Thefootertemplateisactuallyexactlythesame:records/templates/records/footer.html

<divclass="footer">

Goback<ahref="{%url"home"%}">home</a>.

</div>

ListingallrecordsThistemplatetolistallrecordsisfairlysimple:records/templates/records/list.html

{%extends"records/base.html"%}

{%loadrecord_extras%}

{%blocktitle%}Records{%endblocktitle%}

{%blockpage-content%}

<h1>Records</h1><spanname="top"></span>

{%include"records/messages.html"%}

{%forrecordinrecords%}

<divclass="record{%cycle'row-light-blue''row-white'%}"

id="record-{{record.pk}}">

<divclass="record-left">

<divclass="record-list">

<spanclass="record-span">Title</span>{{record.title}}

</div>

<divclass="record-list">

<spanclass="record-span">Username</span>

{{record.username}}

</div>

<divclass="record-list">

<spanclass="record-span">Email</span>{{record.email}}

</div>

<divclass="record-list">

<spanclass="record-span">URL</span>

<ahref="{{record.url}}"target="_blank">

{{record.url}}</a>

</div>

<divclass="record-list">

<spanclass="record-span">Password</span>

{%hide_passwordrecord.plaintext%}

</div>

</div>

<divclass="record-right">

<divclass="record-list">

<spanclass="record-span">Notes</span>

<textarearows="3"cols="40"class="record-notes"

readonly>{{record.notes}}</textarea>

</div>

<divclass="record-list">

<spanclass="record-span">Lastmodified</span>

{{record.last_modified}}

</div>

<divclass="record-list">

<spanclass="record-span">Created</span>

{{record.created}}

</div>

</div>

<divclass="record-list-actions">

<ahref="{%url"records:edit"pk=record.pk%}">»edit</a>

<ahref="{%url"records:delete"pk=record.pk%}">»delete

</a>

</div>

</div>

{%endfor%}

{%endblockpage-content%}

{%blockfooter%}

<p><ahref="#top">Gobacktotop</a></p>

{%include"records/footer.html"%}

{%endblockfooter%}

Forthistemplateaswell,IhavehighlightedthepartsI’dlikeyoutofocuson.Firstly,Iloadacustomtagsmodule,record_extras,whichwe’llneedlater.Ihavealsoaddedananchoratthetop,sothatwe’llbeabletoputalinktoitatthebottomofthepage,toavoidhavingtoscrollallthewayup.

Then,IincludedatemplatetoprovidemewiththeHTMLcodetodisplayDjangomessages.It’saverysimpletemplatewhichI’llshowyoushortly.

Then,wedefinealistofdivelements.EachRecordinstancehasacontainerdiv,inwhichtherearetwoothermaindivelements:record-leftandrecord-right.Inordertodisplaythemsidebyside,Ihavesetthisclassinthemain.cssfile:

.record-left{float:left;width:300px;}

Theoutermostdivcontainer(theonewithclassrecord),hasanidattribute,whichIhaveusedasananchor.Thisallowsustoclickoncancelontherecorddeletepage,sothatifwechangeourmindsanddon’twanttodeletetherecord,wecangetbacktothelistpage,andattherightposition.

Eachattributeoftherecordisthendisplayedindivelementswhoseclassisrecord-list.Mostoftheseclassesarejusttheretoallowmetosetabitofpaddinganddimensionson

theHTMLelements.

Thenextinterestingbitisthehide_passwordtag,whichtakestheplaintext,whichistheunencryptedpassword.Thepurposeofthiscustomtagistodisplayasequenceof'*'characters,aslongastheoriginalpassword,sothatifsomeoneispassingbywhileyou’reonthepage,theywon’tseeyourpasswords.However,hoveringonthatsequenceof'*'characterswillshowyoutheoriginalpasswordinthetooltip.Here’sthecodeforthehide_passwordtag:records/templatetags/record_extras.py

fromdjangoimporttemplate

fromdjango.utils.htmlimportescape

register=template.Library()

@register.simple_tag

defhide_password(password):

return'<spantitle="{0}">{1}</span>'.format(

escape(password),'*'*len(password))

Thereisnothingfancyhere.Wejustregisterthisfunctionasasimpletagandthenwecanuseitwhereverwewant.Ittakesapasswordandputsitasatooltipofaspanelement,whosemaincontentisasequenceof'*'characters.Justnoteonething:weneedtoescapethepassword,sothatwe’resureitwon’tbreakourHTML(thinkofwhatmighthappenifthepasswordcontainedadouble-quote`"`,forexample).

Asfarasthelist.htmltemplateisconcerned,thenextinterestingbitisthatwesetthereadonlyattributetothetextareaelement,soasnottogivetheimpressiontotheuserthattheycanmodifynotesonthefly.

Then,wesetacoupleoflinksforeachRecordinstance,rightatthebottomofthecontainerdiv.Thereisonefortheeditpage,andanotherforthedeletepage.Notethatweneedtopasstheurltagnotonlythenamespace:namestring,butalsotheprimarykeyinformation,asrequiredbytheURLsetupwemadeintheurls.pymoduleforthoseviews.

Finally,weimportthefooterandsetthelinktotheanchorontopofthepage.

Now,aspromised,hereisthecodeforthemessages:records/templates/records/messages.html

{%ifmessages%}

{%formessageinmessages%}

<pclass="{{message.tags}}">{{message}}</p>

{%endfor%}

{%endif%}

Thiscodetakescareofdisplayingmessagesonlywhenthereisatleastonetodisplay.Wegivetheptagclassinformationtodisplaysuccessmessagesingreenanderrormessagesinred.

Ifyougrabthemain.cssfilefromthesourcecodeforthebook,youwillnowbeableto

visualizethelistpage(yourswillbeblank,youstillneedtoinsertdataintoit),anditshouldlooksomethinglikethis:

Asyoucansee,Ihavetworecordsinthedatabaseatthemoment.I’mhoveringonthepasswordofthefirstone,whichismyplatformaccountatmysister’sschool,andthepasswordisdisplayedinthetooltip.Thedivisionintwodivelements,leftandright,helpsinmakingrowssmallersothattheoverallresultismorepleasingtotheeye.Theimportantinformationisontheleftandtheancillaryinformationisontheright.Therowcoloralternatesbetweenaverylightshadeofblueandwhite.

Eachrowhasaneditanddeletelink,atitsbottomleft.We’llshowthepagesforthosetwolinksrightafterweseethecodeforthetemplatesthatcreatethem.

TheCSScodethatholdsalltheinformationforthisinterfaceisthefollowing:records/static/records/css/main.css

html,body,*{

font-family:'TrebuchetMS',Helvetica,sans-serif;}

a{color:#333;}

.record{

clear:both;padding:1em;border-bottom:1pxsolid#666;}

.record-left{float:left;width:300px;}

.record-list{padding:2px0;}

.fieldWrapper{padding:5px;}

.footer{margin-top:1em;color:#333;}

.home-option{padding:.6em0;}

.record-span{font-weight:bold;padding-right:1em;}

.record-notes{vertical-align:top;}

.record-list-actions{padding:4px0;clear:both;}

.record-list-actionsa{padding:04px;}

#pwd-info{padding:06px;font-size:1.1em;font-weight:bold;}

#id_notes{vertical-align:top;}

/*Messages*/

.success,.errorlist{font-size:1.2em;font-weight:bold;}

.success{color:#25B725;}

.errorlist{color:#B12B2B;}

/*colors*/

.row-light-blue{background-color:#E6F0FA;}

.row-white{background-color:#fff;}

.green{color:#060;}

.orange{color:#FF3300;}

.red{color:#900;}

Pleaseremember,I’mnotaCSSgurusojusttakethisfileasitis,afairlynaivewaytoprovidestylingtoourinterface.

CreatingandeditingrecordsNowfortheinterestingpart.Creatingandupdatingarecord.We’llusethesametemplateforboth,soweexpectsomedecisionallogictobetherethatwilltellusinwhichofthetwosituationsweare.Asitturnsout,itwillnotbethatmuchcode.Themostexcitingpartofthistemplate,however,isitsassociatedJavaScriptfilewhichwe’llexaminerightafterwards.records/templates/records/record_add_edit.html

{%extends"records/base.html"%}

{%loadstaticfromstaticfiles%}

{%blocktitle%}

{%ifupdate%}Update{%else%}Create{%endif%}Record

{%endblocktitle%}

{%blockpage-content%}

<h1>{%ifupdate%}Updatea{%else%}Createanew{%endif%}

Record

</h1>

{%include"records/messages.html"%}

<formaction="."method="post">{%csrf_token%}

{{form.non_field_errors}}

<divclass="fieldWrapper">{{form.title.errors}}

{{form.title.label_tag}}{{form.title}}</div>

<divclass="fieldWrapper">{{form.username.errors}}

{{form.username.label_tag}}{{form.username}}</div>

<divclass="fieldWrapper">{{form.email.errors}}

{{form.email.label_tag}}{{form.email}}</div>

<divclass="fieldWrapper">{{form.url.errors}}

{{form.url.label_tag}}{{form.url}}</div>

<divclass="fieldWrapper">{{form.password.errors}}

{{form.password.label_tag}}{{form.password}}

<spanid="pwd-info"></span></div>

<buttontype="button"id="validate-btn">

ValidatePassword</button>

<buttontype="button"id="generate-btn">

GeneratePassword</button>

<divclass="fieldWrapper">{{form.notes.errors}}

{{form.notes.label_tag}}{{form.notes}}</div>

<inputtype="submit"

value="{%ifupdate%}Update{%else%}Insert{%endif%}">

</form>

{%endblockpage-content%}

{%blockfooter%}

<br>{%include"records/footer.html"%}<br>

Goto<ahref="{%url"records:list"%}">therecordslist</a>.

{%endblockfooter%}

{%blockscripts%}

{{block.super}}

<scriptsrc="{%static"records/js/api.js"%}"></script>

{%endblockscripts%}

Asusual,Ihavehighlightedtheimportantparts,solet’sgothroughthiscodetogether.

Youcanseethefirstbitofdecisionlogicinthetitleblock.Similardecisionlogicisalsodisplayedlateron,intheheaderofthepage(theh1HTMLtag),andinthesubmitbuttonattheendoftheform.

Apartfromthislogic,whatI’dlikeyoutofocusonistheformandwhat’sinsideit.Wesettheactionattributetoadot,whichmeansthispage,sothatwedon’tneedtocustomizeitaccordingtowhichviewisservingthepage.Also,weimmediatelytakecareofthecross-siterequestforgerytoken,asexplainedinChapter10,WebDevelopmentDoneRight.

Notethat,thistime,wecannotleavethewholeformrenderinguptoDjangosincewewanttoaddinacoupleofextrathings,sowegodownonelevelofgranularityandaskDjangotorendereachindividualfieldforus,alongwithanyerrors,alongwithitslabel.Thiswaywestillsavealotofeffort,andatthesametime,wecanalsocustomizetheformaswelike.Insituationslikethis,it’snotuncommontowriteasmalltemplatetorenderafield,inordertoavoidrepeatingthosethreelinesforeachfield.Inthiscasethough,theformissosmallIdecidedtoavoidraisingthecomplexitylevelupanyfurther.

Thespanelement,pwd-info,containstheinformationaboutthepasswordthatwegetfromtheAPI.Thetwobuttonsafterthat,validate-btnandgenerate-btn,arehookedupwiththeAJAXcallstotheAPI.

Attheendofthetemplate,inthescriptsblock,weneedtoloadtheapi.jsJavaScriptfilewhichcontainsthecodetoworkwiththeAPI.Wealsoneedtouseblock.super,

whichwillloadwhatevercodeisinthesameblockintheparenttemplate(forexample,jQuery).block.superisbasicallythetemplateequivalentofacalltosuper(ClassName,self)inPython.It’simportanttoloadjQuerybeforeourlibrary,sincethelatterisbasedontheformer.

TalkingtotheAPILet’snowtakealookatthatJavaScript.Idon’texpectyoutounderstandeverything.Firstly,thisisaPythonbookandsecondly,you’resupposedtobeabeginner(thoughbynow,ninjatrained),sofearnot.However,asJavaScripthas,bynow,becomeessentialifyou’redealingwithawebenvironment,havingaworkingknowledgeofitisextremelyimportantevenforaPythondeveloper,sotryandgetthemostoutofwhatI’mabouttoshowyou.We’llseethepasswordgenerationfirst:records/static/records/js/api.js

varbaseURL='http://127.0.0.1:5555/password';

vargetRandomPassword=function(){

varapiURL='{url}/generate'.replace('{url}',baseURL);

$.ajax({

type:'GET',

url:apiURL,

success:function(data,status,request){

$('#id_password').val(data[1]);

},

error:function(){alert('Unexpectederror');}

});

}

$(function(){

$('#generate-btn').click(getRandomPassword);

});

Firstly,wesetavariableforthebaseAPIURL:baseURL.Then,wedefinethegetRandomPasswordfunction,whichisverysimple.Atthebeginning,itdefinestheapiURLextendingbaseURLwithareplacementtechnique.EvenifthesyntaxisdifferentfromthatofPython,youshouldn’thaveanyissuesunderstandingthisline.

AfterdefiningtheapiURL,theinterestingbitcomesup.Wecall$.ajax,whichisthejQueryfunctionthatperformstheAJAXcalls.That$isashortcutforjQuery.Asyoucanseeinthebodyofthecall,it’saGETrequesttoapiURL.Ifitsucceeds(success:…),ananonymousfunctionisrun,whichsetsthevalueoftheid_passwordtextfieldtothesecondelementofthereturneddata.We’llseethestructureofthedatawhenweexaminetheAPIcode,sodon’tworryaboutthatnow.Ifanerroroccurs,wesimplyalerttheuserthattherewasanunexpectederror.

NoteThereasonwhythepasswordfieldintheHTMLhasid_passwordastheIDisduetothewayDjangorendersforms.Youcancustomizethisbehaviorusingacustomprefix,forexample.Inthiscase,I’mhappywiththeDjangodefaults.

Afterthefunctiondefinition,werunacoupleoflinesofcodetobindtheclickeventonthegenerate-btnbuttontothegetRandomPasswordfunction.Thismeansthat,afterthiscodehasbeenrunbythebrowserengine,everytimeweclickthegenerate-btnbutton,thegetRandomPasswordfunctioniscalled.

Thatwasn’tsoscary,wasit?Solet’sseewhatweneedforthevalidationpart.

Nowthereisavalueinthepasswordfieldandwewanttovalidateit.WeneedtocalltheAPIandinspectitsresponse.Sincepasswordscanhaveweirdcharacters,Idon’twanttopassthemontheURL,thereforeIwilluseaPOSTrequest,whichallowsmetoputthepasswordinitsbody.Todothis,Ineedthefollowingcode:

varvalidatePassword=function(){

varapiURL='{url}/validate'.replace('{url}',baseURL);

$.ajax({

type:'POST',

url:apiURL,

data:JSON.stringify({'password':$('#id_password').val()}),

contentType:"text/plain",//AvoidCORSpreflight

success:function(data,status,request){

varvalid=data['valid'],infoClass,grade;

varmsg=(valid?'Valid':'Invalid')+'password.';

if(valid){

varscore=data['score']['total'];

grade=(score<10?'Poor':(score<18?'Medium':'Strong'));

infoClass=(score<10?'red':(score<18?'orange':'green'));

msg+='(Score:{score},{grade})'

.replace('{score}',score).replace('{grade}',grade);

}

$('#pwd-info').html(msg);

$('#pwd-info').removeClass().addClass(infoClass);

},

error:function(data){alert('Unexpectederror');}

});

}

$(function(){

$('#validate-btn').click(validatePassword);

});

Theconceptisthesameasbefore,onlythistimeit’sforthevalidate-btnbutton.ThebodyoftheAJAXcallissimilar.WeuseaPOSTinsteadofaGETrequest,andwedefinethedataasaJSONobject,whichistheequivalentofusingjson.dumps({'password':'some_pwd'})inPython.

ThecontentTypelineisaquickhacktoavoidproblemswiththeCORSpreflightbehaviorofthebrowser.Cross-originresourcesharing(CORS)isamechanismthatallowsrestrictedresourcesonawebpagetoberequestedfromanotherdomainoutsideofthedomainfromwhichtherequestoriginated.Inanutshell,sincetheAPIislocatedat127.0.0.1:5555andtheinterfaceisrunningat127.0.0.1:8000,withoutthishack,thebrowserwouldn’tallowustoperformthecalls.Inaproductionenvironment,youmaywanttocheckthedocumentationforJSONP,whichisamuchbetter(albeitmore

complex)solutiontothisissue.

Thebodyoftheanonymousfunctionwhichisrunifthecallsucceedsisapparentlyonlyabitcomplicated.Allweneedtodoisunderstandifthepasswordisvalid(fromdata['valid']),andassignitagradeandaCSSclassbasedonitsscore.ValidityandscoreinformationcomefromtheAPIresponse.

TheonlytrickybitinthiscodeistheJavaScriptternaryoperator,solet’sseeacomparativeexampleforit:

#Python

error='critical'iferror_level>50else'medium'

//JavaScriptequivalent

error=(error_level>50?'critical':'medium');

Withthisexample,youshouldn’thaveanyissuereadingtherestofthelogicinthefunction.Iknow,Icouldhavejustusedaregularif(...),butJavaScriptcodersusetheternaryoperatorallthetime,soyoushouldgetusedtoit.It’sgoodtrainingtoscratchourheadsabitharderinordertounderstandcode.

Lastly,I’dlikeyoutotakealookattheendofthatfunction.Wesetthehtmlofthepwd-infospanelementtothemessageweassembled(msg),andthenwestyleit.Inoneline,weremovealltheCSSclassesfromthatelement(removeClass()withnoparametersdoesthat),andweaddtheinfoClasstoit.infoClassiseither'red','orange',or'green'.Ifyougobacktothemain.cssfile,you’llseethematthebottom.

Nowthatwe’veseenboththetemplatecodeandtheJavaScripttomakethecalls,let’sseeascreenshotofthepage.We’regoingtoeditthefirstrecord,theoneaboutmysister’sschool.

Inthepicture,youcanseethatIupdatedthepasswordbyclickingontheGeneratePasswordbutton.Then,Isavedtherecord(soyoucouldseethenicemessageontop),and,finally,IclickedontheValidatePasswordbutton.

Theresultisshowningreenontheright-handsideofthePasswordfield.It’sstrong(23isactuallythemaximumscorewecanget)sothemessageisdisplayedinaniceshadeofgreen.

DeletingrecordsTodeletearecord,gotothelistandclickonthedeletelink.You’llberedirectedtoapagethatasksyouforconfirmation;youcanthenchoosetoproceedanddeletethepoorrecord,ortocanceltherequestandgobacktothelistpage.Thetemplatecodeisthefollowing:records/templates/records/record_confirm_delete.html

{%extends"records/base.html"%}

{%blocktitle%}Deleterecord{%endblocktitle%}

{%blockpage-content%}

<h1>ConfirmRecordDeletion</h1>

<formaction="."method="post">{%csrf_token%}

<p>Areyousureyouwanttodelete"{{object}}"?</p>

<inputtype="submit"value="Confirm"/>&nbsp;

<ahref="{%url"records:list"%}#record-{{object.pk}}">

»cancel</a>

</form>

{%endblockpage-content%}

SincethisisatemplateforastandardDjangoview,weneedtousethenamingconventionsadoptedbyDjango.Therefore,therecordinquestioniscalledobjectinthetemplate.The{{object}}tagdisplaysastringrepresentationfortheobject,whichisnotexactlybeautifulatthemoment,sincethewholelinewillread:Areyousureyouwanttodelete“Recordobject”?.

Thisisbecausewehaven’taddeda__str__methodtoourModelclassyet,whichmeansthatPythonhasnoideaofwhattoshowuswhenweaskforastringrepresentationofaninstance.Let’schangethisbycompletingourmodel,addingthe__str__methodatthebottomoftheclassbody:records/models.py

classRecord(models.Model):

...

def__str__(self):

return'{}'.format(self.title)

Restarttheserverandnowthepagewillread:Areyousureyouwanttodelete“SomeBank”?whereSomeBankisthetitleoftherecordwhosedeletelinkIclickedon.

Wecouldhavejustused{{object.title}},butIprefertofixtherootoftheproblem,notjusttheeffect.Addinga__str__methodisinfactsomethingthatyououghttodofor

allofyourmodels.

Theinterestingbitinthislasttemplateisactuallythelinkforcancelingtheoperation.Weusetheurltagtogobacktothelistview(records:list),butweaddanchorinformationtoitsothatitwilleventuallyreadsomethinglikethis(thisisforpk=2):http://127.0.0.1:8000/records/#record-2

ThiswillgobacktothelistpageandscrolldowntothecontainerdivthathasIDrecord2,whichisnice.

Thisconcludestheinterface.EventhoughthissectionwassimilartowhatwesawinChapter10,WebDevelopmentDoneRight,we’vebeenabletoconcentratemoreonthecodeinthischapter.We’veseenhowusefulDjangoclass-basedviewsare,andweeventouchedonsomecoolJavaScript.Run$pythonmanage.pyrunserverandyourinterfaceshouldbeupandrunningathttp://127.0.0.1:8000.

NoteIfyouarewondering,127.0.0.1meansthelocalhost—yourcomputer—while8000istheporttowhichtheserverisbound,tolistenforincomingrequests.

Nowit’stimetospicethingsupabitwiththesecondpartofthisproject.

ImplementingtheFalconAPIThestructureoftheFalconprojectwe’reabouttocodeisnowherenearasextendedastheinterfaceone.We’llcodefivefilesaltogether.Inyourch12folder,createanewonecalledpwdapi.Thisisitsfinalstructure:

$tree-Apwdapi/

pwdapi/

├──core

│├──handlers.py

│└──passwords.py

├──main.py

└──tests

└──test_core

├──test_handlers.py

└──test_passwords.py

TheAPIwasallcodedusingTDD,sowe’realsogoingtoexplorethetests.However,Ithinkit’sgoingtobeeasierforyoutounderstandthetestsifyoufirstseethecode,sowe’regoingtostartwiththat.

ThemainapplicationThisisthecodefortheFalconapplication:main.py

importfalcon

fromcore.handlersimport(

PasswordValidatorHandler,

PasswordGeneratorHandler,

)

validation_handler=PasswordValidatorHandler()

generator_handler=PasswordGeneratorHandler()

app=falcon.API()

app.add_route('/password/validate/',validation_handler)

app.add_route('/password/generate/',generator_handler)

AsintheexampleinChapter10,WebDevelopmentDoneRight,westartbycreatingoneinstanceforeachofthehandlersweneed,thenwecreateafalcon.APIobjectand,bycallingitsadd_routemethod,wesetuptheroutingtotheURLsofourAPI.We’llgettothedefinitionsofthehandlersinamoment.Firstly,weneedacoupleofhelpers.

WritingthehelpersInthissection,wewilltakealookatacoupleofclassesthatwe’lluseinourhandlers.It’salwaysgoodtofactoroutsomelogicfollowingtheSingleResponsibilityPrinciple.

NoteInOOP,theSingleResponsibilityPrinciple(SRP)statesthateveryclassshouldhaveresponsibilityforasinglepartofthefunctionalityprovidedbythesoftware,andthatresponsibilityshouldbeentirelyencapsulatedbytheclass.Allofitsservicesshouldbenarrowlyalignedwiththatresponsibility.

TheSingleResponsibilityPrincipleistheSinS.O.L.I.D.,anacronymforthefirstfiveOOPandsoftwaredesignprinciplesintroducedbyRobertMartin.

Iheartilysuggestyoutoopenabrowserandreaduponthissubject,itisveryimportant.

Allthecodeinthehelperssectionbelongstothecore/passwords.pymodule.Here’showitbegins:

frommathimportceil

fromrandomimportsample

fromstringimportascii_lowercase,ascii_uppercase,digits

punctuation='!#$%&()*+-?@_|'

allchars=''.join(

(ascii_lowercase,ascii_uppercase,digits,punctuation))

We’llneedtohandlesomerandomizedcalculationsbutthemostimportantparthereistheallowedcharacters.Wewillallowletters,digits,andasetofpunctuationcharacters.Toeasewritingthecode,wewillmergethosepartsintotheallcharsstring.

CodingthepasswordvalidatorThePasswordValidatorclassismyfavoritebitoflogicinthewholeAPI.Itexposesanis_validandascoremethod.Thelatterrunsalldefinedvalidators(“private”methodsinthesameclass),andcollectsthescoresintoasingledictwhichisreturnedasaresult.I’llwritethisclassmethodbymethodsothatitdoesnotgettoocomplicated:

classPasswordValidator:

def__init__(self,password):

self.password=password.strip()

Itbeginsbysettingpassword(withnoleadingortrailingspaces)asaninstanceattribute.Thiswaywewon’tthenhavetopassitaroundfrommethodtomethod.Allthemethodsthatwillfollowbelongtothisclass.

defis_valid(self):

return(len(self.password)>0and

all(charinallcharsforcharinself.password))

Apasswordisvalidwhenitslengthisgreaterthan0andallofitscharactersbelongtotheallcharsstring.Whenyoureadtheis_validmethod,it’spracticallyEnglish(that’show

amazingPythonis).allisabuilt-infunctionthattellsyouifalltheelementsoftheiterableyoufeedtoitareTrue.

defscore(self):

result={

'length':self._score_length(),

'case':self._score_case(),

'numbers':self._score_numbers(),

'special':self._score_special(),

'ratio':self._score_ratio(),

}

result['total']=sum(result.values())

returnresult

Thisistheothermainmethod.It’sverysimple,itjustpreparesadictwithalltheresultsfromthevalidators.Theonlyindependentbitoflogichappensattheend,whenwesumthegradesfromeachvalidatorandassignittoa'total'keyinthedict,justforconvenience.

Asyoucansee,wescoreapasswordbylength,bylettercase,bythepresenceofnumbers,andspecialcharacters,and,finally,bytheratiobetweenlettersandnumbers.Lettersallowacharactertobebetween26*2=52differentpossiblechoices,whiledigitsallowonly10.Therefore,passwordswhoseletterstodigitsratioishigheraremoredifficulttocrack.

Let’sseethelengthvalidator:

def_score_length(self):

scores_list=([0]*4)+([1]*4)+([3]*4)+([5]*4)

scores=dict(enumerate(scores_list))

returnscores.get(len(self.password),7)

Weassign0pointstopasswordswhoselengthislessthanfourcharacters,1pointforthosewhoselengthislessthan8,3foralengthlessthan12,5foralengthlessthan16,and7foralengthof16ormore.

Inordertoavoidawaterfallofif/elifclauses,Ihaveadoptedafunctionalstylehere.Ipreparedascore_list,whichisbasically[0,0,0,0,1,1,1,1,3,...].Then,byenumeratingit,Igota(length,score)pairforeachlengthlessthan16.Iputthosepairsintoadict,whichgivesmetheequivalentindictform,soitshouldlooklikethis:{0:0,1:0,2:0,3:0,4:1,5:1,...}.Ithenperformagetonthisdictwiththelengthofthepassword,settingavalueof7asthedefault(whichwillbereturnedforlengthsof16ormore,whicharenotinthedict).

Ihavenothingagainstif/elifclauses,ofcourse,butIwantedtotaketheopportunitytoshowyoudifferentcodingstylesinthisfinalchapter,tohelpyougetusedtoreadingcodewhichdeviatesfromwhatyouwouldnormallyexpect.It’sonlybeneficial.

def_score_case(self):

lower=bool(set(ascii_lowercase)&set(self.password))

upper=bool(set(ascii_uppercase)&set(self.password))

returnint(lowerorupper)+2*(lowerandupper)

Thewaywevalidatethecaseisagainwithanicetrick.lowerisTruewhenthe

intersectionbetweenthepasswordandalllowercasecharactersisnon-empty,otherwiseit’sFalse.upperbehavesinthesameway,onlywithuppercasecharacters.

Tounderstandtheevaluationthathappensonthelastline,let’susetheinside-outtechniqueoncemore:lowerorupperisTruewhenatleastoneofthetwoisTrue.Whenit’sTrue,itwillbeconvertedtoa1bytheintclass.Thisequatestosaying,ifthereisatleastonecharacter,regardlessofthecasing,thescoregets1point,otherwiseitstaysat0.

Nowforthesecondpart:lowerandupperisTruewhenbothofthemareTrue,whichmeansthatwehaveatleastonelowercaseandoneuppercasecharacter.Thismeansthat,tocrackthepassword,abrute-forcealgorithmwouldhavetoloopthrough52lettersinsteadofjust26.Therefore,whenthat’sTrue,wegetanextratwopoints.

Thisvalidatorthereforeproducesaresultintherange(0,1,3),dependingonwhatthepasswordis.

def_score_numbers(self):

return2if(set(self.password)&set(digits))else0

Scoringonthenumbersissimpler.Ifwehaveatleastonenumber,wegettwopoints,otherwiseweget0.Inthiscase,Iusedaternaryoperatortoreturntheresult.

def_score_special(self):

return4if(

set(self.password)&set(punctuation))else0

Thespecialcharactersvalidatorhasthesamelogicasthepreviousonebut,sincespecialcharactersaddquiteabitofcomplexitywhenitcomestocrackingapassword,wehavescoredfourpointsinsteadofjusttwo.

Thelastonevalidatestheratiobetweenthelettersandthedigits.

def_score_ratio(self):

alpha_count=sum(

1ifc.lower()inascii_lowercaseelse0

forcinself.password)

digits_count=sum(

1ifcindigitselse0forcinself.password)

ifdigits_count==0:

return0

returnmin(ceil(alpha_count/digits_count),7)

Ihighlightedtheconditionallogicintheexpressionsinthesumcalls.Inthefirstcase,wegeta1foreachcharacterwhoselowercaseversionisinascii_lowercase.Thismeansthatsummingallthose1’supgivesusexactlythecountofalltheletters.Then,wedothesameforthedigits,onlyweusethedigitsstringforreference,andwedon’tneedtolowercasethecharacter.Whendigits_countis0,alpha_count/digits_countwouldcauseaZeroDivisionError,thereforewecheckondigits_countandwhenit’s0wereturn0.Ifwehavedigits,wecalculatetheceilingoftheletters:digitsratio,andreturnit,cappedat7.

Ofcourse,therearemanydifferentwaystocalculateascoreforapassword.Myaimhereisnottogiveyouthefinestalgorithmtodothat,buttoshowyouhowyoucouldgoabout

implementingit.

CodingthepasswordgeneratorThepasswordgeneratorisamuchsimplerclassthanthevalidator.However,Ihavecodeditsothatwewon’tneedtocreateaninstancetouseit,justtoshowyouyetagainadifferentcodingstyle.

classPasswordGenerator:

@classmethod

defgenerate(cls,length,bestof=10):

candidates=sorted([

cls._generate_candidate(length)

forkinrange(max(1,bestof))

])

returncandidates[-1]

@classmethod

def_generate_candidate(cls,length):

password=cls._generate_password(length)

score=PasswordValidator(password).score()

return(score['total'],password)

@classmethod

def_generate_password(cls,length):

chars=allchars*(ceil(length/len(allchars)))

return''.join(sample(chars,length))

Ofthethreemethods,onlythefirstoneismeanttobeused.Let’sstartouranalysiswiththelastone:_generate_password.

Thismethodsimplytakesalength,whichisthedesiredlengthforthepasswordwewant,andcallsthesamplefunctiontogetapopulationoflengthelementsoutofthecharsstring.Thereturnvalueofthesamplefunctionisalistoflengthelements,andweneedtomakeitastringusingjoin.

Beforewecancallsample,thinkaboutthis,whatifthedesiredlengthexceedsthelengthofallchars?ThecallwouldresultinValueError:Samplelargerthanthepopulation.

Becauseofthis,wecreatethecharsstringinawaythatitismadebyconcatenatingtheallcharsstringtoitselfjustenoughtimestocoverthedesiredlength.Togiveyouanexample,let’ssayweneedapasswordof27characters,andlet’spretendallcharsis10characterslong.length/len(allchars)gives2.7,which,whenpassedtotheceilfunction,becomes3.Thismeansthatwe’regoingtoassigncharstoatripleconcatenationoftheallcharsstring,hencecharswillbe10*3=30characterslong,whichisenoughtocoverourrequirements.

Notethat,inorderforthesemethodstobecalledwithoutcreatinganinstanceofthisclass,weneedtodecoratethemwiththeclassmethoddecorator.Theconventionisthentocallthefirstargument,cls,insteadofself,becausePython,behindthescenes,willpasstheclassobjecttothecall.

Thecodefor_generate_candidateisalsoverysimple.Wejustgenerateapasswordand,giventhelength,wecalculateitsscore,andreturnatuple(score,password).

Wedothissothatinthegeneratemethodwecangenerate10(bydefault)passwordseachtimethemethodiscalledandreturntheonethathasthehighestscore.Sinceourgenerationlogicisbasedonarandomfunction,it’salwaysagoodwaytoemployatechniquelikethistoavoidworstcasescenarios.

Thisconcludesthecodeforthehelpers.

WritingthehandlersAsyoumayhavenoticed,thecodeforthehelpersisn’trelatedtoFalconatall.ItisjustpurePythonthatwecanreusewhenweneedit.Ontheotherhand,thecodeforthehandlersisofcoursebasedonFalcon.Thecodethatfollowsbelongstothecore/handlers.pymoduleso,aswedidbefore,let’sstartwiththefirstfewlines:

importjson

importfalcon

from.passwordsimportPasswordValidator,PasswordGenerator

classHeaderMixin:

defset_access_control_allow_origin(self,resp):

resp.set_header('Access-Control-Allow-Origin','*')

Thatwasverysimple.Weimportjson,falcon,andourhelpers,andthenwesetupamixinwhichwe’llneedinbothhandlers.TheneedforthismixinistoallowtheAPItoserverequeststhatcomefromsomewhereelse.ThisistheothersideoftheCORScointowhatwesawintheJavaScriptcodefortheinterface.Inthiscase,weboldlygowherenosecurityexpertwouldeverdare,andallowrequeststocomefromanydomain('*').Wedothisbecausethisisanexerciseand,inthiscontext,itisfine,butdon’tdoitinproduction,okay?

CodingthepasswordvalidatorhandlerThishandlerwillhavetorespondtoaPOSTrequest,thereforeIhavecodedanon_postmethod,whichisthewayyoureacttoaPOSTrequestinFalcon.

classPasswordValidatorHandler(HeaderMixin):

defon_post(self,req,resp):

self.process_request(req,resp)

password=req.context.get('_body',{}).get('password')

ifpasswordisNone:

resp.status=falcon.HTTP_BAD_REQUEST

returnNone

result=self.parse_password(password)

resp.body=json.dumps(result)

defparse_password(self,password):

validator=PasswordValidator(password)

return{

'password':password,

'valid':validator.is_valid(),

'score':validator.score(),

}

defprocess_request(self,req,resp):

self.set_access_control_allow_origin(resp)

body=req.stream.read()

ifnotbody:

raisefalcon.HTTPBadRequest('Emptyrequestbody',

'AvalidJSONdocumentisrequired.')

try:

req.context['_body']=json.loads(

body.decode('utf-8'))

except(ValueError,UnicodeDecodeError):

raisefalcon.HTTPError(

falcon.HTTP_753,'MalformedJSON',

'JSONincorrectornotutf-8encoded.')

Let’sstartwiththeon_postmethod.Firstofall,wecalltheprocess_requestmethod,whichdoesasanitycheckontherequestbody.Iwon’tgointofinestdetailbecauseit’stakenfromtheFalcondocumentation,andit’sastandardwayofprocessingarequest.Let’sjustsaythat,ifeverythinggoeswell(thehighlightedpart),wegetthebodyoftherequest(alreadydecodedfromJSON)inreq.context['_body'].Ifthingsgobadlyforanyreason,wereturnanappropriateerrorresponse.

Let’sgobacktoon_post.Wefetchthepasswordfromtherequestcontext.Atthispoint,process_requesthassucceeded,butwestilldon’tknowifthebodywasinthecorrectformat.We’reexpectingsomethinglike:{'password':'my_password'}.

Soweproceedwithcaution.Wegetthevalueforthe'_body'keyand,ifthatisnotpresent,wereturnanemptydict.Wegetthevaluefor'password'fromthat.WeusegetinsteadofdirectaccesstoavoidKeyErrorissues.

IfthepasswordisNone,wesimplyreturna400error(badrequest).Otherwise,wevalidateitandcalculateitsscore,andthensettheresultasthebodyofourresponse.

Youcanseehoweasyitistovalidateandcalculatethescoreofthepasswordintheparse_passwordmethod,byusingourhelpers.

Wereturnadictwiththreepiecesofinformation:password,valid,andscore.Thepasswordinformationistechnicallyredundantbecausewhoevermadetherequestwouldknowthepasswordbut,inthiscase,Ithinkit’sagoodwayofprovidingenoughinformationforthingssuchaslogging,soIaddedit.

WhathappensiftheJSON-decodedbodyisnotadict?Iwillleaveituptoyoutofixthecode,addingsomelogictocaterforthatedgecase.

CodingthepasswordgeneratorhandlerThegeneratorhandlerhastohandleaGETrequestwithonequeryparameter:thedesiredpasswordlength.

classPasswordGeneratorHandler(HeaderMixin):

defon_get(self,req,resp):

self.process_request(req,resp)

length=req.context.get('_length',16)

resp.body=json.dumps(

PasswordGenerator.generate(length))

defprocess_request(self,req,resp):

self.set_access_control_allow_origin(resp)

length=req.get_param('length')

iflengthisNone:

return

try:

length=int(length)

assertlength>0

req.context['_length']=length

except(ValueError,TypeError,AssertionError):

raisefalcon.HTTPBadRequest('Wrongqueryparameter',

'`length`mustbeapositiveinteger.')

Wehaveasimilarprocess_requestmethod.Itdoesasanitycheckontherequest,eventhoughabitdifferentlyfromtheprevioushandler.Thistime,weneedtomakesurethatifthelengthisprovidedonthequerystring(whichmeans,forexample,http://our-api-url/?length=23),it’sinthecorrectformat.Thismeansthatlengthneedstobeapositiveinteger.

So,tovalidatethat,wedoanintconversion(req.get_param('length')returnsastring),thenweassertthatlengthisgreaterthanzeroand,finally,weputitincontextunderthe'_length'key.

DoingtheintconversionofastringwhichisnotasuitablerepresentationforanintegerraisesValueError,whileaconversionfromatypethatisnotastringraisesTypeError,thereforewecatchthosetwointheexceptclause.

WealsocatchAssertionError,whichisraisedbytheassertlength>0linewhenlengthisnotapositiveinteger.Wecanthensafelyguaranteethatthelengthisasdesiredwithonesingletry/exceptblock.

TipNotethat,whencodingatry/exceptblock,youshouldusuallytryandbeasspecificaspossible,separatinginstructionsthatwouldraisedifferentexceptionsifaproblemarose.Thiswouldallowyoumorecontrolovertheissue,andeasierdebugging.Inthiscasethough,sincethisisasimpleAPI,it’sfinetohavecodewhichonlyreactstoarequestforwhichlengthisnotintherightformat.

Thecodefortheon_getmethodisquitestraightforward.Itstartsbyprocessingtherequest,thenthelengthisfetched,fallingbackto16(thedefaultvalue)whenit’snotpassed,andthenapasswordisgeneratedanddumpedtoJSON,andthensettobethebodyoftheresponse.

RunningtheAPIInordertorunthisapplication,youneedtorememberthatwesetthebaseURLintheinterfacetohttp://127.0.0.1:5555.Therefore,weneedthefollowingcommandtostarttheAPI:

$gunicorn-b127.0.0.1:5555main:app

Runningthatwillstarttheappdefinedinthemainmodule,bindingtheserverinstancetoport5555onlocalhost.FormoreinformationaboutGunicorn,pleaserefertoeitherChapter10,WebDevelopmentDoneRightordirectlytotheproject’shomepage(http://gunicorn.org/).

ThecodefortheAPIisnowcompletesoifyouhaveboththeinterfaceandtheAPIrunning,youcantrythemouttogether.Seeifeverythingworksasexpected.

TestingtheAPIInthissection,let’stakealookatthetestsIwroteforthehelpersandforthehandlers.Testsforthehelpersareheavilybasedonthenose_parameterizedlibrary,asmyfavoritetestingstyleisinterfacetesting,withaslittlepatchingaspossible.Usingnose_parameterizedallowsmetowriteteststhatareeasiertoreadbecausethetestcasesareveryvisible.

Ontheotherhand,testsforthehandlershavetofollowthetestingconventionsfortheFalconlibrary,sotheywillbeabitdifferent.Thisis,ofcourse,idealsinceitallowsmetoshowyouevenmore.

DuetothelimitedamountofpagesIhaveleft,I’llshowyouonlyapartofthetests,somakesureyoucheckthemoutinfullinthesourcecode.

TestingthehelpersLet’sseethetestsforthePasswordGeneratorclass:tests/test_core/test_passwords.py

classPasswordGeneratorTestCase(TestCase):

deftest__generate_password_length(self):

forlengthinrange(300):

assert_equal(

length,

len(PasswordGenerator._generate_password(length))

)

deftest__generate_password_validity(self):

forlengthinrange(1,300):

password=PasswordGenerator._generate_password(

length)

assert_true(PasswordValidator(password).is_valid())

deftest__generate_candidate(self):

score,password=(

PasswordGenerator._generate_candidate(42))

expected_score=PasswordValidator(password).score()

assert_equal(expected_score['total'],score)

@patch.object(PasswordGenerator,'_generate_candidate')

deftest__generate(self,_generate_candidate_mock):

#checks`generate`returnsthehighestscorecandidate

_generate_candidate_mock.side_effect=[

(16,'&a69Ly+0H4jZ'),

(17,'UXaF4stRfdlh'),

(21,'aB4Ge_KdTgwR'),#thewinner

(12,'IRLT*XEfcglm'),

(16,'$P92-WZ5+DnG'),

(18,'Xi#36jcKA_qQ'),

(19,'?p9avQzRMIK0'),

(17,'4@sY&bQ9*H!+'),

(12,'Cx-QAYXG_Ejq'),

(18,'C)RAV(HP7j9n'),

]

assert_equal(

(21,'aB4Ge_KdTgwR'),PasswordGenerator.generate(12))

Withintest__generate_password_lengthwemakesurethe_generate_passwordmethodhandlesthelengthparametercorrectly.Wegenerateapasswordforeachlengthintherange[0,300),andverifythatithasthecorrectlength.

Inthetest__generate_password_validitytest,wedosomethingsimilarbut,thistime,wemakesurethatwhateverlengthweaskfor,thegeneratedpasswordisvalid.WeusethePasswordValidatorclasstocheckforvalidity.

Finally,weneedtotestthegeneratemethod.Thepasswordgenerationisrandom,therefore,inordertotestthisfunction,weneedtomock_generate_candidate,thuscontrollingitsoutput.Wesettheside_effectargumentonitsmocktobealistof10candidates,fromwhichweexpectthegeneratemethodtochoosetheonewiththehighestscore.Settingside_effectonamocktoalistcausesthatmocktoreturntheelementsofthatlist,oneatatime,eachtimeit’scalled.Toavoidambiguity,thehighestscoreis21,andonlyonecandidatehasscoredthathigh.Wecallthemethodandmakesurethatthatparticularoneisthecandidatewhichisreturned.

NoteIfyouarewonderingwhyIusedthosedoubleunderscoresinthetestnames,it’sverysimple:thefirstoneisaseparatorandthesecondoneistheleadingunderscorethatispartofthenameofthemethodundertest.

TestingthePasswordValidatorclassrequiresmanymorelinesofcode,soI’llshowonlyaportionofthesetests:pwdapi/tests/test_core/test_passwords.py

fromunittestimportTestCase

fromunittest.mockimportpatch

fromnose_parameterizedimportparameterized,param

fromnose.toolsimport(

assert_equal,assert_dict_equal,assert_true)

fromcore.passwordsimportPasswordValidator,PasswordGenerator

classPasswordValidatorTestCase(TestCase):

@parameterized.expand([

(False,''),

(False,''),

(True,'abcdefghijklmnopqrstuvwxyz'),

(True,'ABCDEFGHIJKLMNOPQRSTUVWXYZ'),

(True,'0123456789'),

(True,'!#$%&()*+-?@_|'),

])

deftest_is_valid(self,valid,password):

validator=PasswordValidator(password)

assert_equal(valid,validator.is_valid())

Westartbytestingtheis_validmethod.WetestwhetherornotitreturnsFalsewhenit’sfedanemptystring,aswellasastringmadeupofonlyspaces,whichmakessurewe’retestingwhetherwe’recalling.strip()whenweassignthepassword.

Then,weuseallthecharactersthatwewanttobeacceptedtomakesurethefunctionacceptsthem.

Iunderstandthesyntaxbehindtheparameterize.expanddecoratorcanbechallengingatfirstbutreally,allthereistoitisthateachtupleconsistsofanindependenttestcasewhich,inturn,meansthatthetest_is_validtestisrunindividuallyforeachtuple,andthatthetwotupleelementsarepassedtothemethodasarguments:validandpassword.

Wethentestforinvalidcharacters.Weexpectthemalltofailsoweuseparam.explicit,whichrunsthetestforeachofthecharactersinthatweirdstring.

@parameterized.expand(

param.explicit(char)forcharin'>]{<`\\;,[^/"\'~:}=.'

)

deftest_is_valid_invalid_chars(self,password):

validator=PasswordValidator(password)

assert_equal(False,validator.is_valid())

TheyallevaluatetoFalse,sowe’regood.

@parameterized.expand([

(0,''),#0-3:score0

(0,'a'),#0-3:score0

(0,'aa'),#0-3:score0

(0,'aaa'),#0-3:score0

(1,'aaab'),#4-7:score1

...

(5,'aaabbbbccccddd'),#12-15:score5

(5,'aaabbbbccccdddd'),#12-15:score5

])

deftest__score_length(self,score,password):

validator=PasswordValidator(password)

assert_equal(score,validator._score_length())

Totestthe_score_lengthmethod,Icreated16testcasesforthelengthsfrom0to15.Thebodyofthetestsimplymakessurethatthescoreisassignedappropriately.

deftest__score_length_sixteen_plus(self):

#allpasswordwhoselengthis16+score7points

password='x'*255

forlengthinrange(16,len(password)):

validator=PasswordValidator(password[:length])

assert_equal(7,validator._score_length())

Theprecedingtestisforlengthsfrom16to254.Weonlyneedtomakesurethatanylengthafter15gets7asascore.

Iwillskipoverthetestsfortheotherinternalmethodsandjumpdirectlytotheoneforthescoremethod.Inordertotestit,Iwanttocontrolexactlywhatisreturnedbyeachofthe_score_*methodssoImockthemoutandinthetest,Isetareturnvalueforeachofthem.

Notethattomockmethodsofaclass,weuseavariantofpatch:patch.object.Whenyousetreturnvaluesonmocks,it’snevergoodtohaverepetitionsbecauseyoumaynotbesurewhichmethodreturnedwhat,andthetestwouldn’tfailinthecaseofaswap.So,alwaysreturndifferentvalues.Inmycase,Iamusingthefirstfewprimenumberstobesurethereisnopossibilityofconfusion.

@patch.object(PasswordValidator,'_score_length')

@patch.object(PasswordValidator,'_score_case')

@patch.object(PasswordValidator,'_score_numbers')

@patch.object(PasswordValidator,'_score_special')

@patch.object(PasswordValidator,'_score_ratio')

deftest_score(

self,

_score_ratio_mock,

_score_special_mock,

_score_numbers_mock,

_score_case_mock,

_score_length_mock):

_score_ratio_mock.return_value=2

_score_special_mock.return_value=3

_score_numbers_mock.return_value=5

_score_case_mock.return_value=7

_score_length_mock.return_value=11

expected_result={

'length':11,

'case':7,

'numbers':5,

'special':3,

'ratio':2,

'total':28,

}

validator=PasswordValidator('')

assert_dict_equal(expected_result,validator.score())

Iwanttopointoutexplicitlythatthe_score_*methodsaremocked,soIsetupmyvalidatorinstancebypassinganemptystringtotheclassconstructor.Thismakesitevenmoreevidenttothereaderthattheinternalsoftheclasshavebeenmockedout.Then,IjustcheckiftheresultisthesameaswhatIwasexpecting.

ThislasttestistheonlyoneinthisclassinwhichIusedmocks.Alltheothertestsforthe_score_*methodsareinaninterfacestyle,whichreadsbetterandusuallyproducesbetterresults.

TestingthehandlersLet’sbrieflyseeoneexampleofatestforahandler:pwdapi/tests/test_core/test_handlers.py

importjson

fromunittest.mockimportpatch

fromnose.toolsimportassert_dict_equal,assert_equal

importfalcon

importfalcon.testingastesting

fromcore.handlersimport(

PasswordValidatorHandler,PasswordGeneratorHandler)

classPGHTest(PasswordGeneratorHandler):

defprocess_request(self,req,resp):

self.req,self.resp=req,resp

returnsuper(PGHTest,self).process_request(req,resp)

classPVHTest(PasswordValidatorHandler):

defprocess_request(self,req,resp):

self.req,self.resp=req,resp

returnsuper(PVHTest,self).process_request(req,resp)

BecauseofthetoolsFalcongivesyoutotestyourhandlers,IcreatedachildforeachoftheclassesIwantedtotest.TheonlythingIchanged(byoverridingamethod)isthatintheprocess_requestmethod,whichiscalledbybothclasses,beforeprocessingtherequestImakesureIsetthereqandrespargumentsontheinstance.Thenormalbehavioroftheprocess_requestmethodisthusnotalteredinanyotherway.Bydoingthis,whateverhappensoverthecourseofthetest,I’llbeabletocheckagainstthoseobjects.

It’squitecommontousetrickslikethiswhentesting.Weneverchangethecodetoadaptforatest,itwouldbebadpractice.Wefindawayofadaptingourteststosuitourneeds.

classTestPasswordValidatorHandler(testing.TestBase):

defbefore(self):

self.resource=PVHTest()

self.api.add_route('/password/validate/',self.resource)

ThebeforemethodiscalledbytheFalconTestBaselogic,anditallowsustosetuptheresourcewewanttotest(thehandler)andarouteforit(whichisnotnecessarilythesameastheoneweuseinproduction).

deftest_post(self):

self.simulate_request(

'/password/validate/',

body=json.dumps({'password':'abcABC0123#&'}),

method='POST')

resp=self.resource.resp

assert_equal('200OK',resp.status)

assert_dict_equal(

{'password':'abcABC0123#&',

'score':{'case':3,'length':5,'numbers':2,

'special':4,'ratio':2,'total':16},

'valid':True},

json.loads(resp.body))

Thisisthetestforthehappypath.AllitdoesissimulateaPOSTrequestwithaJSONpayloadasbody.Then,weinspecttheresponseobject.Inparticular,weinspectitsstatusanditsbody.Wemakesurethatthehandlerhascorrectlycalledthevalidatorandreturneditsresults.

Wealsotestthegeneratorhandler:

classTestPasswordGeneratorHandler(testing.TestBase):

defbefore(self):

self.resource=PGHTest()

self.api.add_route('/password/generate/',self.resource)

@patch('core.handlers.PasswordGenerator')

deftest_get(self,PasswordGenerator):

PasswordGenerator.generate.return_value=(7,'abc123')

self.simulate_request(

'/password/generate/',

query_string='length=7',

method='GET')

resp=self.resource.resp

assert_equal('200OK',resp.status)

assert_equal([7,'abc123'],json.loads(resp.body))

Forthisoneaswell,Iwillonlyshowyouthetestforthehappypath.WemockoutthePasswordGeneratorclassbecauseweneedtocontrolwhichpassworditwillgenerateand,unlesswemock,wewon’tbeabletodoit,asitisarandomprocess.

Oncewehavecorrectlysetupitsreturnvalue,wecansimulatetherequestagain.Inthiscase,it’saGETrequest,withadesiredlengthof7.Weuseatechniquesimilartotheoneweusedfortheotherhandler,andchecktheresponsestatusandbody.

ThesearenottheonlytestsyoucouldwriteagainsttheAPI,andthestylecouldbedifferentaswell.Somepeoplemockoften,ItendtomockonlywhenIreallyhaveto.Justtrytoseeifyoucanmakesomesenseoutofthem.Iknowthey’renotreallyeasybutthey’llbegoodtrainingforyou.Testsareextremelyimportantsogiveityourbestshot.

Wheredoyougofromhere?Ifyoulikedthisprojectandyoufeellikeexpandingit,hereareafewsuggestions:

ImplementtheencryptioninthemechanismofacustomDjangofield.Amendthetemplatefortherecordlistsothatyoucansearchforaparticularrecord.AmendtheJavaScripttouseJSONPwithacallbacktoovercometheCORSissue.AmendtheJavaScripttofirethevalidationcallwhenthepasswordfieldchanges.WriteaDjangocommandthatallowsyoutoencryptanddecryptthedatabasefile.Whenyoudoitfromthecommandline,incorporatethatbehaviorintothewebsite,possiblyonthehomepage,sothatyoudon’thaveaccesstotherecordsunlessyouareauthenticated.Thisisdefinitelyahardchallengeasitrequireseitheranotherdatabasewithanauthenticationpasswordstoredproperlywithaonewayhash,orsomeseriousreworkingofthedatastructureusedtoholdtherecordmodeldata.Evenifyoudon’thavethemeanstodoitnow,tryandthinkabouthowyouwouldsolvethisproblem.SetupPostgreSQLonyourmachineandswitchtousingitinsteadoftheSQLitefilethatisthedefault.Addtheabilitytoattachafiletoarecord.Playwiththeapplication,trytofindoutwhichfeaturesyouwanttoaddorchange,andthendoit.

SummaryInthischapter,we’veworkedonafinalprojectthatinvolvesaninterfaceandanAPI.Wehaveusedtwodifferentframeworkstoaccomplishourgoal:DjangoandFalcon.Theyareverydifferentandhaveallowedustoexploredifferentconceptsandtechniquestocraftoursoftwareandmakethisfunapplicationcomealive.

Wehaveseenanexampleofsymmetricencryptionandexploredcodethatwaswritteninamorefunctionalstyle,asopposedtoamoreclassiccontrolflow-orientedapproach.WehavereusedandextendedtheDjangoclass-basedviews,reducingtoaminimumtheamountofcodewehadtowrite.

WhencodingtheAPI,wedecoupledhandlingrequestsfrompasswordmanagement.Thiswayit’smucheasiertoseewhichpartofthecodedependsontheFalconframeworkandwhichisindependentfromit.

Finally,wesawafewtestsforthehelpersandhandlersoftheAPI.WehavebrieflytouchedonatechniquethatIusetoexpandclassesundertestinordertobeabletotestagainstthosepartsofthecodewhichwouldnotnormallybeavailable.

Myaiminthischapterwastoprovideyouwithaninterestingexamplethatcouldbeexpandedandimprovedindifferentways.Ialsowantedtogiveyouafewexamplesofdifferentcodingstylesandtechniques,whichiswhyIchosetospreadthingsapartandusedifferentframeworks.

AwordoffarewellIhopethatyouarestillthirstyandthatthisbookwillbejustthefirstofmanystepsyoutaketowardsPython.It’satrulywonderfullanguage,wellworthlearningdeeply.

Ihopethatyouenjoyedthisjourneywithme,Ididmybesttomakeitinterestingforyou.Itsurewasforme,Ihadsuchagreattimewritingthesepages.

Pythonisopensource,sopleasekeepsharingitandconsidersupportingthewonderfulcommunityaroundit.

Tillnexttime,myfriend,farewell!

IndexA

adhocpolymorphismabout/Polymorphism–abriefoverview

AJAXabout/Thefutureofwebdevelopment,Thetemplatelayer

AnacondaURL/Wheredowegofromhere?

anonymousfunctionsabout/Anonymousfunctions

APItesting/TestingtheAPIhelpers,testing/Testingthehelpershandlers,testing/Testingthehandlers

applicationtesting/Testingyourapplication

applicationprogramminginterface(API)about/Writingaunittest

assertionabout/Assertions

BBase64

about/Theimportsbaseclass

about/Inheritanceandcompositionbinarysearch

about/WheretoinspectBitbucket

URL/Exceptionsblack-boxtests

about/TestingyourapplicationBokeh

URL/Wheredowegofromhere?boundaries

about/Aspecializedelse:elifboundary

about/Boundariesandgranularitybuilt-inexceptionshierarchy

URL/Exceptionsbuilt-infunctions

about/Built-infunctionsbuilt-inscope

about/Scopesbusinesslogic,GUIapplication

webpage,fetching/Thebusinesslogic,Fetchingthewebpageimages,saving/Savingtheimagesuser,alerting/Alertingtheuser

Ccallback

about/Thelayoutlogiccampaign

about/Preparingthedataclasses

about/Objectandclasses,Object-orientedprogrammingclassmethods

about/Classmethodscode

writing,guidelines/Guidelinesonhowtowritegoodcodedocumenting/Documentingyourcode

collectionsmoduleabout/Thecollectionsmodulenamedtuples/Namedtuplesdefaultdict/DefaultdictChainMap/ChainMap

comma-separatedvalues(CSV)about/SavingtheDataFrametoafile

commandpromptabout/SettingupthePythoninterpreter

compositionabout/Inheritanceandcomposition

comprehensionsabout/map,zip,andfilter,Comprehensionsnestedcomprehensions/Nestedcomprehensionsfiltering/Filteringacomprehensiondictcomprehensions/dictcomprehensionssetcomprehensions/setcomprehensionsandgenerators/Don’toverdocomprehensionsandgenerators

conditionalprogrammingabout/Conditionalprogrammingelif/Aspecializedelse:elifternaryoperator/Theternaryoperator

considerationsthreading/Threadingconsiderations

consoleabout/SettingupthePythoninterpreter,Yourfriend,theconsole

constructorabout/Initializinganinstance

containerdatatypesdefining/Thecollectionsmodule

contextmanager

using/Thebusinesslogiccontextobject

preparing/Thetemplatelayercookies

about/HowdoestheWebwork?CPC(CostPerClick)

about/UnpackingtheuserdataCPI(CostPerImpression)

about/UnpackingtheuserdataCron

about/RunningPythonscriptsCross-originresourcesharing(CORS)

about/TalkingtotheAPIcross-siterequestforgery(CSRF)attack

about/CreatingtheformCSS(CascadingStyleSheets)

about/AregexwebsiteCTR(ClickThroughRate)

about/Unpackingtheuserdatacustomexceptions

writing/Exceptionscustomiterator

writing/Writingacustomiteratorcypertext

about/Themodellayer

Ddata

dealingwith/Dealingwithdatanotebook,settingup/Settingupthenotebookpreparing/Preparingthedatacleaning/CleaningthedataDataFrame,creating/CreatingtheDataFrameDataFrame,savingtofile/SavingtheDataFrametoafileresults,visualizing/Visualizingtheresults

DataFramecampaignname,unpacking/Unpackingthecampaignnameuserdata,unpacking/Unpackingtheuserdatadefining/Cleaningeverythingup

datamigrationsabout/AddingtheEntrymodel

datastructuresselecting/Howtochoosedatastructures

debuggingtechniquesabout/Debuggingtechniquesdebugging,withprint/Debuggingwithprintdebugging,withcustomfunction/Debuggingwithacustomfunctiontraceback,inspecting/InspectingthetracebackPythondebugger,using/UsingthePythondebuggerlogfiles,inspecting/Inspectinglogfilesprofiling/Profilingassertions/Assertionsinformation,finding/Wheretofindinformation

decorate-sort-undecorateabout/mapURL/map

decorationabout/Decorators

decorationpointabout/Decorators

decoratorabout/Decorators

decoratorfactoryabout/Adecoratorfactory

decoratorsabout/Decorators

defaultvaluesabout/Keywordargumentsanddefaultvalues

deterministprofiling

about/ProfilingPythondiamond

about/Multipleinheritancedictionaries

about/Mappingtypes–dictionariesdiscounts

applying/Example2–applyingdiscountsdispatcher

about/Example2–applyingdiscountsDjango

settingup/SettingupDjangoURL/SettingupDjangoproject,starting/Startingtheprojectusers,creating/Creatingusersabout/Writingthetemplates

Djangointerfaceimplementing/Ourimplementation,ImplementingtheDjangointerfacesetup/Thesetupmodellayer/Themodellayersimpleform/Asimpleformviewlayer/TheviewlayerURLs,settingup/SettinguptheURLstemplatelayer/Thetemplatelayer

DjangoURLdispatcherdefining/TheDjangoURLdispatcherregularexpression/Regularexpressions

Djangowebframeworkdefining/TheDjangowebframeworkmodellayer/Themodellayerviewlayer/Theviewlayertemplatelayer/Thetemplatelayer

docstringsabout/Documentingyourcode

DRY(Don’tRepeatYourself)principle/Howdoweusemodulesandpackagesdunder

about/Goingbeyondnext

Eenclosingscope

about/Scopesentries

about/Startingtheprojectenvironment

settingup/Settinguptheenvironmentequalities

about/Aspecializedelse:elifEuclid’salgorithm

about/Don’toverdocomprehensionsandgeneratorsEuclideanformula

about/Don’toverdocomprehensionsandgeneratorsexception

about/Aspecialelseclauseexceptions

about/Exceptionsexecutionmodel,Python

about/Python’sexecutionmodel

Ffactorial

about/HowdoweusemodulesandpackagesFalcon

JSONquoteserver,building/BuildingaJSONquoteserverinFalconabout/BuildingaJSONquoteserverinFalconURL/BuildingaJSONquoteserverinFalcon

FalconAPIimplementing/ImplementingtheFalconAPImainapplication/Themainapplicationhelpers,writing/Writingthehelpershandlers,writing/WritingthehandlersAPI,running/RunningtheAPIAPI,testing/TestingtheAPI

features,Pythonportability/Portabilitycoherence/Coherencedeveloperproductivity/Developerproductivityextensivelibrary/Anextensivelibrarysoftwarequality/Softwarequalitysoftwareintegration/Softwareintegrationsatisfactionandenjoyment/Satisfactionandenjoyment

Fibonaccisequenceexampleabout/Onelastexample

filterabout/map,zip,andfilterdefining/filter

FlaskURL/WritingaFlaskviewabout/WritingaFlaskview

Flaskviewwriting/WritingaFlaskview

floatingpointnumbersURL/Reals

formabout/Creatingtheform

frameworkabout/TheDjangowebframework

functionabout/Howdoweusemodulesandpackages

functionattributesabout/Functionattributes

functions

using/Whyusefunctions?codeduplication,reducing/Reducecodeduplicationcomplextask,splitting/Splittingacomplextaskimplementationdetails,hiding/Hideimplementationdetailsreadability,improving/Improvereadabilitytraceability,improving/Improvetraceabilitywriting,tips/Afewusefultipsexample/Onefinalexample

Ggenerationbehavior,inbuilt-ins

about/Generationbehaviorinbuilt-insgeneratorexpressions

about/Generatorexpressionsgeneratorfunctions

about/Generatorfunctionsgeneratorobjects

about/Goingbeyondnextgenerators

about/map,zip,andfilter,Generatorsgeneratorfunctions/Generatorsgeneratorexpressions/Generatorsandcomprehensions/Don’toverdocomprehensionsandgenerators

getterabout/Thepropertydecorator

gettersandsettersabout/Thepropertydecorator

Gitabout/Guidelinesonhowtowritegoodcode

GitHubURL/Exceptions

globalscopeabout/Scopes

gray-boxtestingabout/Testingyourapplication

greatestcommondivisor(GCD)about/Don’toverdocomprehensionsandgenerators

Greenphaseabout/Test-drivendevelopment

GUI(GraphicalUserInterface)/RunningPythonasaGUIapplicationGUIapplication

Python,runningas/RunningPythonasaGUIapplicationabout/Secondapproach–aGUIapplicationimports/Secondapproach–aGUIapplication,Theimportslayoutlogic/Thelayoutlogicbusinesslogic/Thebusinesslogicimproving/Howtoimprovetheapplication?

GunicornURL/RunningtheAPI

Gunicorn(GreenUnicorn)about/BuildingaJSONquoteserverinFalcon

Hhandlers

writing/Writingthehandlerspasswordvalidatorhandler,coding/Codingthepasswordvalidatorhandlerpasswordgeneratorhandler,coding/Codingthepasswordgeneratorhandler

Hashabilityabout/Settypes

helperswriting/Writingthehelperspasswordvalidator,coding/Codingthepasswordvalidatorpasswordgenerator,coding/Codingthepasswordgenerator

HTMLDocumentObjectModel(DOM)about/Creatingtheform

HypertextMarkupLanguage(HTML)about/Theviewlayer

HypertextTransferProtocol(HTTP)about/WhatistheWeb?

IIDLE(IntegratedDeveLopmentEnvironment)/RunningthePythoninteractiveshellimmutable

about/Aproperintroduction,Mutableorimmutable?Thatisthequestionimmutablesequences

about/Immutablesequencesstringsandbytes/Stringsandbytestuples/Tuples

implicitconcatenationabout/BuildingaJSONquoteserverinFalcon

in-placeabout/Unpackingtheuserdata

indexingabout/Aboutindexingandslicing

inequalitiesabout/Aspecializedelse:elif

infiniteloopabout/Thewhileloop

inheritanceabout/Inheritanceandcomposition

initializerabout/Objectandclasses,Initializinganinstance

innerabout/Aspecializedelse:elif

inputparametersabout/Inputparametersargumentpassing/Argumentpassingassignment,toargumentnames/Assignmenttoargumentnamesdon’taffectthecallermutable,changing/Changingamutableaffectsthecallerspecifying/Howtospecifyinputparameterspositionalarguments/Positionalargumentskeywordarguments/Keywordargumentsanddefaultvaluesdefaultvalues/Keywordargumentsanddefaultvaluesvariablepositionalarguments/Variablepositionalargumentsvariablekeywordarguments/Variablekeywordargumentskeyword-onlyarguments/Keyword-onlyarguments,combining/Combininginputparametersmutabledefaults/Avoidthetrap!Mutabledefaults

inside-outtechniqueabout/Iteratingoverasequence

installingPython/InstallingPython

instanceattributesabout/Classandobjectnamespaces

instancesofclassesabout/Objectandclasses

integerdivision(//)about/Integers

IntegratedDevelopmentEnvironments(IDEs)/AnoteontheIDEsabout/Thepropertydecorator

interfacetestingabout/Interfacetesting

Internetabout/WhatistheWeb?

ipdblibraryabout/UsingthePythondebugger

IPythonURL/Wheredowegofromhere?

Ipythonabout/IPythonandJupyternotebookURL/IPythonandJupyternotebook

iterableabout/Writingacustomiterator

iteratorabout/Iteratorsanditerables,Writingacustomiterator

itertoolsmoduleabout/Aquickpeekattheitertoolsmoduleinfiniteiterators/Infiniteiteratorsterminating,onshortestinputsequence/Iteratorsterminatingontheshortestinputsequencecombinatoricgenerators/Combinatoricgenerators

JjQuery

about/ThetemplatelayerURL/Thetemplatelayer

JSON(JavaScriptObjectNotation)about/Exceptions

JSONquoteserverbuilding,inFalcon/BuildingaJSONquoteserverinFalcon

Jupyternotebookabout/IPythonandJupyternotebookURL/IPythonandJupyternotebook

KKeepassX

about/Thechallengekeyword-onlyparameter

about/Keyword-onlyargumentskeywordarguments

about/Keywordargumentsanddefaultvalues

Llambdas

about/Anonymousfunctionslibrary

about/Howdoweusemodulesandpackageslistcomprehension

about/Listslocal,enclosing,global,built-in(LEGB)/Scopeslocalscope

about/Scopeslogfiles

about/Inspectinglogfilesloggers/Inspectinglogfileshandlers/Inspectinglogfilesfilters/Inspectinglogfilesformatters/Inspectinglogfiles

loggingURL/Inspectinglogfiles

loopiterating,overrange/Iteratingoverarangeiterating,oversequence/Iteratingoverasequence

loopingabout/Loopingforloop/Theforloopiteratorsanditerables/Iteratorsanditerablesiterating,overmultiplesequences/Iteratingovermultiplesequenceswhileloop/Thewhileloopbreakandcontinuestatements/Thebreakandcontinuestatementselseclause/Aspecialelseclause

Mmagicmethod

about/Objectandclassesmagicmethods

about/Goingbeyondnextmap

about/map,zip,andfilterdefining/map

markdownabout/IPythonandJupyternotebook

masterpasswordabout/Thechallenge

MatplotlibURL/Wheredowegofromhere?

Mercurialabout/Guidelinesonhowtowritegoodcode

mergeandinsertionsortabout/Lists

metaclassesabout/Objectandclasses,ThesimplestPythonclass

metaprogrammingabout/ThesimplestPythonclass

methodabout/Goingbeyondnext

methodresolutionorder(MRO)about/Methodresolutionorder

methodsabout/Aproperintroduction

middlewareclassabout/Writingthetemplates

migrationabout/AddingtheEntrymodel

migrationsapplying/Startingtheproject

mixinsabout/Multipleinheritance

mocksabout/Mockobjectsandpatching

modelabout/Themodellayer

model-template-view(MTV)patternabout/Djangodesignphilosophy

model-view-controller(MVC)

about/Djangodesignphilosophymodules

using/Howdoweusemodulesandpackagesmutable

about/Aproperintroduction,Mutableorimmutable?Thatisthequestionmutablesequences

about/Mutablesequenceslists/Listsbytearrays/Bytearrays

NNameErrorexception/Scopesnamelocalization

about/Namelocalizationnamemangling

about/Privatemethodsandnamemanglingnames

about/Namesandnamespacesdefining/Aboutthenames

namespacesabout/Namesandnamespaces

nanoabout/Usingconsoleeditors

negativeindexingabout/Aboutindexingandslicing

nose-parameterizedURL/Amoreinterestingexample

NumbaURL/Wheredowegofromhere?

numbersabout/Numbersintegers/Integersbooleans/Booleansrealnumbers/Realscomplexnumbers/Complexnumbersfractionsanddecimals/Fractionsanddecimals

NumericPythonabout/CreatingtheDataFrame

NumPyURL/Wheredowegofromhere?

Oobject

defining/Everythingisanobjectmutable/Mutableorimmutable?Thatisthequestionimmutable/Mutableorimmutable?Thatisthequestion

object-orientedprogrammingabout/Object-orientedprogrammingPythonclass/ThesimplestPythonclassclassandobjectnamespaces/Classandobjectnamespacesattributeshadowing/Attributeshadowingselfvariable,using/I,me,andmyself–usingtheselfvariableinstance,initializing/Initializinganinstancecode,reusing/OOPisaboutcodereuseinheritanceandcomposition/Inheritanceandcompositionbaseclass,accessing/Accessingabaseclassmultipleinheritance/Multipleinheritancemethodresolutionorder/Methodresolutionorderclassandstaticmethods/Staticandclassmethodsprivatemethods/Privatemethodsandnamemanglingnamemangling/Privatemethodsandnamemanglingpropertydecorator/Thepropertydecoratoroperatoroverloading/Operatoroverloadingpolymorphism/Polymorphism–abriefoverview

object-relationalmapping(ORM)about/Themodellayer

objectsabout/Aproperintroduction,Objectandclasses,Object-orientedprogrammingimporting/Importingobjectsrelativeimports/Relativeimports

onewayencryptionalgorithmabout/Themodellayer

onewayhashfunctionabout/Themodellayer

operatingsystem(OS)about/Portability

operatoroverloadingabout/Lists,Operatoroverloading

outerabout/Aspecializedelse:elif

Ppackage

about/HowisPythoncodeorganizedpackages

using/HowdoweusemodulesandpackagesPandas

URL/Wheredowegofromhere?patching

about/MockobjectsandpatchingURL/Aclassicunittestexample

pdbabout/UsingthePythondebugger

PEP328about/Relativeimports

performanceconsiderationsabout/Someperformanceconsiderations

pivottableabout/Visualizingtheresults

plaintextabout/Themodellayer

polymorphism/Polymorphism–abriefoverviewprimarykey

about/Themodellayer,AddingtheEntrymodelprimegenerator

about/Example1–aprimegeneratorprimitive

about/Don’toverdocomprehensionsandgeneratorsprincipleofleastastonishment

about/Theprincipleofleastastonishmentprinciples,Djangowebframework

DRY/DjangodesignphilosophyLoosecoupling/DjangodesignphilosophyLesscode/DjangodesignphilosophyConsistency/Djangodesignphilosophy

profilingabout/Exceptions,Whentoprofile?

propertiesabout/Aproperintroduction

protocolsabout/WhatistheWeb?

pullprotocolabout/HowdoestheWebwork?

pushprotocol

about/HowdoestheWebwork?PyGTK

about/Secondapproach–aGUIapplication,wxPython,PyQt,andPyGTKPyPy

URL/Whatarethedrawbacks?PyQt

about/Secondapproach–aGUIapplication,wxPython,PyQt,andPyGTKpytest

URL/AmoreinterestingexamplePythagoreantriple

about/FilteringacomprehensionPython

about/EnterthePythonfeatures/AboutPython,Softwareintegrationdrawbacks/Whatarethedrawbacks?users/WhoisusingPythontoday?installing/InstallingPythonreferences/SettingupthePythoninterpreterrunning,asservice/RunningPythonasaservicerunning,asGUIapplication/RunningPythonasaGUIapplicationexecutionmodel/Python’sexecutionmodelprofiling/ProfilingPython

Python2versusPython3/Python2versusPython3–thegreatdebate

Pythoncodeorganizing/HowisPythoncodeorganized

Pythoncultureabout/ThePythonculture

PythonEnhancementProposal(PEP)/GuidelinesonhowtowritegoodcodePythoninteractiveshell

running/RunningthePythoninteractiveshellPythoninterpreter

settingup/SettingupthePythoninterpreterPythonmodule

reference/EverythingisanobjectPythonPackageIndex(PyPI)/AnextensivelibraryPythonprogram

running/HowyoucanrunaPythonprogramPythonscripts

running/RunningPythonscripts

Qqualityassurance(QA)

about/Testingyourapplication

Rradix-64

about/Theimportsrecursivefunctions

about/RecursivefunctionsRedphase

about/Test-drivendevelopmentRefactor

about/Test-drivendevelopmentregexwebsite

defining/AregexwebsiteDjango,settingup/SettingupDjangoEntrymodel,adding/AddingtheEntrymodeladminpanel,customizing/Customizingtheadminpanelform,creating/Creatingtheformviews,writing/WritingtheviewsURLsandviews,using/TyingupURLsandviewstemplates,writing/Writingthetemplates

regularexpression/Regularexpressionsrelationaldatabase

about/Themodellayerrelativeimports

URL/Relativeimportsrequest-responseclient-serverprotocol

about/HowdoestheWebwork?returnvalues

about/Returnvaluesmultiplevalues,returning/Returningmultiplevalues

Sscheduler

about/Threadingconsiderationsschemamigration

about/AddingtheEntrymodelSchwartziantransform

about/mapScikit-Learn

URL/Wheredowegofromhere?SciPy

URL/Wheredowegofromhere?scopes

about/Scopeslocal/Scopesenclosing/Scopesglobal/Scopesbuilt-in/Scopes

scopesandnameresolutiondefining/Scopesandnameresolutionglobalandnonlocalstatements/Theglobalandnonlocalstatements

scriptingabout/Firstapproach–scriptingimports/Firstapproach–scripting,Theimportsarguments,parsing/Parsingargumentsbusinesslogic/Thebusinesslogic

servicePython,runningas/RunningPythonasaservice

service-orientedarchitecture(SOA)about/Inspectinglogfiles

service-orientedarchitecturesabout/Inspectinglogfiles

setterabout/Thepropertydecorator

settypesabout/Settypes

Single-PageApplication(SPA)about/Thefutureofwebdevelopment

SingleResponsibilityPrinciple(SRP)about/Writingthehelpers

slicingabout/Aboutindexingandslicing

smallvaluescachingabout/Smallvaluescaching

spaceabout/Someperformanceconsiderations

Sphinxabout/Documentingyourcode

SQL(StructuredQueryLanguage)about/Themodellayer

staticmethodsabout/Staticmethods

statisticalprofilingabout/ProfilingPython

stringsencoding/Encodinganddecodingstringsdecoding/Encodinganddecodingstringsindexing/Indexingandslicingstringsslicing/Indexingandslicingstrings

strobjectsabout/Stringsandbytes

symmetricencryptionalgorithmabout/Themodellayer

system-exitingexceptionsabout/Exceptions

TTcl(ToolCommandLanguage)/RunningPythonasaGUIapplicationTCP/IP(TransmissionControlProtocol/InternetProtocol)

about/HowdoestheWebwork?templatelayer

about/Thetemplatelayerhomeandfootertemplates/Homeandfootertemplatesrecords,listing/Listingallrecordsrecords,creating/Creatingandeditingrecordsrecords,editing/CreatingandeditingrecordsAPI,defining/TalkingtotheAPIrecords,deleting/Deletingrecords

terminalabout/SettingupthePythoninterpreter

ternaryoperatorabout/Theternaryoperator

testdefining/Theanatomyofatestpreparation/Theanatomyofatestexecution/Theanatomyofatestverification/Theanatomyofatestfailing/Makingatestfail

test-drivendevelopment(TDD)about/Test-drivendevelopmentbenefits/Test-drivendevelopmentdisadvantages/Test-drivendevelopment

testingguidelinesdefining/Testingguidelines

testsfront-endtests/Testingyourapplicationscenariotests/Testingyourapplicationintegrationtests/Testingyourapplicationsmoketests/TestingyourapplicationAcceptancetests/Testingyourapplicationfunctionaltests/Testingyourapplicationdestructivetests/Testingyourapplicationperformancetests/Testingyourapplicationusabilitytests/Testingyourapplicationsecurityandpenetrationtests/Testingyourapplicationunittests/Testingyourapplicationregressiontests/Testingyourapplicationcomparing,withmocks/Comparingtestswithandwithoutmockscomparing,withoutmocks/Comparingtestswithandwithoutmocks

boundaries/Boundariesandgranularitygranularity/Boundariesandgranularityexample/Amoreinterestingexample

threadabout/Threadingconsiderations

timeabout/Someperformanceconsiderations

Timsortabout/Lists

Tk/RunningPythonasaGUIapplicationtkinter

about/Secondapproach–aGUIapplicationTkinter/RunningPythonasaGUIapplicationtkinter.tix(TkInterfaceExtension)module

about/Thetkinter.tixmoduletkinter.tixmodule

about/Thetkinter.tixmoduleTkinterface

about/Secondapproach–aGUIapplicationtriangulation

about/Comparingtestswithandwithoutmockstroubleshootingguidelines

about/Troubleshootingguidelinesconsoleeditors,using/Usingconsoleeditorsinspecting/Wheretoinspecttestsused,fordebugging/Usingteststodebugmonitoring/Monitoring

truedivision(/)about/Integers

tupleabout/Tuples

turtlemoduleabout/Theturtlemodule

Uunicodecodepoints

about/StringsandbytesUniformResourceLocator(URL)

about/TheDjangoURLdispatcherunittest

about/Unittestingwriting/Writingaunittestmockobjects/Mockobjectsandpatchingpatching/Mockobjectsandpatchingassertions/Assertionsexample/Aclassicunittestexample

unittestingdefining/Unittesting

unpackingabout/Variablepositionalarguments

Upcastingabout/Booleans

useracceptancetesting(UAT)about/Testingyourapplication

userexperience(UX)about/Testingyourapplication

Utf-8about/Encodinganddecodingstrings

Vviewlayer

about/Theviewlayerimportsandhomeview/Importsandhomeviewrecords,listing/Listingallrecordsrecords,creating/Creatingrecordsrecords,updating/Updatingrecordsrecords,deleting/Deletingrecords

viewswriting/Writingtheviewshomeview/Thehomeviewentrylistview/Theentrylistviewformview/Theformview

vimabout/Usingconsoleeditors

virtualenvabout/Aboutvirtualenvreferencelink/Aboutvirtualenv

virtualenvironmentcreating/Yourfirstvirtualenvironment

Wwastedtime

about/AboutthenamesWeb(WorldWideWeb)

defining/WhatistheWeb?working/HowdoestheWebwork?

webdevelopmentdefining/Thefutureofwebdevelopment

webframeworkabout/TheDjangowebframework

white-boxtestsabout/Testingyourapplication

wxPythonabout/Secondapproach–aGUIapplication,wxPython,PyQt,andPyGTK

Yyieldfromexpression

about/Theyieldfromexpression

Zzip

about/map,zip,andfilterdefining/zip