what's new in ecto 2blog.plataformatec.com.br/wp-content/uploads/2016/11/... · 2016. 12....

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

1.9

1.10

1.11

1.12

1.13

TableofContentsForeword

Introduction

1.EctoisnotyourORM

2.Schemalessqueries

3.Schemalesschangesets

4.Dynamicqueries

5.Multitenancywithqueryprefixes

6.Aggregatesandsubqueries

7.Improvedassociationsandfactories

8.Manytomanyandcasting

9.Manytomanyandupserts

10.ComposabletransactionswithEcto.Multi

11.ConcurrenttestswiththeSQLSandbox

2

ForewordInJanuary2017,wewillcelebrate5yearssincewedecidedtoinvestinElixir.Backin2012,JoséValim,ourco-founderandpartner,presentedustheideaofaprogramminglanguagethatwouldbeexpressive,embraceproductivityinitstooling,andleveragetheErlangVMtonotonlytackletheproblemsinwritingconcurrentsoftwarebutalsotobuildfault-tolerantanddistributedsystems.

Elixircontinued,insomesense,tobeariskyprojectformonths.Wewerecertainlyexcitedaboutspreadingfunctional,concurrentanddistributedprogrammingconceptstomoreandmoredevelopers,hopingitwouldleadtoapositiveimpactonthesoftwaredevelopmentindustry,butdevelopingalanguageisalong-termeffortthatmayneverbecomeconcrete.

Duringthesummerof2013,othercompaniesanddevelopersstartedtoshowinterestonElixir.Weheardaboutcompaniesusingitinproduction,moredevelopersbegantocontributeandcreatetheirownprojects,differentpublisherswerewritingbooksonthelanguage,andsoon.SucheventsgaveustheconfidencetoinvestmoreinElixirandbringthelanguagetoversion1.0.

OnceElixir1.0waslaunchedinSeptember2014,weturnedourfocustothewebplatform.WetidiedupPlug,thebuildingblockforwritingwebapplicationsinElixir.WealsofocusedintensivelyonEcto,bringingittoversion1.0togetherwiththeEctoteam,andthenworkedalongsideChrisMcCordandteamtogetthefirstmajorPhoenixreleaseout.Duringthistimewealsostartedothercommunitycentricinitiatives,suchasElixirRadar,andbeganourfirstcommercialElixirprojects.

Today,boththecommunityandouropensourceprojectsareshowingsteadyandhealthygrowth.Elixirisastablelanguagewithcontinuousimprovementslandedinminorversions.PlugcontinuestobeasolidfoundationforframeworkssuchasPhoenix.Ecto,however,requiredmorethanasmallnudgeintherightdirection.Werealizedthatweneededtoletgoofold,harmfulhabitsandmakeEctolessofanabstractionlayerandmoreofatoolyoucontrolandapplytodifferentproblems.

ThisbookisthefinaleffortbehindEcto2.0.ItshowcasesthenewdirectionwehaveplannedforEcto,thestructuralimprovementsmadebytheEctoteamandmanyofitsnewfeatures.Wehopeyouwillenjoyit.Afterall,itistimetoletgoofpasthabits.

Havefun,

-ThePlataformatecteam

Foreword

3

http://plataformatec.com.br/elixir-radar/?utm_source=ebook-ecto-2-beta-1&utm_medium=referral&utm_campaign=ebook-ecto-2&utm_content=foreword

Foreword

4

CONTACTUS

http://plataformatec.com.br/contact?utm_source=ebook-ecto-2-beta-3&utm_medium=referral&utm_campaign=ebook-ecto-2&utm_content=foreword

Foreword

5

IntroductionAuthornote:WelcometotheearlycutofourEcto2.0book.Thisisourlastbetaeditionwithallbookchaptersreadyforreview.Ifyouhavesuggestionsorstumbleduponerrors,[email protected].

Ecto2.0isasubstantialdeparturefromearlierversions.Insteadofthinkingaboutmodels,Ecto2.0aimstoprovidedevelopersawiderangeofdata-centrictools.Therefore,inordertouseEcto2.0effectively,wemustlearnhowtowieldthosetoolsproperly.That'sthegoalofthisbook.

Thisbook,however,isnotanintroductiontoEcto.IfyouhaveneverusedEctobefore,werecommendyoutogetstartedwithEcto'sdocumentationandlearnmoreaboutrepositories,queries,schemasandchangesets.Weassumethereaderisfamiliarwiththesebuildingblocksandhowtheyrelatetoeachother.

ThefirstchaptersofthebookwillcoverthebiggestconceptualchangesinEcto2.0.Wewilltalkaboutrelationalmappersin"EctoisnotyourORM"andthenexploreSchemalessQueriesandtherelationshipbetweenSchemasandChangesets.

Afterwewilltakeadeeperlookintoqueries,discussinghowEcto2.0makesiteasiertobuilddynamicqueries,howtotargetdifferentdatabasesviaqueryprefixes,aswellasthenewaggregateandsubqueryfeatures.

Thenwewillgobacktoschemasanddiscusstheschema-relatedenhancementsthatarepartofEcto2.0,suchastheimprovedassociationsupport,many_to_manyassociationsandEcto's2.1upsertsupport.

Finally,wewillexplorebrandnewtopics,likethenewEctoSQLSandbox,thatallowsdeveloperstoruntestsagainstthedatabaseconcurrently,aswellasEcto.Multi,whichmakesworkingwithtransactionssimplerthanever.

Introduction

6

mailto:[email protected]

http://hexdocs.pm/ecto/

EctoisnotyourORMDependingonyourperspective,thisisaratherboldorobviousstatementtostartthisbook.Afterall,Elixirisnotanobject-orientedlanguage,soEctocan'tbeanObject-relationalMapper.However,thisstatementisslightlymorenuancedthanitlooksandthereareimportantlessonstobelearnedhere.

OisforObjectsAtitscore,objectscouplestateandbehaviourtogether.Inthesameuserobject,youcanhavedata,liketheuser.name,aswellasbehaviour,likeconfirmingaparticularuseraccountviauser.confirm().Whilesomelanguagesenforcedifferentsyntaxesbetweenaccessingdata(user.namewithoutparentheses)andbehaviour(user.confirm()withparentheses),otherlanguagesfollowtheUniformAccessPrincipleinwhichanobjectshouldnotmakeadistinctionbetweenthetwosyntaxes.EiffelandRubyarelanguagesthatfollowsuchprinciple.

Elixirfailsthe"couplingofstateandbehaviour"test.InElixir,weworkwithdifferentdatastructuressuchastuples,lists,mapsandothers.Behaviourcannotbeattachedtodatastructures.Behaviourisalwaysaddedtomodulesviafunctions.

Whenthereisaneedtoworkwithstructureddata,Elixirprovidesstructs.Structsdefineasetoffields.Astructwillbereferencedbythenameofthemodulewhereitisdefined:

defmoduleUserdo

defstruct[:name,:email]

end

user=%User{name:"JohnDoe",email:"[email protected]"}

Onceauserstructiscreated,wecanaccessitsemailviauser.email.However,structsareonlydata.Itisimpossibletoinvokeuser.confirm()onaparticularstructinawayitwillexecutecoderelatedtoe-mailconfirmation.

Althoughwecannotattachbehaviourtostructs,itispossibletoaddfunctionstothesamemodulethatdefinesthestruct:

1.EctoisnotyourORM

7

https://en.wikipedia.org/wiki/Uniform_access_principle

defmoduleUserdo

defstruct[:name,:email]

defconfirm(user)do

#Confirmtheuseremail

end

end

Evenwiththedefinitionabove,itisimpossibleinElixirtoconfirmagivenuserbycallinguser.confirm().Instead,theUserprefixisrequiredandtheuserstructmustbeexplicitlygivenasargument,asinUser.confirm(user).Attheendoftheday,thereisnostructuralcouplingbetweentheuserstructandthefunctionsintheUsermodule.HenceElixirdoesnothavemethods,ithasfunctions.

Withouthavingobjects,Ectocertainlycan'tbeanORM.However,ifweletgooftheletter"O"forasecond,canEctostillbearelationalmapper?

RelationalmappersAnObject-RelationalMapperisatechniqueforconvertingdatabetweenincompatibletypesystems,commonlydatabases,toobjects,andback.

Similarly,EctoprovidesschemasthatmapsanydatasourceintoanElixirstruct.Whenappliedtoyourdatabase,Ectoschemasarerelationalmappers.Therefore,whileEctoisnotarelationalmapper,itcontainsarelationalmapperaspartofthemanydifferenttoolsitoffers.

Forexample,theschemabelowtiesthefieldsname,email,inserted_atandupdated_attofieldssimilarlynamedintheuserstable:

defmoduleMyApp.Userdo

useEcto.Schema

schema"users"do

field:name

field:email

timestamps()

end

end

Theappealbehindschemasisthatyoudefinetheshapeofthedataonceandyoucanusethisshapetoretrievedatafromthedatabaseaswellascoordinatechangeshappeningonthedata:

1.EctoisnotyourORM

8

http://hexdocs.pm/ecto/Ecto.Schema.html

MyApp.User

|>MyApp.Repo.get!(13)

|>Ecto.Changeset.cast([name:"newname"],[:name,:email])

|>MyApp.Repo.update!

Byrelyingontheschemainformation,Ectoknowshowtoreadandwritedatawithoutextrainputfromthedeveloper.Insmallapplications,thiscouplingbetweenthedataanditsrepresentationisdesired.However,whenusedwrongly,itleadstocomplexcodebasesandsubparsolutions.

ItisimportanttounderstandtherelationshipbetweenEctoandrelationalmappersbecausesaying"EctoisnotyourORM"doesnotautomaticallysaveEctoschemasfromsomeofthedownsidesmanydevelopersassociateORMswith.

HerearesomeexamplesofissuesoftenassociatedwithORMsthatEctodevelopersmayrunintowhenusingschemas:

ProjectsusingEctomayend-upwith"GodSchemas",commonlyreferredas"GodModels","FatModels"or"CanonicalModels"insomelanguagesandframeworks.Suchschemascouldcontainhundredsoffields,oftenreflectingbaddecisionsdoneatthedatalayer.Insteadofprovidingonesingleschemawithfieldsthatspanmultipleconcerns,itisbettertobreaktheschemaacrossmultiplecontexts.Forexample,insteadofasingleMyApp.Userschemawithdozensoffields,considerbreakingitintoMyApp.Accounts.User,MyApp.Purchases.Userandsoon.Eachstructwithfieldsexclusivetoitsenclosingcontext

Developersmayexcessivelyrelyonschemaswhensometimesthebestwaytoretrievedatafromthedatabaseisintoregulardatastructures(likemapsandtuples)andnotpre-definedshapesofdatalikestructs.Forexample,whendoingsearches,generatingreportsandothers,thereisnoreasontorelyorreturnschemasfromsuchqueries,asitoftenreliesondatacomingfrommultipletableswithdifferentrequirements

Developersmaytrytousethesameschemaforoperationsthatmaybequitedifferentstructurally.Manyapplicationswouldboltfeaturessuchasregistration,accountlogin,intoasingleUserschema,whilehandlingeachoperationindividually,possiblyusingdifferentschemas,wouldleadtosimplerandclearersolutions

Inthenexttwochapters,wewanttobreakthose"badpractices"apartbyexploringhowtouseEctowithnoormultipleschemaspercontext.Bylearninghowtoinsert,delete,manipulateandvalidatedatawithandwithoutschemas,wehopedeveloperswillfeelcomfortablewithbuildingcomplexapplicationswithoutrelyingonone-size-fits-allschemas.

1.EctoisnotyourORM

9

1.EctoisnotyourORM

10

SchemalessqueriesMostqueriesinEctoarewrittenusingschemas.Forexample,toretrieveallpostsinadatabase,onemaywrite:

MyApp.Repo.all(Post)

Intheconstructabove,Ectoknowsallfieldsandtheirtypesintheschema,rewritingthequeryaboveto:

MyApp.Repo.all(frompinPost,select:%Post{title:p.title,body:p.body,...})

Interestingly,backinEcto'searlydays,therewasnosuchthingasschemas.Queriescouldonlybewrittendirectlyagainstadatabasetablebypassingthetablenameasastring:

MyApp.Repo.all(frompin"posts",select:{p.title,p.body})

Whenwritingschemalessqueries,theselectexpressionmustbeexplicitlywrittenwithallthedesiredfields.

WhiletheabovesyntaxmadeitintoEcto1.0,bythetimeEcto1.0waslaunched,mostofthedevelopmentfocusinEctohadchangedtowardsschemas.Thismeanswhiledeveloperswereabletoreaddatawithoutschemas,theywereoftentooverbose.Notonlythat,ifyouwantedtoinsertentriestoyourdatabasewithoutschemas,youwereoutofluck.

Ecto2.0levelsupthegamebyaddingmanyimprovementstoschemalessqueries,notonlybyimprovingthesyntaxforreadingandupdatingdata,butalsobyallowingalldatabaseoperationstobeexpressedwithoutaschema.

insert_allOneofthefunctionsaddedtoEcto2.0isEcto.Repo.insert_all/3.Withinsert_all,developerscaninsertmultipleentriesatonceintoarepository:

MyApp.Repo.insert_all(Post,[[title:"hello",body:"world"],

[title:"another",body:"post"]])

2.Schemalessqueries

11

https://hexdocs.pm/ecto/Ecto.Repo.html#c:insert_all/3

Althoughinsert_allisjustaregularElixirfunction,itplaysanimportantroleinEcto2.0asitallowsdeveloperstoread,create,updateanddeleteentrieswithoutaschema.insert_allwasthelastpieceofthepuzzle.Let'sseesomeexamples.

Ifyouarewritingareportingview,itmaybecounter-productivetothinkhowyourexistingapplicationschemasrelatetothereportbeinggenerated.Itisoftensimplertowriteaquerythatreturnsonlythedatayouneed,withouttryingtofitthedataintoexistingschemas:

importEcto.Query

defrunning_activities(start_at,end_at)

MyApp.Repo.all(

fromuin"users",

join:ain"activities",

on:a.user_id==u.id,

where:a.start_at>type(^start_at,Ecto.DateTime)and

a.end_at<type(^end_at,Ecto.DateTime),

group_by:a.user_id,

select:%{user_id:a.user_id,interval:a.start_at-a.end_at,count:count(u.id

)}

)

end

Thefunctionabovedoesnotrelyonschemas.Itreturnsonlythedatathatmattersforbuildingthereport.Noticehowweusethetype/2functiontospecifywhatistheexpectedtypeoftheargumentweareinterpolating,benefitingfromthesametypecastingguaranteesaschemawouldgive.

Inserts,updatesanddeletescanalsobedonewithoutschemasviainsert_all,update_allanddelete_allrespectively:

#InsertdataintopostsandreturnitsID

[%{id:id}]=

MyApp.Repo.insert_all"posts",[[title:"hello"]],returning:[:id]

#UsetheIDtotriggerupdates

post=frompin"posts",where:[id:^id]

{1,_}=MyApp.Repo.update_allpost,set:[title:"newtitle"]

#Aswellasfordeletes

{1,_}=MyApp.Repo.delete_allpost

ItisnothardtoseehowtheseoperationsdirectlymaptotheirSQLvariants,keepingthedatabaseatyourfingertipswithouttheneedtointermediatealloperationsthroughschemas.

2.Schemalessqueries

12

SimplerqueriesBesidessupportingschemalessinserts,updatesanddeletesqueries,withvaryingdegreesofcomplexity,Ecto2.0alsomakesregularschemalessqueriesmoreexpressive.

Oneexampleistheabilitytoselectalldesiredfieldswithoutduplication.Inearlyversions,youwouldhavetowriteverboseselectexpressionssuchas:

frompin"posts",select:%{title:p.title,body:p.body}

WithEcto2.0youcansimplypassthedesiredlistoffieldsdirectly:

from"posts",select:[:title,:body]

Thetwoqueriesaboveareequivalent.Whenalistoffieldsisgiven,Ectowillautomaticallyconvertthelistoffieldstoamaporastruct.

SupportforpassingalistoffieldsorkeywordlistshasbeenaddedtoalmostallqueryconstructsinEcto2.0.Forexample,wecanuseanupdatequerytochangethetitleofagivenpostwithoutaschema:

defupdate_title(post,new_title)do

query=from"posts",where:[id:^post.id],update:[set:[title:^new_title]]

MyApp.Repo.update_all(query)

end

Theupdateconstructsupportsfourcommands:

:set-setsthegivencolumntothegivenvalues:inc-incrementsthegivencolumnbythegivenvalue:push-pushes(appends)thegivenvaluetotheendofanarraycolumn:pull-pulls(removes)thegivenvaluefromanarraycolumn

Forexample,wecanincrementacolumnatomicallybyusingthe:inccommand,withorwithoutschemas:

defincrement_page_views(post)do

query=from"posts",where:[id:^post.id],update:[inc:[page_views:1]]

MyApp.Repo.update_all(query)

end

2.Schemalessqueries

13

https://hexdocs.pm/ecto/Ecto.Query.html#update/3

Byallowingregulardatastructurestobegiventomostqueryoperations,Ecto2.0makesquerieswithandwithoutschemasmoreaccessible.Notonlythat,italsoenablesdeveloperstowritedynamicqueries,wherefields,filters,orderingcannotbespecifiedupfront.Wewillexploresuchwithmoredetailsinupcomingchapters.Fornow,let'scontinueexploringschemasinthecontextofchangesets.

2.Schemalessqueries

14

SchemasandchangesetsInthelastchapterwelearnedhowtoperformalldatabaseoperations,frominsertiontodeletion,withoutusingaschema.Whilewehavebeenexploringtheabilitytowriteconstructswithoutschemas,wehaven'tdiscussedwhatschemasactuallyare.Inthischapter,wewillrectifythat.

Inthischapterwewilltakealookattheroleschemasplaywhenvalidatingandcastingdatathroughchangesets.Aswewillsee,sometimesthebestsolutionisnottocompletelyavoidschemas,butbreakalargeschemaintosmallerones.Maybeoneforreadingdata,anotherforwriting.Maybeoneforyourdatabase,anotherforyourforms.

SchemasaremappersTheEctodocumentationsays:

AnEctoschemaisusedtomapanydatasourceintoanElixirstruct.

WeputemphasisonanybecauseitisacommonmisconceptiontothinkEctoschemasmaponlytoyourdatabasetables.

Forinstance,whenyouwriteawebapplicationusingPhoenixandyouuseEctotoreceiveexternalchangesandapplysuchchangestoyourdatabase,wehavethismapping:

Database<->Ectoschema<->Forms/API

AlthoughthereisasingleEctoschemamappingtobothyourdatabaseandyourAPI,inmanysituationsitisbettertobreakthismappingintwo.Let'sseesomepracticalexamples.

Imagineyouareworkingwithaclientthatwantsthe"SignUp"formtocontainthefields"Firstname","Lastname"alongside"E-mail"andotherinformation.Youknowthereareacoupleproblemswiththisapproach.

Firstofall,noteveryonehasafirstandlastname.Althoughyourclientisdecidedonpresentingbothfields,theyareaUIconcern,andyoudon'twanttheUItodictatetheshapeofyourdata.Furthermore,youknowitwouldbeusefultobreakthe"SignUp"informationacrosstwotables,the"accounts"and"profiles"tables.

Giventherequirementsabove,howwouldweimplementtheSignUpfeatureinthebackend?


15

https://hexdocs.pm/ecto/Ecto.Schema.html

Oneapproachwouldbetohavetwoschemas,AccountandProfile,withvirtualfieldssuchasfirst_nameandlast_name,anduseassociationsalongsidenestedformstotietheschemastoyourUI.Oneofsuchschemaswouldbe:

defmoduleProfiledo

useEcto.Schema

schema"profiles"do

field:name

field:first_name,:string,virtual:true

field:last_name,:string,virtual:true

...

end

end

ItisnothardtoseehowwearepollutingourProfileschemawithUIrequirementsbyaddingfieldssuchfirst_nameandlast_name.IftheProfileschemaisusedforbothreadingandwritingdata,itmayend-upinanawkwardplacewhereitisnotusefulforany,asitcontainsfieldsthatmapjusttooneortheotheroperation.

Onealternativesolutionistobreakthe"Database<->Ectoschema<->Forms/API"mappingintwoparts.Thefirstwillcastandvalidatetheexternaldatawithitsownstructurewhichyouthentransformandwritetothedatabase.Forsuch,let'sdefineaschemanamedRegistrationthatwilltakecareofcastingandvalidatingtheformdataexclusively,mappingdirectlytotheUIfields:

defmoduleRegistrationdo

useEcto.Schema

embedded_schemado

field:first_name

field:last_name

field:email

end

end

Weusedembedded_schemabecauseitisnotourintenttopersistitanywhere.Withtheschemainhand,wecanuseEctochangesetsandvalidationstoprocessthedata:


16

http://blog.plataformatec.com.br/2015/08/working-with-ecto-associations-and-embeds/

fields=[:first_name,:last_name,:email]

changeset=

%Registration{}

|>Ecto.Changeset.cast(params["sign_up"],fields)

|>validate_required(...)

|>validate_length(...)

Nowthattheregistrationchangesaremappedandvalidated,wecancheckiftheresultingchangesetisvalidandactaccordingly:

ifchangeset.valid?do

#Getthemodifiedregistrationstructoutofthechangeset

registration=Ecto.Changeset.apply_changes(changeset)

MyApp.Repo.transactionfn->

MyApp.Repo.insert_all"accounts",[Registration.to_account(registration)]

MyApp.Repo.insert_all"profiles",[Registration.to_profile(registration)]

end

{:ok,registration}

else

#AnnotatetheactionwetriedtoperformsotheUIshowserrors

changeset=%{changeset|action::registration}

{:error,changeset}

end

Theto_account/1andto_profile/1functionsinRegistrationwouldreceivetheregistrationstructandsplittheattributesapartaccordingly:

defto_account(registration)do

Map.take(registration,[:email])

end

defto_profile(%{first_name:first,last_name:last})do

%{name:"#{first}#{last}"}

end

Intheexampleabove,bybreakingapartthemappingbetweenthedatabaseandElixirandbetweenElixirandtheUI,ourcodebecomesclearerandourdatastructuressimpler.

NotewehaveusedMyApp.Repo.insert_all/2toadddatatoboth"accounts"and"profiles"tablesdirectly.Wehavechosentobypassschemasaltogether.However,thereisnothingstoppingyoufromalsodefiningbothAccountandProfileschemasandchangingto_account/1andto_profile/1torespectivelyreturn%Account{}and%Profile{}


17

structs.Oncestructsarereturned,theycouldbeinsertedthroughtheusualMyApp.Repo.insert/2operation.Doingsocanbeespeciallyusefulifthereareuniquenessorotherconstraintsthatyouwanttocheckduringinsertion.

SchemalesschangesetsAlthoughwechosetodefineaRegistrationschematouseinthechangeset,Ecto2.0alsoallowsdeveloperstousechangesetswithoutschemas.Wecandynamicallydefinethedataandtheirtypes.Let'srewritetheregistrationchangesetabovetobypassschemas:

data=%{}

types=%{first_name::string,last_name::string,email::string}

changeset=

{data,types}#Thedata+typestupleisequivalentto%Registration{}

|>Ecto.Changeset.cast(params["sign_up"],Map.keys(types))

|>validate_required(...)

|>validate_length(...)

YoucanusethistechniquetovalidateAPIendpoints,searchforms,andothersourcesofdata.Thechoiceofusingschemasdependsmostlyifyouwanttousethesamemappingindifferentplacesorifyoudesirethecompile-timeguaranteesElixirstructsgivesyou.Otherwise,youcanbypassschemasaltogether,beitwhenusingchangesetsorinteractingwiththerepository.

However,themostimportantlessoninthischapterisnotwhentouseornottouseschemas,butratherunderstandwhenabigproblemcanbebrokenintosmallerproblemsthatcanbesolvedindependentlyleadingtoanoverallcleanersolution.Thechoiceofusingschemasornotabovedidn'taffectthesolutionasmuchasthechoiceofbreakingtheregistrationproblemapart.


18

DynamicqueriesEctowasdesignedfromthegrounduptohaveanexpressivequeryAPIthatleveragesElixirsyntaxtoprovidequeriesthatarepre-compiledtobeperformantandsafe.Whenbuildingqueries,wemayusethekeywordssyntax

importEcto.Query

frompinPost,

where:p.author=="José"andp.category=="Elixir",

where:p.published_at>^minimum_date,

order_by:[desc:p.published_at]

orthepipe-basedone

importEcto.Query

Post

|>where([p],p.author=="José"andp.category=="Elixir")

|>where([p],p.published_at>^minimum_date)

|>order_by([p],desc:p.published_at)

Whilemanydeveloperspreferthepipe-basedsyntax,havingtorepeatthebindingpmadeitquiteverbosecomparedtothekeywordone.Furthermore,thecompile-timeaspectofEctoquerieswasatoddswithbuildingqueriesdynamically.Imagineforexampleawebapplicationthatprovidessearchfunctionalityontopofexistingposts.Theusershouldbeabletospecifymultiplecriteria,suchastheauthorname,thepostcategory,publishinginterval,etc.

InEcto1.0,theonlywaytowritesuchfunctionalitywouldbeviaEnum.reduce/3:

4.Dynamicqueries

19

deffilter(params)do

Enum.reduce(params,Post,&filter/2)

end

defpfilter({"author",author},query)do

where(query,[p],p.author==^author)

end

defpfilter({"category",category},query)do

where(query,[p],p.category==^category)

end

defpfilter({"published_at",minimum_date},query)do

where(query,[p],p.published_at>^minimum_date)

end

defpfilter({"order_by","published_at_asc"},query)do

order_by(query,[p],asc:p.published_at)

end

defpfilter({"order_by","published_at_desc"},query)do

order_by(query,[p],desc:p.published_at)

end

defpfilter(_ignore_unknown,query)do

query

end

Whilethecodeaboveworksfine,itcouplestheprocessingoftheparameterswiththequerygeneration.Itisaverboseimplementationthatisalsohardtotestsincetheresultoffilteringandhandlingofparametersarestoreddirectlyinsidethequerystruct.

Apreferredapproachwouldbetoprocesstheparametersintoregulardatastructuresandthenbuildthequeryaslateaspossible.That'sexactlywhatEcto2.0andlaterallowustodo.

FocusingondatastructuresEcto2.0providesasimplerAPIforbothkeywordandpipebasedqueriesbymakingdatastructuresfirst-class.Let'srewritetheoriginalqueriestousedatastructureswhenpossible:

frompinPost,

where:[author:"José",category:"Elixir"],

where:p.published_at>^minimum_date,

order_by:[desc::published_at]

and

4.Dynamicqueries

20

Post

|>where(author:"José",category:"Elixir")


|>order_by(desc::published_at)

Noticehowwewereabletoditchthepselectorinmostexpressions.InEcto2.0,allconstructs,fromselectandorder_bytowhereandgroup_by,acceptdatastructuresasinput.Thedatastructurecanbespecifiedatcompile-time,asabove,andalsodynamicallyatruntime,shownbelow:

where=[author:"José",category:"Elixir"]

order_by=[desc::published_at]

Post

|>where(^where)


|>order_by(^order_by)

Theadvantageofinterpolatingdatastructuresasaboveisthatwecandecoupletheprocessingofparametersfromthequerygeneration.Howevernotallexpressionscanbeconvertedtothedatastructureshape.Sincewhereconvertsakey-valuetoakey==valuecomparison,order-basedcomparisonssuchasp.published_at>^minimum_datestillneedtobewrittenaspartofthequeryasbefore.

Luckily,theupcomingEcto2.1releasesolvesthisissue.

ThedynamicmacroForcaseswherewecannotrelyondatastructuresbutstilldesiretobuildqueriesdynamically,Ecto2.1includestheEcto.Query.dynamic/2macro.

Inordertounderstandhowthedynamicmacroworkslet'srewritethefilter/1functionfromthebeginningofthischapterusingbothdatastructuresandthedynamicmacro.NotetheexamplebelowrequiresEcto2.1:

4.Dynamicqueries

21

deffilter(params)do

Post

|>order_by(^filter_order_by(params["order_by"]))

|>where(^filter_where(params))

|>where(^filter_published_at(params["published_at"]))

end

deffilter_order_by("published_at_desc"),do:[desc::published_at]

deffilter_order_by("published_at"),do:[asc::published_at]

deffilter_order_by(_),do:[]

deffilter_where(params)do

forkey<-[:author,:category],

value=params[Atom.to_string(params)],

do:{key,value}

end

deffilter_published_at(date)whenis_binary(date),

do:dynamic([p],p.published_at>^date)

deffilter_published_at(_date),

do:true

Thedynamicmacroallowsustobuilddynamicexpressionsthatarelaterinterpolatedintothequery.dynamicexpressionscanalsobeinterpolatedintodynamicexpressions,allowingdeveloperstobuildcomplexexpressionsdynamicallywithouthassle.

Becausewewereabletobreakourproblemintosmallerfunctionsthatreceiveregulardatastructures,wecanuseallthetoolsavailableinElixirtoworkwithdata.Forhandlingtheorder_byparameter,itmaybebesttosimplypatternmatchontheorder_byparameter.Forbuildingthewhereclause,wecantraversethelistofknownkeysandconvertthemtotheformatexpectedbyEcto.Forcomplexconditions,weusethedynamicmacro.

Testingalsobecomessimpleraswecantesteachfunctioninisolation,evenwhenusingdynamicqueries:

test"filterpublishedatbasedonthegivendate"do

assertinspect(filter_published_at("2010-04-17"))==

"dynamic([p],p.published_at>^\"2010-04-17\")"

assertinspect(filter_published_at(nil))==

"true"

end

WhileattheendofthedaysomedevelopersmayfeelmorecomfortablewithusingtheEnum.reduce/3approachthewholeway,Ecto2.0andlatergivesustheoptiontochoosewhichapproachworksbest.

ThankstoMichałMuskałaforsuggestionsandfeedbackonthischapter.

4.Dynamicqueries

22

4.Dynamicqueries

23

MultitenancywithqueryprefixesEcto2.0introducestheabilitytorunqueriesindifferentprefixesusingasinglepoolofdatabaseconnections.FordatabasesenginessuchasPostgres,Ecto'sprefixmapstoPostgres'DDLschemas.ForMySQL,eachprefixisadifferentdatabaseonitsown.

Queryprefixesmaybeusefulindifferentscenarios.Forexample,multitenantappsrunningonPostgreswoulddefinemultipleprefixes,usuallyoneperclient,underasingledatabase.Theideaisthatprefixeswillprovidedataisolationbetweenthedifferentusersoftheapplication,guaranteeingeithergloballyoratthedatalevelthatqueriesandcommandsactonaspecificprefix.

Prefixesmayalsobeusefulonhigh-trafficapplicationswheredataispartitionedupfront.Forexample,agamingplatformmaybreakgamedataintoisolatedpartitions,eachgivenbyadifferentprefix.Apartitionforagivenplayeriseitherchosenatrandomforeachgameplayorcalculatedbasedontheplayerinformation.

Whilequeryprefixesweredesignedwiththetwoscenariosaboveinmind,theymayalsobeusedinothercircumstances,whichwewillexplorethroughoutthischapter.AlltheexamplesbelowassumeyouareusingPostgres.Otherdatabasesenginesmayrequireslightlydifferentsolutions.

GlobalprefixesAsastartingpoint,let'sstartwithasimplescenario:yourapplicationmustconnecttoaparticularprefixwhenrunninginproduction.Thismaybeduetoinfrastructureconditions,databaseadministrationrulesorothers.

Let'sdefinearepositoryandaschematogetstarted:


24

https://www.postgresql.org/docs/current/static/ddl-schemas.html

#lib/repo.ex

defmoduleMyApp.Repodo

useEcto.Repo,otp_app::my_app

end

#lib/sample.ex

defmoduleMyApp.Sampledo

useEcto.Schema

schema"samples"do

field:name

timestamps

end

end

Nowlet'sconfiguretherepository:

#config/config.exs

config:my_app,MyApp.Repo,

adapter:Ecto.Adapters.Postgres,

username:"postgres",

password:"postgres",

database:"demo",

hostname:"localhost",

pool_size:10

Anddefineamigration:

#priv/repo/migrations/20160101000000_create_sample.exs

defmoduleMyApp.Repo.Migrations.CreateSampledo

useEcto.Migration

defchangedo

createtable(:samples)do

add:name,:string

timestamps()

end

end

end

Nowlet'screatethedatabase,migrateitandthenstartanIExsession:


25

$mixecto.create

$mixecto.migrate

$iex-Smix

InteractiveElixir(1.4.0-dev)-pressCtrl+Ctoexit(typeh()ENTERforhelp)

iex(1)>MyApp.Repo.allMyApp.Sample

[]

Wehaven'tdoneanythingunusualsofar.Wecreatedourdatabaseinstance,madeituptodatebyrunningmigrationsandthensuccessfullymadeaqueryagainstthe"samples"table,whichreturnedanemptylist.

Bydefault,connectionstoPostgres'databasesrunonthe"public"prefix.Whenwerunmigrationsandqueries,theyareallrunningagainstthe"public"prefix.Howeverimagineyourapplicationhasarequirementtorunaparticularprefixinproduction,let'scallit"global_prefix".

LuckilyPostgresallowsustochangetheprefixourdatabaseconnectionsrunonbysettingtheschemasearchpath.Thebestmomenttochangetheschemasearchpathisrightafterwesetupthedatabaseconnection,ensuringallofourquerieswillrunonthatparticularprefix,throughouttheconnectionlife-cycle.

Todoso,let'schangeourdatabaseconfigurationin"config/config.exs"andspecifyan:after_connectoption.:after_connectexpectsatuplewithmodule,functionandargumentsitwillinvokewiththeconnectionprocessassoonasadatabaseconnectionisestablished:

config:my_app,MyApp.Repo,

adapter:Ecto.Adapters.Postgres,

username:"postgres",

password:"postgres",

database:"demo_dev",

hostname:"localhost",

pool_size:10,

after_connect:{Postgrex,:query!,["SETsearch_pathTOglobal_prefix",[]]}

Nowlet'strytorunthesamequeryasbefore:

$iex-Smix



**(Postgrex.Error)ERROR(undefined_table):relation"samples"doesnotexist

Ourpreviouslysuccessfulquerynowfailsbecausethereisnotable"samples"underthenewprefix.Let'strytofixthatbyrunningmigrations:


26

$mixecto.migrate

**(Postgrex.Error)ERROR(invalid_schema_name):noschemahasbeenselectedtocreate

in

Oops.Nowmigrationsaysthereisnosuchschemaname.That'sbecausePostgresautomaticallycreatesthe"public"prefixeverytimewecreateanewdatabase.Ifwewanttouseadifferentprefix,wemustexplicitlycreateitonthedatabasewearerunningon:

$psql-ddemo_dev-c"CREATESCHEMAglobal_prefix"

Nowwearereadytomigrateandrunourqueries:

$mixecto.migrate

$iex-Smix



[]

Dataindifferentprefixesareisolated.Writingtothe"samples"tableinoneprefixcannotbeaccessedbytheotherunlesswechangetheprefixintheconnectionorusingtheEctoconvenienceswewilldiscussbelow.

Per-queryandper-structprefixesWhilestillconfiguredtoconnecttothe"global_prefix"on:after_connect,let'srunsomequeries:

iex(1)MyApp.Repo.allMyApp.Sample

[]

iex(2)MyApp.Repo.insert%MyApp.Sample{name:"mary"}

{:ok,%MyApp.Sample{...}}

iex(3)MyApp.Repo.allMyApp.Sample

[%MyApp.Sample{...}]

Nowwhathappensiftrytorunthesamplequeryonthe"public"prefix?Todoso,let'sbuildaquerystructandsettheprefixfieldmanually:

iex(4)>query=Ecto.Queryable.to_queryMyApp.Sample

#Ecto.Query<fromsinMyApp.Sample>

iex(5)>MyApp.Repo.all%{query|prefix:"public"}

[]


27

Noticehowwewereabletochangetheprefixthequeryrunson.Backinthedefault"public"prefix,thereisnodata!

Ecto2.1alsosupportsthe:prefixoptiononallrelevantrepositoryoperations:


[%MyApp.Sample{...}]

iex(7)>MyApp.Repo.allMyApp.Sample,prefix:"public"

[]

OneinterestingaspectofprefixesinEctoisthattheprefixinformationiscarriedalongeachstruct:

iex(8)[sample]=MyApp.Repo.allMyApp.Sample

[%MyApp.Sample{}]

iex(9)>Ecto.get_meta(sample,:prefix)

nil

Theexampleabovereturnednil,whichmeansnoprefixwasspecifiedbyEcto,andthereforethedatabaseconnectiondefaultwillbeused.Inthiscase,"global_prefix"willbeusedbecauseofthe:after_connectcallbackweaddedatthebeginningofthischapter.

Sincetheprefixdataiscarriedinthestruct,wecanusesuchtocopydatafromoneprefixtotheother.Let'scopythesampleabovefromthe"global_prefix"tothe"public"one:

iex(10)>public_sample=Ecto.put_meta(sample,prefix:"public")

%MyApp.Sample{}

iex(11)>MyApp.Repo.insertpublic_sample

{:ok,%MyApp.Sample{}}

iex(12)>MyApp.Repo.allMyApp.Sample,prefix:"public"

[%MyApp.Sample{}]

Nowwehavedatainsertedinbothprefixes.

Prefixesinqueriesandstructsalwayscascade.Forexample,ifyourunMyApp.Repo.preload(sample,[:some_association]),theassociationwillbequeriedforandloadedinthesameprefixasthesamplestruct.IfsamplehasassociationsandyoucallMyApp.Repo.insert(sample)orMyApp.Repo.update(sample),theassociateddatawillalsobeinserted/updatedinthesameprefixassample.That'sbydesigntofacilitateworkingwithgroupsofdatainthesameprefix,andespeciallybecausedataindifferentprefixesmustbekeptisolated.

Migrationprefixes


28

SofarwehaveexploredhowtosetaglobalprefixusingPostgres'andhowtosettheprefixatthequeryorstructlevel.Whentheglobalprefixisset,italsochangestheprefixmigrationsrunon.Howeveritisalsopossibletosettheprefixthroughthecommandlineorpertableinthemigrationitself.

Forexample,imagineyouareagamingcompanywherethegameisbrokenin128partitions,named"prefix_1","prefix_2","prefix_3"upto"prefix_128".Now,wheneveryouneedtomigratedata,youneedtomigratedataonalldifferent128prefixes.Therearetwowaysofachievethat.

Thefirstmechanismistoinvokemixecto.migratemultipletimes,onceperprefix,passingthe--prefixoption:

$mixecto.migrate--prefix"prefix_1"



...


Theotherapproachisbychangingeachdesiredmigrationtorunacrossmultipleprefixes.Forexample:

defmoduleMyApp.Repo.Migrations.CreateSampledo

useEcto.Migration

defchangedo

fori<-1..128do

prefix="prefix_#{i}"

createtable(:samples,prefix:prefix)do

add:name,:string

timestamps()

end

#Executethecommandsonthecurrentprefix

#beforemovingontothenextprefix

flush()

end

end

end

SchemaprefixesFinally,Ecto2.1addstheabilitytosetaparticularschematorunonaspecificprefix.Imagineyouarebuildingamulti-tenantapplication.Eachclientdatabelongstoaparticularprefix,suchas"client_foo","client_bar"andsoforth.Yetyourapplicationmaystillrelyona


29

setoftablesthatissharedacrossallclients.OneofsuchtablesmaybeexactlythetablethatmapstheClientIDtoitsdatabaseprefix.Let'sassumewewanttostorethisdatainaprefixnamed"main":

defmoduleMyApp.Mappingdo

useEcto.Schema

@schema_prefix"main"

schema"mappings"do

field:client_id,:integer

field:db_prefix

timestamps

end

end

NowrunningMyApp.Repo.allMyApp.Mappingwillbydefaultrunonthe"main"prefix,regardlessofthevalueconfiguredgloballyonthe:after_connectcallback.Similarwillhappentoinsert,update,andsimilaroperations,unlessthe:prefixisexplicitlychangedviaEcto.put_meta/2orbypassingthe:prefixoptiontotherepositoryoperation.

Keepinmind,however,thatqueriesrunonasingleprefix.Forexample,ifMyApp.Mappingonprefix"main"dependsonaschemanamedMyApp.Otheronprefix"another",aquerystartingwithMyApp.Mappingwillalwaysrunonthe"main"prefix.Bydesignitisnotpossibletoperformqueryjoinsacrossprefixes.Ifdatabelongstodifferentprefixes,itisbesttonotcouplethemstructurallynorviaqueries,inordertokeepdataindifferentprefixesisolated.

SummingupEcto2.0providesmanyconveniencesforworkingwithqueryingprefixes.ThoseconvenienceshavebeenfurtherimprovedinEcto2.1,allowingdeveloperstoconfigureprefixwithdifferentlevelofgranularity:

globalprefixes>schemaprefix>query/structprefixes

Thisallowsdeveloperstotackledifferentscenarios,fromproductionrequirementstomulti-tenantapplications.Ourjourneyonexploringthenewqueryconstructsisalmostover.Thenextandlastquerychapterisonaggregatesandsubqueries.


30

AggregatesandsubqueriesThelastfeatureswewilldiscussregardingEctoqueriesareaggregatesandsubqueries.Aswewilllearn,thoseareintrinsicallyrelated.

AggregatesEcto2.0includesaconveniencefunctioninrepositoriestocalculateaggregates.

Forexample,tofindtheaveragenumberofvisitsacrossallposts:

MyApp.Repo.aggregate(MyApp.Post,:avg,:visits)

#=>#Decimal<1743>

Behindthescenes,thequeryabovetranslatesto:

MyApp.Repo.one(frompinMyApp.Post,select:avg(p.visits))

Theaggregate/3functionsupportsanyoftheaggregatefunctionslistedintheEctoQueryAPI.

Atfirst,itlooksliketheimplementationofaggregate/3isquitestraight-forward.YoucouldevenstarttowonderwhyitwasaddedtoEctointhefirstplace.However,complexitiesstarttoariseonqueriesthatrelyonlimit,offsetordistinctclauses.

Imaginethatinsteadofcalculatingtheaverageofallposts,youwanttheaverageofonlythetop10.Youmaytrytoitasfollows:

MyApp.Repo.one(frompinMyApp.Post,

order_by:[desc::visits],

limit:10,

select:avg(p.visits))

#=>#Decimal<1743>

Oops.Thequeryabovereturnedthesamevalueasthequeriesbefore.Theoptionlimit:10hasnoeffectheresinceitislimitingtheaggregatedresultandquerieswithaggregatesreturnonlyasinglerowanyway.Inordertoretrievethecorrectresult,wewouldneedtofirstfindthetop10postsandonlythenaggregate.That'sexactlywhataggregate/3does:


31

https://hexdocs.pm/ecto/Ecto.Query.API.html

query=fromMyApp.Post,order_by:[desc::visits],limit:10

MyApp.Repo.aggregate(query,:avg,:visits)#=>#Decimal<4682>

Whenlimit,offsetordistinctisspecifiedinthequery,aggregate/3automaticallywrapsthegivenqueryinasubquery.Thequeryexecutedbyaggregate/3abovewouldbeequivalentto:


MyApp.Repo.one(fromqinsubquery(query),select:avg(q.visits))

Let'stakeacloserlookatsubqueries.

SubqueriesIntheprevioussectionwehavealreadylearnedsomequeriesthatwouldbehardtoexpresswithoutsupportforsubqueries.That'soneofmanyexamplesthatledtosubquerysupportbeingaddedtoEcto.

SubqueriesinEctoarecreatedbycallingEcto.Query.subquery/1.ThefunctionreceivesanydatastructurethatcanbeconvertedtoaqueryviatheEcto.Queryableprotocolandreturnsasubqueryconstruct(whichisalsoqueryable).

InEcto2.0,itisallowedforasubquerytoselectawholetable(p)orafield(p.field).Allfieldsselectedinasubquerycanbeaccessedfromtheparentquery.Let'srevisittheaggregatequerywesawintheprevioussection:


MyApp.Repo.one(fromqinsubquery(query),select:avg(q.visits))

Becausethequerydoesnotspecifya:selectclause,itwillreturnselect:pwherepiscontrolledbyMyApp.Postschema.SincethequerywillreturnallfieldsinMyApp.Post,whenweconvertittoasubquery,allofthefieldsfromMyApp.Postwillbeavailableontheparentquery,suchasq.visits.Infact,Ectowillkeeptheschemapropertiesacrossqueries.Forexample,ifyouwriteq.field_that_does_not_exist,yourEctoquerywon'tcompile.

Ecto2.1furtherimprovessubqueriesbyallowinganElixirmaptobereturnedfromasubquery,makingthemapfieldsdirectlyavailabletotheparentquery.

Let'sseeonelastexample.Imagineyoumanagealibrary(anactuallibraryintherealworld)andthereisatablethatlogseverytimethelibrarylendsabook.The"lendings"tableusesauto-incrementingindexesandcanbebackedbythefollowingschema:


32

defmoduleLibrary.Lendingdo

useEcto.Schema

schema"lendings"do

belongs_to:book,MyApp.Book#definesbook_id

belongs_to:visitor,MyApp.Visitor#definesvisitor_id

end

end

Nowconsiderwewanttoretrievethenameofeverybookalongsidethenameofthelastpersonthelibraryhaslentitto.Todoso,weneedtofindthelastlendingIDofeverybook,andthenjoinonthebookandvisitortables.Withsubqueries,that'sstraight-forward:

last_lendings=

fromlinMyApp.Lending,

group_by:l.book_id,

select:%{book_id:l.book_id,last_lending_id:max(l.id)}

fromlinLending,

join:lastinsubquery(last_lendings),

on:last.last_lending_id==l.id,

join:binassoc(l,:book),

join:vinassoc(l,:visitor),

select:{b.name,v.name}

SubqueriesareanimportantimprovementtoEctowhichmakesitpossibletoexpressqueriesthatwerenotpossiblebefore.Ontopofthat,wewereabletoaddfeaturessuchasaggregates,whichprovideusefulfunctionalitywhileshieldingtheuserfromcornercases.


33

ImprovedassociationsandfactoriesEcto2.0largelyimprovedhowassociationswork.Tounderstandwhyandhow,let'stalkaboutEcto'soriginaldesigngoals.

EctofirststartedasaSummerofCodeprojectfromEricMeadows-Jönsson,todaypartoftheElixirteamandcreatorofHex,withJoséValim,creatorofElixir,asmentor.Thiswasbackin2013andElixirwasstillatversion0.9!

Similartomanyprojectsatthetime,oneofthegoalsbehindEctowastovalidateElixiritselfasaprogramminglanguage.Oneofthequestionsitaimedtoanswerwas:"canElixirbeusedtocreateadatabasewrapperthatisperformantandsecure?".Bybecomingastable,performantandsecurefoundation,Ectowouldbeabletoaddsyntaxsugar,conveniencesanddynamismlateron-whiletheoppositedirectionwouldhavebeenexceptionallyhardintheexperienceoftheEctoteam.

Ecto1.0becamethissecureandperformantfoundation.Asaconsequence,itfeltrigidinmanyaspectsandthosewerefrequentlyreportedaslimitationsbythecommunity.

Ecto2.0improvesonthemistakesmadebyEcto1.0andthenbuildsontopofitsfoundationbyaddingtheflexibilitiesusershavelongedfor.Wehaveexploredmanyoftheminthefirstchaptersofthisbook:schemalessqueries,schemalesschangesets,dynamicqueriesandmore.Inthenextthreechapters,wewillexploretheenhancementsdonetoschemasandassociations.

Inthisparticularchapter,wewilllearnhowEctoiscapableofinsertingcomplexdatastructureswithouttheneedtousechangesetsandhowtousethisfeaturetomanagecomplexdata,incasessuchasyourapplicationtestsuite,withoutaneedtorelyonexternalprojects.

LesschangesetsAtthesametimeEcto2.0bringsmanyfeaturestochangesets,itmakeschangesetslessnecessarythroughoutEctoAPIs.Forexample,inEcto1.0,Ecto.Repo.insert/2requiredchangesets.Thismeansthat,inordertoinsertanyentrytothedatabase,suchasapost,wehadtowrapitinachangesetfirst:

%Post{title:"helloworld"}

|>Ecto.Changeset.change

|>Repo.insert!()


34

ThisreflectedthroughoutEcto1.0APIs.Ifyouwantedtocreateapostwithsomecomments,youhadtowrapeachcommentinachangesetandthenputitinthepostchangeset:

comment1=%Comment{body:"excellentarticle"}|>Ecto.Changeset.change

comment2=%Comment{body:"Ilearnedsomethingnew"}|>Ecto.Changeset.change

%Post{title:"helloworld"}

|>Ecto.Changeset.put_assoc(:comments,[comment1,comment2])

|>Repo.insert!()

Furthermore,whenhandlingassociations,Ecto1.0forcedyoutoalwayswritetheparentchangesetfirstandthenthechildren.Sotheexampleabovewhereinsertapost(theparent)withmultiplecomments(children)workedbutthefollowingexamplewouldnot:

post=%Post{title:"helloworld"}|>Ecto.Changeset.change()

%Comment{body:"excellentarticle"}

|>Ecto.Changeset.put_assoc(:post,[post])

|>Repo.insert!()

Ecto2.0goesawaywithallofthoselimitations.YoucannowpassstructstotherepositoryandchangesetsandEctowilltakecareofbuildingthechangesetsforyoubehindthescenes.InEcto2.0,apostwithcommentscanbeinserteddirectlyasfollows:

Repo.insert!%Post{

title:"helloworld",

comments:[

%Comment{body:"excellentarticle"},

%Comment{body:"Ilearnedsomethingnew"}

]

}

Youarealsoabletoinsertandupdateassociationsfromanydirection,beitfromparenttochildorchildtoparent:

Repo.insert!%Comment{

body:"excellentarticle",

post:%Post{title:"helloworld"}

}

Thisfeatureisnotonlyusefulwhenwritingourapplicationsbutalsowhentestingthem,aswewillseenext.


35

TestfactoriesManyprojectsdependonexternallibrariestobuildtheirtestdata.Someofthoselibrariesarecalledfactoriesbecausetheyprovideconveniencefunctionsforbuildingdifferentgroupsofdata.However,givenEcto2.0isabletomanagecomplexdatatrees,wecanimplementsuchfunctionalitywithoutrelyingonthird-partyprojects.

Togetstarted,let'screateafileat"test/support/factory.ex"withthefollowingcontents:

defmoduleMyApp.Factorydo

aliasMyApp.Repo

#Factories

defbuild(:post)do

%MyApp.Post{title:"helloworld"}

end

defbuild(:comment)do

%MyApp.Comment{body:"goodpost"}

end

defbuild(:post_with_comments)do

%MyApp.Post{

title:"hellowithcomments",

comments:[

build(:comment,body:"first"),

build(:comment,body:"second")

]

}

end

defbuild(:user)do

%MyApp.User{

email:"hello#{System.unique_integer()}",

username:"hello#{System.unique_integer()}"

}

end

#ConvenienceAPI

defbuild(factory_name,attributes)do

factory_name|>build()|>struct(attributes)

end

definsert!(factory_name,attributes\\[])do

Repo.insert!build(factory_name,attributes)

end

end


36

Ourfactorymoduledefinesfour"factories"asdifferentclausestothebuildfunction::post,:comment,:post_with_commentsand:user.Eachclausedefinestructswiththefieldsthatrequiredbythedatabase.Incertaincases,thegeneratedstructalsoneedtogenerateuniquefields,suchastheuser'semailandusername.WedidsobycallingElixir'sSystem.unique_integer()-youcouldcallSystem.unique_integer([:positive])ifyouneedastrictlypositivenumber.

Attheend,wedefinedtwofunctions,build/2andinsert!/2,whichareconveniencesforbuildingstructswithspecificattributesorforinsertingdatadirectlyintherepository.

That'sliterallyallthatisnecessaryforbuildingourfactories.Wearenowreadytousefactoriesinourtests.First,openupyour"mix.exs"fileandlet'smakesurethe"test/support/factory.ex"fileiscompiled:

defprojectdo

[...,

elixirc_paths:elixirc_paths(Mix.env),

...]

end

defpelixirc_paths(:test),do:["lib","test/support"]

defpelixirc_paths(_),do:["lib"]

Nowinanyoftheteststhatneedtogeneratedata,wecanimporttheMyApp.Factorymoduleanduseitsfunctions:

importMyApp.Factory

build(:post)

#=>%MyApp.Post{id:nil,title:"helloworld",...}

build(:post,title:"customtitle")

#=>%MyApp.Post{id:nil,title:"customtitle",...}

insert!(:post,title:"customtitle")

#=>%MyApp.Post{id:...,title:"customtitle"}

BybuildingthefunctionalityweneedontopofEctocapabilities,weareabletoextendandimproveourfactoriesonwhateverwaywedesire,withoutbeingconstrainedtothird-partylimitations.


37

ManytomanyandcastingBesidesbelong_to,has_many,has_oneand:throughassociations,Ecto2.0alsoincludesmany_to_many.many_to_manyrelationships,asthenamesays,allowsXtohavemanyassocaitedentriesYandvice-versa.Althoughmany_to_manyassociationscanbewrittenasahas_many:through,usingmany_to_manymayconsiderablysimplifysomeworkflows.

Inthischapter,wewilltalkaboutpolymorphicassociationsandhowmany_to_manycanconsiderablyremoveboilerplatefromcertainapproachescomparedtohas_many:through.

Todolistsv65131Thewebhasseenitsshareoftodolistapplications.Butthatwon'tstopusfromcreatingourown!

Inourcase,thereisoneaspectoftodolistapplicationsweareinterestedon,whichistherelationshipthetodolistwhichhasmanytodoitems.WehaveexploredthisexactscenarioindetailinanarticlewepostedonPlataformatec'sblogaboutnestedassociationsandembeds.Let'srecaptheimportantpoints.

Inordertomodelatodolistapp,weneedtohavetwoschemas,Todo.ListandTodo.Item:

defmoduleMyApp.TodoListdo

useEcto.Schema

schema"todo_lists"do

field:title

has_many:todo_items,MyApp.TodoItem

timestamps()

end

end

defmoduleMyApp.TodoItemdo

useEcto.Schema

schema"todo_items"do

field:description

timestamps()

end

end


38

http://blog.plataformatec.com.br/2015/08/working-with-ecto-associations-and-embeds/

OneofthewaystointroduceatodolistwithmultipleitemsintothedatabaseistocoupleourUIrepresentationtoourschemas.That'stheapproachwetookintheblogpostabovewithPhoenix.Roughly:

<%=form_for@todo_list_changeset,todo_list_path(@conn,:create),fnf->%>

<%=text_inputf,:title%>

<%=inputs_forf,:todo_items,fni->%>

...

<%end%>

<%end%>

WhensuchaformissubmittedinPhoenix,itwillsendparameterswiththefollowingshape:

%{"todo_list"=>%{

"title"=>"shippinglist",

"todo_items"=>%{

0=>%{"description"=>"bread"},

1=>%{"description"=>"eggs"},

}

}}

WecouldthenretrievethoseparametersandpassittoanEctochangesetandEctowouldautomaticallyfigureoutwhattodo:

#InMyApp.TodoList

defchangeset(struct,params\\%{})do

struct

|>Ecto.Changeset.cast(params,[:title])

|>Ecto.Changeset.cast_assoc(:todo_list_items,required:true)

end

#AndtheninMyApp.TodoItem


struct

|>Ecto.Changeset.cast(params,[:description])

end

BycallingEcto.Changeset.cast_assoc/3,Ectowilllookfora"todo_items"keyinsidetheparametersgivenoncast,andcomparethoseparameterswiththeitemsstoredinthetodoliststructs.Ectowillautomaticallygenerateinstructionstoinsert,updateordeletetodoitemssuchthat:

ifatodoitemsentasparameterhasanIDanditmatchesanexistingassociatedtodoitem,weconsiderthattodoitemshouldbeupdatedifatodoitemsentasparameterdoesnothaveanID(noramatchingID),weconsider


39

thattodoitemshouldbeinsertedifatodoitemiscurrenetlyassociatedbutitsIDwasnotsentasparameter,weconsiderthetodoitemisbeingreplacedandweactaccordingtothe:on_replacecallback.Bydefault:on_replacewillraisesoyouchooseabehaviourbetweenreplacing,deleting,ignoringornilifyingtheassociation

Theadvantageofusingcast_assoc/3isthatEctoisabletodoallofthehardworkofkeepingtheentriesassociated,aslongaswepassthedataexactlyintheformatthatEctoexpects.However,aswelearnedinthefirstthreechaptersofthisbook,suchapproachisnotalwayspreferrableandinmanysituationsitisbettertodesignourassociationsdifferentlyordecoupleourUIsfromourdatabaserepresentation.

PolymorphictodoitemsToshowanexampleofwhereusingcast_assoc/3isjusttoocomplicatedtobeworthit,let'simagineyouwantyour"todoitems"tobepolymorphic.Forexample,youwanttobeabletoaddtodoitemsnotonlyto"todolists"buttomanyotherpartsofyourapplication,suchasprojects,dates,younameit.

Firstofall,itisimportanttorememberEctodoesnotprovidethesametypeofpolymorphicassociationsknowninframeworkssuchasRailsandLaravel.Insuchframeworks,apolymorphicassociationusestwocolumns,theparent_idandparent_type.Forexample,onetodoitemwouldhaveparent_idof1withparent_typeof"TodoList"whileanotherwouldhaveparent_idof1withparent_typeof"Project".

Theissuewiththedesignaboveisthatitbreaksdatabasereferences.Thedatabaseisnolongercapableofguaranteeingtheitemyouassociatetoexistsorwillcontinuetoexistinthefuture.Thisleadstoaninconsistentdatabasewhichend-uppushingworkaroundstoyourapplication.

Thedesignaboveisalsoextremelyinefficient.Inthepastwehaveworkedwithalargeclientonremovingsuchpolymorphicreferencesbecausefrequentpolymorphicqueriesweregrindingthedatabasetoahaltevenafteraddingindexesandoptimizingthedatabase.

Luckily,thedocumentationforthebelongs_tomacroincludesexamplesonhowtodesignsaneandperformantassociations.Oneofthoseapproachesconsistsinusingmanyjointables.Besidesthe"todo_lists"and"projects"tablesandthe"todo_items"table,wewouldcreate"todo_list_items"and"project_items"toassociatetodoitemstotodolistsandtodoitemstoprojectsrespectively.Intermsofmigrations,wearelookingatthefollowing:


40

https://hexdocs.pm/ecto/Ecto.Schema.html#belongs_to/3-polymorphic-associations

createtable("todo_lists")do

add:title

timestamps()

end

createtable("projects")do

add:name

timestamps()

end

createtable("todo_items")do

add:description

timestamps()

end

createtable("todo_lists_items")do

add:todo_item_id,references(:todo_items)

add:todo_list_id,references(:todo_lists)

timestamps()

end

createtable("projects_items")do


add:project_id,references(:projects)

timestamps()

end

Firstlet'sseehowimplementthisfunctionalityusingahas_many:throughandthenusemany_to_manytoremovealotoftheboilerplatewewereforcedtointroduce.

Polymorphismwithhas_many:throughGivenwewantourtodoitemstobepolymorphic,wecannolongerassociateatodolisttotodoitemsdirectly.InsteadwewillcreateanintermediateschematotieMyApp.TodoListandMyApp.TodoItemtogether.


41


useEcto.Schema


field:title

has_many:todo_list_items,MyApp.TodoListItem

has_many:todo_items,through:[:todo_list_items,:todo_item]

timestamps()

end

end

defmoduleMyApp.TodoListItemdo

useEcto.Schema

schema"todo_list_items"do

belongs_to:todo_list,MyApp.TodoList

belongs_to:todo_item,MyApp.TodoItem

timestamps()

end

end


useEcto.Schema


field:description

timestamps()

end

end

AlthoughweintroducedMyApp.TodoListItemasanintermediateschema,has_many:throughallowsustoaccessalltodoitemsforanytodolisttransparently:

todo_lists|>Repo.preload(:todo_items)

Thetroubleisthat:throughassociationsareread-onlysinceEctodoesnothaveenoughinformationtofillintheintermediateschema.Thismeansthat,ifwestillwanttousecast_assoctoinsertatodolistwithmanytodoitemsdirectlyfromtheUI,wewouldneedtofirstcast_assoc(:todo_list_items)fromTodoListandthencallcast_assoc(:todo_item)fromtheTodoListItemschema:


42

#InMyApp.TodoList


struct


|>Ecto.Changeset.cast_assoc(:todo_list_items,required:true)

end

#AndthenintheMyApp.TodoListItem


struct

|>Ecto.Changeset.cast_assoc(:todo_item,required:true)

end



struct


end

Tofurthercomplicatethings,remembercast_assocexpectsaparticularshapeofdatathatreflectsyourassociations.Inthiscase,becauseoftheintermediateschema,thedatasentthroughyourformsinPhoenixwouldhavetolookasfollows:

%{"todo_list"=>%{


"todo_list_items"=>%{

0=>%{"todo_item"=>%{description"=>"bread"}},

1=>%{"todo_item"=>%{description"=>"eggs"}},

}

}}

Tomakemattersworse,youwouldhavetoduplicatethislogicforeveryintermediateschema,bydefiningMyApp.TodoListItemfortodolists,MyApp.ProjectItemforprojects,etc.

Luckily,many_to_manyallowsustoremoveallofthisboilerplate.

Polymorphismwithmany_to_manyInaway,theideabehindmany_to_manyassociationsisthatitallowsustoassociatetwoschemasviaanintermediateschemaswhileautomaticallytakingcareofalldetailsabouttheintermediateschema.Let'srewritetheschemasabovetousemany_to_many:


43


useEcto.Schema


field:title

many_to_many:todo_items,join_through:MyApp.TodoListItem

timestamps()

end

end

defmoduleMyApp.TodoListItemdo

useEcto.Schema

schema"todo_list_items"do

belongs_to:todo_list,MyApp.TodoList

belongs_to:todo_item,MyApp.TodoItem

timestamps()

end

end


useEcto.Schema


field:description

timestamps()

end

end

NoticeMyApp.TodoListnolongerneedstodefineahas_manyassociationpointingtotheMyApp.TodoListItemschemaandinsteadwecanjustassociateto:todo_itemsusingmany_to_many.

Differentlyfromhas_many:through,many_to_manyassociationsarealsowriteable.Thismeanswecansenddatathroughourformsexactlyaswedidatthebeginningofthischapter:

%{"todo_list"=>%{


"todo_items"=>%{

0=>%{"description"=>"bread"},

1=>%{"description"=>"eggs"},

}

}}

Andwenolongerneedtodefineachangesetfunctionintheintermediateschema:


44

#InMyApp.TodoList


struct


|>Ecto.Changeset.cast_assoc(:todo_items,required:true)

end



struct


end

Inotherwords,wecanuseexactlythesamecodewehadinthe"todolistshas_manytodoitems"case.Soevenwhenexternalconstraintsrequireustouseajointable,many_to_manyassociationscanautomaticallymanagethemforus.Everythingyouknowaboutassociationswilljustworkwithmany_to_manyassociations,includingtheimprovementswediscussedinthepreviouschapter.

Finally,eventhoughwehavespecifiedaschemaasthe:join_throughoptioninmany_to_many,many_to_manycanalsoworkwithoutintermediateschemasaltogetherbysimplygivingitatablename:


useEcto.Schema


field:title

many_to_many:todo_items,join_through:"todo_list_items"

timestamps()

end

end

Inthiscase,youcancompletelyremovetheMyApp.TodoListItemschemafromyourapplicationandthecodeabovewillstillwork.Theonlydifferenceisthatwhenusingtables,anyautogeneratedvaluethatisfilledbyEctoschema,suchastimestamps,won'tbefilled(aswenolongerhaveaschema).Tosolvethis,youcaneitherdropthosefieldsfromyourmigrationsorsetadefaultatthedatabaselevel.

SummaryInthischapterweusedmany_to_manyassociationstodrasticallyimproveapolymorphicassociationdesignthatreliedonhas_many:through.Ourgoalwastoallow"todo_items"toassociatetodifferententitiesinourcodebase,suchas"todo_lists"and"projects".Wehave


45

donethisbycreatingintermediatetablesandbyusingmany_to_manyassociationstoautomaticallymanagethosejointables.

Attheend,ourschemasmaylooklike:


useEcto.Schema


field:title

many_to_many:todo_items,join_through:"todo_list_items"

timestamps()

end


struct



end

end

defmoduleMyApp.Projectdo

useEcto.Schema


field:name

many_to_many:todo_items,join_through:"project_items"

timestamps()

end


struct

|>Ecto.Changeset.cast(params,[:name])


end

end


useEcto.Schema


field:description

timestamps()

end


struct


end

end


46

Andthedatabasemigration:

createtable("todo_lists")do

add:title

timestamps()

end

createtable("projects")do

add:name

timestamps()

end

createtable("todo_items")do

add:description

timestamps()

end

#Primarykeyandtimestampsarenotrequiredifusingmany_to_manywithoutschemas

createtable("todo_lists_items",primary_key:false)do


add:todo_list_id,references(:todo_lists)

#timestamps()

end

#Primarykeyandtimestampsarenotrequiredifusingmany_to_manywithoutschemas

createtable("projects_items",primary_key:false)do


add:project_id,references(:projects)

#timestamps()

end

Overallourcodelooksstructurallythesameashas_manywould,althoughatthedatabaselevelourrelationshipsareexpressedwithjointables.

Whileinthischapterwechangedourcodetocopewiththeparameterformatrequiredbycast_assoc,inthenextchapterwewilldropcast_assocaltogetheranduseput_assocwhichbringsmoreflexibilitieswhenworkingwithassociations.


47

ManytomanyandupsertsInthepreviouschapterwehavelearnedaboutmany_to_manyassociationsandhowtomapexternaldatatoassociatedentrieswiththehelpofEcto.Changeset.cast_assoc/3.Whileinthepreviouschapterwewereabletofollowtherulesimposedbycast_assoc/3,doingsoisnotalwayspossiblenordesired.

Inthischapter,wearegoingtolookatEcto.Changeset.put_assoc/4incontrasttocast_assoc/3andexploresomeexamples.WewillalsopeekattheupsertfeaturescominginEcto2.1.

put_assocvscast_assocImaginewearebuildinganapplicationthathasblogpostsandsuchpostsmayhavemanytags.Notonlythat,agiventagmayalsobelongtomanyposts.Thisisaclassicscenariowherewewouldusemany_to_manyassociations.Ourmigrationswouldlooklike:

createtable(:posts)do

add:title

add:body

timestamps()

end

createtable(:tags)do

add:name

timestamps()

end

createunique_index(:tags,[:name])

createtable(:posts_tags,primary_key:false)do

add:post_id,references(:posts)

add:tag_id,references(:tags)

end

Noteweaddedauniqueindextothetagnamebecausewedon'twanttohaveduplicatedtagsinourdatabase.Itisimportanttoaddanindexatthedatabaselevelinsteadofusingavalidationsincethereisalwaysachancetwotagswiththesamenamewouldbevalidatedandinsertedsimultaneously,passingthevalidationandleadingtoduplicatedentries.


48

https://hexdocs.pm/ecto/Ecto.Schema.html#many_to_many/3

https://hexdocs.pm/ecto/Ecto.Changeset.html#cast_assoc/3

https://hexdocs.pm/ecto/Ecto.Changeset.html#put_assoc/4

Nowlet'salsoimaginewewanttheuserinputsuchtagsasalistofwordssplitbycomma,suchas:"elixir,erlang,ecto".Oncethisdataisreceivedintheserver,wewillbreakitapartintomultipletagsandassociatethemtothepost,creatinganytagthatdoesnotyetexistinthedatabase.

Whiletheconstraintsabovesoundreasonable,that'sexactlywhatputusintroublewithcast_assoc/3.Rememberthecast_assoc/3changesetfunctionwasdesignedtoreceiveexternalparametersandcomparethemwiththeassociateddatainourstructs.Todosocorrectly,Ectorequirestagstobesentasalistofmaps.Howeverhereweexpecttagstobesentinastringseparatedbycomma.

Furthermorecast_assoc/3reliesontheprimarykeyfieldforeachtagsentinordertodecideifitshouldbeinserted,updatedordeleted.Again,becausetheuserissimplypassingastring,wedon'thavetheIDinformationathand.

Whenwecan'tcopewithcast_assoc/3,itistimetouseput_assoc/4.Input_assoc/4,wegiveEctostructsorchangesetsinsteadofparameters,givingustheabilitytomanipulatethedataaswewant.Let'sdefinetheschemaandthechangesetfunctionforapostwhichmayreceivetagsasastring:


49

defmoduleMyApp.Postdo

useEcto.Schema

schema"posts"do

add:title

add:body

many_to_many:tags,MyApp.Tag,join_through:"posts_tags"

timestamps()

end


struct

|>Ecto.Changeset.cast(struct,[:title,:body])

|>Ecto.Changeset.put_assoc(:tags,parse_tags(params))

end

defpparse_tags(params)do

(params["tags"]||"")

|>String.split(",")

|>Enum.map(&String.trim/1)

|>Enum.reject(&&1=="")

|>Enum.map(&get_or_insert_tag/1)

end

defpget_or_insert_tag(name)do

Repo.get_by(MyApp.Tag,name:name)||

Repo.insert!(MyApp.Tag,%Tag{name:name})

end

end

Inthechangesetfunctionabove,wemovedallthehandlingoftagstoaseparatefunction,calledparse_tags/1,whichchecksfortheparameter,breaksitsentriesapartviaString.split/2,thenremovesanyleftoverwhitespacewithString.trim/1,rejectsanyemptystringandfinallychecksifthetagexistsinthedatabaseornot,creatingoneincasenoneexists.

Theparse_tags/1functionisgoingtoreturnalistofMyApp.Tagstructswhicharethenpassedtoput_assoc/3.Bycallingput_assoc/3,wearetellingEctothoseshouldbethetagsassociatedtothepostfromnowon.Incaseaprevioustagwasassociatedtothepostandnotgiveninput_assoc/3,Ectowillalsotakecareofremovingtheassociationbetweenthepostandtheremovedtagfromthedatabase.

Andthat'sallweneedtousemany_to_manyassociationswithput_assoc/3.put_assoc/3workswithhas_many,belongs_toandallothersassociationtypes.However,ourcodeisnotyetreadyforproduction.Let'sseewhy.


50

ConstraintsandraceconditionsRememberweaddedauniqueindextothetag:namecolumnwhencreatingthetagstable.Wedidsotoprotectusfromhavingduplicatetagsinthedatabase.

Byaddingtheuniqueindexandthenusingget_bywithainsert!togetorinsertatag,weintroducedapotentialerrorinourapplication.Iftwopostsaresubmittedatthesametimewithasimilartag,thereisachancewewillcheckifthetagexistsatthesametime,leadingbothsubmissionstobelievethereisnosuchtaginthedatabase.Whenthathappens,onlyoneofthesubmissionswillsucceedwhiletheotheronewillfail.That'saracecondition:yourcodewillerrorfromtimetotime,onlywhencertainconditionsaremet.Andthoseconditionsaretimesensitive.

Manydevelopershaveatendencytothinksucherrorswon'thappeninpracticeor,iftheyhappened,theywouldbeirrelevant.Butinpracticetheyoftenleadtoveryfrustratinguserexperiences.Ihaveheardafirst-handexamplecomingfromamobilegamecompany.Inthegame,aplayerisabletoplayquestsandoneveryquestyouhavetochooseaguestcharacterfromanotherplayeroutofashortlisttogoonthequestwithyou.Attheendofthequest,youhavetheoptiontoaddtheguestcharacterasafriend.

Originallythewholeguestlistwasrandombut,astimepassed,playersstartedtocomplainsometimesoldaccounts,ofteninactive,werebeingshownintheguestsoptionslist.Toimprovethesituation,thegamedevelopersstartedtosorttheguestlistbymostrecentlyactive.Thismeansthat,ifyouhavejustplayedrecently,thereisahigherchanceofyoutobeonsomeoneguestlists.

However,whentheydidsuchchange,manyerrorsstartedtoshowupandusersweresuddenlyfuriousinthegameforum.That'sbecausewhentheysortedplayersbyactivity,assoontwoplayersloggedin,theircharacterswouldlikelyappearoneachothersguestlist.Ifthoseplayerspickedeachotherscharacters,thefirsttoaddtheotherasfriendattheendofaquestwouldbeabletosucceedbutanerrorwouldappearwhenthesecondplayertriedtoaddthatcharacterasafriendsincetherelationshipalreadyexistedinthedatabase!Notonlythat,alltheprogressdoneinthequestwouldbelost,becausetheserverwasunabletoproperlypersistthequestresultstothedatabase.Understandably,playersstartedtofilecomplaints.

Longstoryshort:wemustaddresstheracecondition.

LuckilyEctogivesusamechanismtohandleconstrainterrorsfromthedatabase.

Checkingforconstrainterrors


51

Sinceourget_or_insert_tag(name)functionfailswhenatagalreadyexistsinthedatabase,weneedtohandlesuchscenariosaccordingly.Let'srewriteittakingraceconditionsintoaccountinmind:


%Tag{}

|>Ecto.Changeset.change(name:name)

|>Ecto.Changeset.unique_constraint(:name)

|>Repo.insert

|>casedo

{:ok,tag}->tag

{:error,_}->Repo.get_by!(MyApp.Tag,name:name)

end

end

Insteadofinsertingthetagdirectly,weknowbuildachangeset,whichallowsustousetheunique_constraintannotation.NowiftheRepo.insertoperationfailsbecausetheuniqueindexfor:nameisviolated,Ectowon'traise,butreturnan{:error,changeset}tuple.Therefore,iftheRepo.insertsucceeds,itisbecausethetagwassaved,otherwisethetagalreadyexists,whichwethenfetchwithRepo.get_by!.

Whilethemechanismabovefixestheracecondition,itisaquiteexpensiveone:weneedtoperformtwoqueriesforeverytagthatalreadyexistsinthedatabase:the(failed)insertandthentherepositorylookup.Giventhat'sthemostcommonscenario,wemaywanttorewriteittothefollowing:


Repo.get_by(MyApp.Tag,name:name)||maybe_insert_tag(name)

end

defpmaybe_insert_tag(name)do

%Tag{}

|>Ecto.Changeset.change(name:name)

|>Ecto.Changeset.unique_constraint(:name)

|>Repo.insert

|>casedo

{:ok,tag}->tag

{:error,_}->Repo.get_by!(MyApp.Tag,name:name)

end

end

Theaboveperforms1queryforeverytagthatalreadyexists,2queriesforeverynewtagandpossibly3queriesinthecaseofraceconditions.Whiletheabovewouldperformslightlybetteronaverage,Ecto2.1hasabetteroptioninstock.


52

UpsertsEcto2.1supportstheso-called"upsert"commandwhichisanabbreviationfor"updateorinsert".Theideaisthatwetrytoinsertarecordandincaseitconflictswithanexistingentry,forexampleduetoauniqueindex,wecanchoosehowwewantthedatabasetoactbyeitherraisinganerror(thedefaultbehaviour),ignoringtheinsert(noerror)orbyupdatingtheconflictingdatabaseentries.

"upsert"inEcto2.1isdonewiththe:on_conflictoption.Let'srewriteget_or_insert_tag(name)oncemorebutthistimeusingthe:on_conflictoption.Alsorememberthat"upsert"isanewfeatureinPostgreSQL9.5,somakesureyouareuptodate:


Repo.insert!(%MyApp.Tag{name:name},on_conflict::nothing)

end

Andthat'sit!Wetrytoinsertatagwiththegivennameandifsuchtagalreadyexists,wetellEctotonoterrorandreturnthetagwehavepassedasargumentasifitwaspersisted.Whiletheaboveiscertainlyastepupfromallsolutionssofar,itstillperformsonequerypertag.If10tagsaresent,wewillperform10queries.Canwefurtherimproveit?

Upsertsandinsert_allEcto2.1didnotonlyaddthe:on_conflictoptiontoRepo.insert/2butalsototheRepo.insert_all/3functionintroducedinEcto2.0.Thismeanswecanbuildonequerythatattemptstoinsertallmissingtagsandthenanotherquerythatfetchesallofthematonce.Let'sseehowourPostschemawilllooklikeafterthosechanges:


53


useEcto.Schema

#Schemaisthesame

schema"posts"do

add:title

add:body


timestamps()

end

#Changesetisthesame


struct



end

#Parsetagshasslightlychanged



|>String.split(",")



|>insert_and_get_all()

end

defpinsert_and_get_all([])do

[]

end

defpinsert_and_get_all(names)do

maps=Enum.map(names,&%{name:&1})

Repo.insert_allMyApp.Tag,names,on_conflict::nothing

Repo.allfromtinMyApp.Tag,where:t.namein^names

end

end

Insteadofattemptingtogetandinserteachtagindividually,thecodeaboveworkonalltagsatonce,firstbybuildingalistofmapswhichisgiventoinsert_allandthenbylookingupalltagswiththeexistingnames.Therefore,regardlessofhowmanytagsaresent,wewillperformonly2queries(unlessnotagissent,inwhichwereturnanemptylistbackpromptly).ThissolutionisonlypossibleinEcto2.1thankstothe:on_conflictoption,whichguaranteesinsert_allwon'tfailincaseagiventagnamealreadyexists.

Finally,keepinmindthatwehaven'tusedtransactionsinanyoftheexamplessofar.Suchdecisionwasdeliberate.Sincegettingorinsertingtagsisanidempotentoperation,i.e.wecanrepeatitmanytimesanditwillalwaysgiveusthesameresultback.Therefore,evenifwefailtointroducetheposttothedatabaseduetoavalidationerror,theuserwillbefreeto


54

resubmittheformandwewilljustattempttogetorinsertthesametagsonceagain.Thedownsideofthisapproachisthattagswillbecreatedevenifcreatingthepostfails,whichmeanssometagsmaynothavepostsassociatedtothem.Incasethat'snotdesired,thewholeoperationcouldbewrappedinatransactionormodeledwiththeEcto.Multiabstractionwewilldiscussinthenextchapter.


55

https://hexdocs.pm/ecto/Ecto.Multi.html

ComposabletransactionswithEcto.MultiEctoreliesondatabasetransactionswhenmultipleoperationsmustbeperformedatomically.TransactionscanbeperformedviatheRepo.transactionfunction:

Repo.transaction(fn->

mary|>Ecto.Changeset.change(balance:mary.balance-10)|>Repo.update!

john|>Ecto.Changeset.change(balance:john.balance+10)|>Repo.update!

end)

Whenweexpectbothoperationstosucceed,asabove,transactionsarequitestraight-forward.However,transactionsgetmorecomplicatediftheoperationscanfail:


casemary|>Ecto.Changeset.change(balance:mary.balance-10)|>Repo.updatedo

{:ok,mary}->

casejohn|>Ecto.Changeset.change(balance:john.balance+10)|>Repo.updatedo

{:ok,john}->

{mary,john}

{:error,changeset}->

Repo.rollback({:john,changeset})

end

{:error,changeset}->

Repo.rollback({:mary,changeset})

end

end)

Ontheotherhand,transactionsinEctocanbenestedarbitrarily.Forexample,imaginethetransactionaboveismovedintoitsownfunction,definedastransfer_money(mary,john,10),andbesidestransferringmoneywealsowanttologthetransfer:


casetransfer_money(mary,john,10)do

{:ok,{mary,john}}->

Repo.insert!(%Transfer{from:mary.id,to:john.id,amount:10})

{:error,{who,changeset}}->

Repo.rollback({who,changeset})

end

end)

Thesnippetaboverunsinatransactionandthencallstransfer_money/3thatalsorunsinatransaction.ThisworksbecauseEctoconvertsanynestedtransactionintosavepointsautomatically.Incaseaninnertransactionfails,itrollsbacktoitsspecificsavepoint.


56

Whilenestingtransactionscanbeimprovethecodereadabilitybybreakingalargetransactionsintomultiplesmallertransactions,thereisstillalotofboilerplateinvolvedinhandlingthesuccessandfailurescenarios.Furthermore,compositionisquitelimited,asalloperationsmuststillbeperformedinsidetransactionblocks.

Amoredeclarativeapproachwouldbetodefinealloperationswewanttoperforminatransactiondecoupledfromthetransactionexecution.Thiswaywewouldbeabletocomposetransactionsoperationswithoutworryingaboutitsexecutioncontextorabouteachindividualsuccess/failurescenario.That'sexactlywhatEcto.Multiallowsustobuild.

ComposingwithdatastructuresLet'srewritethesnippetsaboveusingEcto.Multi.Thefirstsnippetthattransfersmoneybetweenmaryandjohncanrewrittento:

Ecto.Multi.new

|>Ecto.Multi.update(:mary,Ecto.Changeset.change(mary,balance:mary.balance-10))

|>Ecto.Multi.update(:john,Ecto.Changeset.change(john,balance:john.balance+10))

Ecto.Multiisadatastructurethatallowsustodefinewhichoperationsmustbeperformedtogether,withoutworryingaboutwhereandhowitwillbeexecuted.Ecto.MultimirrorsmostoftheEcto.RepoAPI,withthedifferenceeachoperationmustbeexplicitlynamed.Intheexampleabove,wehavedefinedtwoupdateoperations,named:maryand:john.Aswewillseelater,thenamesareimportantwhenhandlingsuccessandfailureresults.

SinceEcto.Multiisjustadatastructure,wecanpassitasargumenttootherfunctions,aswellasreturnit.Assumingthemultiaboveismovedintoitsownfunction,definedastransfer_money(mary,john,10),wecannowaddanoperationthatlogsthetransferasfollows:

transfer_money(mary,john,10)

|>Ecto.Multi.insert(:transfer,%Transfer{from:mary.id,to:john.id,amount:10})

Thisisconsiderablysimplerthanthenestedtransactionapproachedwehaveseenearlier!Oncealloperationsaredefinedinthemulti,wecanfinallycallRepo.transaction,thistimepassingthemulti:


57

transfer_money(mary,john,10)

|>Ecto.Multi.insert(:transfer,%Transfer{from:mary.id,to:john.id,amount:10})

|>Repo.transaction()

|>casedo

{:ok,%{mary:mary,john:john,transfer:transfer}}->

#Handlesuccesscase

{:error,name,value,rolled_back_changes}->

#Handlefailurecase

end

Ifalloperationsinthemultisucceed,itreturns{:ok,map}wherethemapcontainsthenameofalloperationsaskeysandtheirsuccessvalue.Ifanyoperationinthemultifails,thetransactionisrolledbackandRepo.transactionreturns{:error,name,value,rolled_back_changes},wherenameisthenameofthefailedoperation,valueisthefailurevalueandrolled_back_changesisamapofthepreviouslysuccessfulmultioperationsthathavebeenrolledbackduetothefailure.

Inotherwords,Ecto.Multitakescareofalltheflowcontrolboilerplatewhiledecouplingthetransactiondefinitionfromitsexecution,allowingustocomposeoperationstrivially.

TestingAnotherupsideofusingEcto.Multiisthatwecanlatertraversealloperationsstoredinthemultiandusethisinformationtowritetests.Forexample,wecouldtestthemultireturnedbytransfer_money/3asfollows:

test"transfersfrommarytojohn"do

multi=transfer_money(mary,john,10)

assert[{:mary,{:update,mary_changeset,_}},

{:john,{:update,john_changeset,_}}]=Ecto.Multi.to_list(multi)

assertmary_changeset.changes.balance==mary.balance-10

assertjohn_changeset.changes.balance==mary.balance-10

end

DependentvaluesBesidesoperationssuchasinsert,updateanddelete,Ecto.Multialsoprovidesfunctionsforhandlingmorecomplexscenarios.Forexample,prependandappendcanbeusedtomergemultistogether.Andmoregenerally,theEcto.Multi.run/3canbeusedtodefineanyoperationthatdependsontheresultsofapreviousmultioperation.


58

Let'sstudyamorepracticalexamplebyrevisitingtheproblemdefinedinthepreviouschapter.Backthen,wewantedtomodifyapostwhilepossiblygivingitalistoftagsasastringseparatedbycommas.Attheendofthechapter,webuiltasolutionthatwouldinsertanymissingtagandthenfetchallofthemusingonlytwoqueries:


useEcto.Schema

#Schemaisthesame

schema"posts"do

add:title

add:body


timestamps()

end

#Changesetisthesame


struct



end

#Parsetagshasslightlychanged



|>String.split(",")



|>insert_and_get_all()

end

defpinsert_and_get_all([])do

[]

end

defpinsert_and_get_all(names)do


Repo.insert_allMyApp.Tag,names,on_conflict::nothing

Repo.allfromtinMyApp.Tag,where:t.namein^names

end

end

Whileinsert_and_get_all/1isidempotent,allowingustorunitmultipletimesandgetthesameresultback,itdoesnotruninsideatransaction,soanyfailurewhileattemptingtomodifytheparentpoststructwouldend-upcreatingtagsthathasnopostsassociatedtothem.

Let'sfixtheproblemabovebyintroducingusingEcto.Multi.Let'sstartbysplittingthelogicintobothPostandTagmodulesandkeepingitfreefromside-effects:


59


useEcto.Schema

schema"posts"do

add:title

add:body


timestamps()

end

defchangeset(struct,tags,params)do

struct


|>Ecto.Changeset.put_assoc(:tags,tags)

end

end

defmoduleMyApp.Tagdo

useEcto.Schema

schema"tags"do

add:name

timestamps()

end

defparse(tags)do

(tags||"")

|>String.split(",")



end

end

Now,wheneverweneedtointroduceapostwithtags,wecancreateamultithatwrapsalloperationsandtherepositoryaccess:


60

definsert_or_update_post_with_tags(post,params)do

Ecto.Multi.new

|>Ecto.Multi.run(:tags,&insert_and_get_all_tags(&1,params))

|>Ecto.Multi.run(:post,&insert_or_update_post(&1,post,params)

|>Repo.transaction()

end

defpinsert_and_get_all_tags(_changes,params)do

caseMyApp.Tag.parse(params["tags"])do

[]->

[]

tags->


Repo.insert_all(MyApp.Tag,names,on_conflict::nothing)

Repo.all(fromtinMyApp.Tag,where:t.namein^names)

end

end

defpinsert_or_update_post(%{tags:tags},post,params)do

Repo.insert_or_updateMyApp.Post.changeset(post,tags,params)

end

IntheexampleabovewehaveusedEcto.Multi.run/3twice,albeitfortwodifferentreasons.

1. InEcto.Multi.run(:tags,...),weusedrun/3becauseweneedtoperformbothinsert_allandalloperations,andwhilethemultiexposesEcto.Multi.insert_all/4,itdoesnotyetexposeaEcto.Multi.all/3.WheneverweneedtoperformarepositoryoperationthatisnotsupposedbyEcto.Multi,wecanalwaysfallbacktorun/3

2. InEcto.Multi.run(:post,...),weusedrun/3becauseweneedtoaccessavalueofapreviousmultioperation.Thefirstargumentofthefunctiongiventorun/3isamapwiththeresultsoftheoperationsperformedsofar.Tograbthetagsreturnedinthepreviousstep,wesimplypatternmatchon%{tags:tags}oninsert_or_update_post

Whilerun/3isveryhandywhenitneedtogobeyondthefunctionalitiesprovidednativelybyEcto.Multi,ithasthedownsidethatoperationsdefinedwithEcto.Multi.run/3areopaque,andthereforetheycannotbetestedusingEcto.Multi.to_list/1aswedidintheprevioussection.Still,Ecto.Multiallowsustogreatlyreducecontrolflowlogicandboilerplatewhenworkingwithtransactions.


61

ConcurrenttestswiththeSQLSandboxOurlastchapterisaboutoneofthemostimportantfeaturesinEcto2.0:theconcurrentSQLsandbox.GivenElixir'scapabilityofusingallofthemachineresourcesavailable,theabilitytorunteststhattalktothedatabaseconcurrentlygivesdevelopersaloweffortopportunitytospeeduptheirtestsuiteby2x,4x,8xormoretimes,dependingonthenumberofcoresavailable.

WheneveryoustartanEctorepositoryinyoursupervisiontree,suchassupervisor(MyApp.Repo,[]),Ectostartsasupervisorwithaconnectionpool.Theconnectionpoolholdsmultipleopenconnectionstothedatabase.Wheneveryouwanttoperformadatabaseoperation,forexampleinawebrequest,Ectoautomaticallygetsaconnectionfromthepool,performstheoperationyourequested,andthenputstheconnectionbackinthepool.

Thismeansthat,whenwritingtestsusingEcto'sdefaultconnectionpool(andnottheSQLsandbox),eachtimeyourunaquery,youwilllikelygetadifferentconnectionfromthepool.Thisisnotgoodfortestssincewewantalloperationsinthesametesttousethesameconnection.

Notonlythat,wealsowantdataisolationbetweenthetests.IfIintroducearecordtothe"users"tableintestA,testBshouldnotseethoseentrieswhenqueryingthesame"users"table.

Ecto1.0solvedthefirstproblembysimplyforcingteststohaveonlyoneconnectioninthedatabasepool.Thesecondproblem,regardingdataisolation,wassolvedbynotallowingteststorunconcurrently!Whileitworked,wewereunabletoleverageconcurrency.

ExplicitcheckoutsEcto2.0solvedtheproblemsabovedifferently.Themainideaisthatwewillallowthepooltohavemultipleconnectionsbut,insteadoftheconnectionbeingcheckedoutimplicitlyeverytimewerunaquery,nowtheconnectionmustbeexplicitlycheckedoutbyeverytest.Thiswayweguaranteethateverytimeaconnectionisusedinatest,itisalwaysthesameconnection.

Onceaconnectionisexplicitlycheckedout,thetestnowownsthatparticularconnectionuntilthetestisover.


62

https://hexdocs.pm/ecto/Ecto.Adapters.SQL.Sandbox.html

Let'sstartbysettingupourdatabasetotheuseEcto.Adapters.SQL.Sandboxpool.Youcansetthoseoptionsinyourconfig/config.exs(orpreferablyconfig/test.exs):

config:my_app,Repo,

pool:Ecto.Adapters.SQL.Sandbox

Bydefaultthesandboxpoolstartsin:automaticmode,whichisexactlyhowEctoworkswithouttheSQLsandboxpool.Inotherwords,theSQLsandboxisinitiallydisabled.Thisallowsustosetupthedatabase,forexamplebyrunningmigrationsorinyourtest/test_helper.exs,asusual.

Beforeourtestsstart,weneedtoconvertthepoolto:manualmode,whereeachconnectionmustbeexplicitlycheckedout.Wedosobycallingthemode/2function,typicallyattheendofthetest/test_helper.exsfile:

#Attheendofyourtest_helper.exs

#Setthepoolmodetomanualforexplicitcheckouts

Ecto.Adapters.SQL.Sandbox.mode(MyApp.Repo,:manual)

Ifyousimplyaddthelineaboveandyoudonotchangeyourteststoexplicitlycheckaconnectionoutfromthepool,allofyourtestswillnowfail.Tosolvethis,youcouldexplicitlycheckouttheconnectiononeachtestbut,toavoidrepetition,let'sdefineaExUnit.CaseTemplatethatautomaticallydoessoinsetup:

defmoduleMyApp.RepoCasedo

useExUnit.CaseTemplate

setupdo

#Explicitlygetaconnectionbeforeeachtest

:ok=Ecto.Adapters.SQL.Sandbox.checkout(MyApp.Repo)

end

end

Nowinyourtests,insteadofuseExUnit.Case,youmaywriteuseMyApp.RepoCase,async:true.Byfollowingthestepsabove,wearenowabletohavemultipletestsrunningconcurrently,eachowningaspecificdatabasetransaction.

However,youmaywonder,howdoesEctoguaranteesthatthedatageneratedinonetestdoesnotaffecttheothertests?

Transactions


63

ThesecondmaininsightbesidesexplicitlycheckoutsintheSQLSandboxistheideaofrunningeachexplicitlycheckedoutconnectioninsideatransactions.EverytimeyourunEcto.Adapters.SQL.Sandbox.checkout(MyApp.Repo)inatest,itdoesnotonlycheckoutaconnectionbutitalsoguaranteesthatconnectionhasopenedatransactiontothedatabase.Thisway,anyinsert,updateordeleteyouperforminyourtestswillbevisibleonlytothattest.

Furthermore,attheendofeverytest,weautomaticallyrollbackthetransaction,effectivelyrevertingallofthedatabasechangesyouhaveperformedinyourtests.Thisguaranteesatestwon'taffecttestsrunningconcurrentlynorteststhatmayrunsubsequently.

Whiletheapproachofusingmultipleconnectionswithtransactionsworksinmanycases,italsoimposessomelimitationsrelatedtohowthedatabaseenginemanagestransactionsandperformsconcurrencycontrol.Forexample,whilebothPostgreSQLandMySQLsupportSQLSandbox,onlyPostgreSQLsupportsconcurrenttestswhilerunningtheSQLSandbox.Therefore,donotuseasync:truewithMySQLasyoumayrunintodeadlocks.

ThereisalsoachanceofrunningintodeadlockswhenrunningtestswithPostgreSQLwhenitcomestosharedresources,suchasdatabaseindexes.ButthosecasesarewelldocumentedintheEcto.Adapters.SQL.Sandbox,undertheFAQsection.

OwnershipWheneveratestexplicitlychecksoutaconnectionfromtheSQLSandboxpool,wesaythetestprocessownstheconnection.Alsorememberthatifatest,oranyotherprocess,doesnotexplicitlycheckaconnectionoutofthepool,thattestwillerrorwithanerrormessagesayingithasnodatabaseconnection.

Let'sseeanexample:

useMyApp.RepoCase,async:true

test"createtwoposts,onesync,anotherasync"do

task=Task.async(fn->

Repo.insert!(%Post{title:"async"})

end)

assert%Post{}=Repo.insert!(%Post{title:"sync"})

assert%Post{}=Task.await(task)

end

Thetestabovewillfailwithanerrorsimilarto:

**(RuntimeError)cannotfindownershipprocessfor#PID<0.35.0>


64

https://hexdocs.pm/ecto/Ecto.Adapters.SQL.Sandbox.html

OncewespawnaTask,thereisnoconnectionassignedtothetaskprocess,causingittofail.

Whilemosttimeswewantdifferentprocessestohavetheirowndatabaseconnection,sometimesatestmayneedtointeractwithmultipleprocesses,allusingthesameconnectionsotheyallbelongtothesametransaction.

Thesandboxmoduleprovidestwowaysofdoingso,viaallowancesorbyrunninginsharedmode.

Allowances

Ifaprocessexplicitlyownsaconnection,thatprocessmayalsoallowotherprocessestousethatconnection.Effectivelyallowingmultipleprocessestocollaborateoverthesameconnectionatthesametime.Let'sgiveitatry:


parent=self()


Ecto.Adapters.SQL.Sandbox.allow(Repo,parent,self())


end)



end

Andthat'sit!Bycallingallow/3,weareexplicitlyassigningtheparent'sconnection(i.e.thetestprocess'connection)tothetask.

Becauseallowancesuseanexplicitmechanism,theiradvantageisthatyoucanstillrunyourtestsinasyncmode.Thedownsideisthatyouneedtoexplicitlycontrolandalloweverysingleprocess,whichisnotalwayspossible.Insuchcases,youmayresorttosharedmode.

Sharedmode

Sharedmodeallowsaprocesstoshareitsconnectionwithanyotherprocessautomatically,withoutrelyingonexplicitallowances.

Let'schangetheexampleabovetousesharedmode:


65


#Settingthesharedmodemustbedoneonlyaftercheckout

Ecto.Adapters.SQL.Sandbox.mode(Repo,{:shared,self()})



end)



end

Bycallingmode({:shared,self()}),anyprocessthatneedstotalktothedatabasewillnowusethesameconnectionastheonecheckedoutbythetestprocess.

Theadvantageofsharedmodeisthatbycallingasinglefunction,youwillensureallupcomingprocessesandoperationswillusethatsharedconnection,withoutaneedtoexplicitlyallowthem.Thedownsideisthattestscannolongerrunconcurrentlyinsharedmode.

SummingupInthischapterwehavelearnedaboutthepowerfulconcurrentSQLsandboxandhowitleveragestransactionsandanownershipmechanismwithexplicitcheckoutsthatallowsteststorunconcurrentlyevenwhentheyneedtocommunicatetothedatabase.

Wehavealsodiscussedtwomechanismforsharingownerships:

Usingallowances-requiresexplicitallowancesviaallow/3.Testsmayrunconcurrently.

Usingsharedmode-doesnotrequireexplicitallowances.Testscannotrunconcurrently.

WhilethroughoutthebookwecoveredhowEctoisacollectionoftoolsforworkingonyourdomain,thelastchaptersalsoshowedEctoprovidestoolstobetterinteracttothedatabase,suchasEcto.MultiwhichleveragesthefunctionalpropertiesbehindElixir,aswellastheSQLSandboxwhichexploitstheconcurrencypowerbehindtheErlangVM.

Wehopeyouhavelearnedalotthroughoutthisjourneyandthatyouarereadytowriteclean,performantandmaintainableapplications.


66


67

CONTACTUS

http://plataformatec.com.br/contact?utm_source=ebook-ecto-2-beta-3&utm_medium=referral&utm_campaign=ebook-ecto-2&utm_content=contracapa


68

what's new in ecto 2blog.plataformatec.com.br/wp-content/uploads/2016/11/... · 2016. 12....

Documents