1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
1.10
1.11
1.12
1.13
TableofContentsForeword
Introduction
1.EctoisnotyourORM
2.Schemalessqueries
3.Schemalesschangesets
4.Dynamicqueries
5.Multitenancywithqueryprefixes
6.Aggregatesandsubqueries
7.Improvedassociationsandfactories
8.Manytomanyandcasting
9.Manytomanyandupserts
10.ComposabletransactionswithEcto.Multi
11.ConcurrenttestswiththeSQLSandbox
2
ForewordInJanuary2017,wewillcelebrate5yearssincewedecidedtoinvestinElixir.Backin2012,JoséValim,ourco-founderandpartner,presentedustheideaofaprogramminglanguagethatwouldbeexpressive,embraceproductivityinitstooling,andleveragetheErlangVMtonotonlytackletheproblemsinwritingconcurrentsoftwarebutalsotobuildfault-tolerantanddistributedsystems.
Elixircontinued,insomesense,tobeariskyprojectformonths.Wewerecertainlyexcitedaboutspreadingfunctional,concurrentanddistributedprogrammingconceptstomoreandmoredevelopers,hopingitwouldleadtoapositiveimpactonthesoftwaredevelopmentindustry,butdevelopingalanguageisalong-termeffortthatmayneverbecomeconcrete.
Duringthesummerof2013,othercompaniesanddevelopersstartedtoshowinterestonElixir.Weheardaboutcompaniesusingitinproduction,moredevelopersbegantocontributeandcreatetheirownprojects,differentpublisherswerewritingbooksonthelanguage,andsoon.SucheventsgaveustheconfidencetoinvestmoreinElixirandbringthelanguagetoversion1.0.
OnceElixir1.0waslaunchedinSeptember2014,weturnedourfocustothewebplatform.WetidiedupPlug,thebuildingblockforwritingwebapplicationsinElixir.WealsofocusedintensivelyonEcto,bringingittoversion1.0togetherwiththeEctoteam,andthenworkedalongsideChrisMcCordandteamtogetthefirstmajorPhoenixreleaseout.Duringthistimewealsostartedothercommunitycentricinitiatives,suchasElixirRadar,andbeganourfirstcommercialElixirprojects.
Today,boththecommunityandouropensourceprojectsareshowingsteadyandhealthygrowth.Elixirisastablelanguagewithcontinuousimprovementslandedinminorversions.PlugcontinuestobeasolidfoundationforframeworkssuchasPhoenix.Ecto,however,requiredmorethanasmallnudgeintherightdirection.Werealizedthatweneededtoletgoofold,harmfulhabitsandmakeEctolessofanabstractionlayerandmoreofatoolyoucontrolandapplytodifferentproblems.
ThisbookisthefinaleffortbehindEcto2.0.ItshowcasesthenewdirectionwehaveplannedforEcto,thestructuralimprovementsmadebytheEctoteamandmanyofitsnewfeatures.Wehopeyouwillenjoyit.Afterall,itistimetoletgoofpasthabits.
Havefun,
-ThePlataformatecteam
Foreword
3
Foreword
4
CONTACTUS
Foreword
5
IntroductionAuthornote:WelcometotheearlycutofourEcto2.0book.Thisisourlastbetaeditionwithallbookchaptersreadyforreview.Ifyouhavesuggestionsorstumbleduponerrors,[email protected].
Ecto2.0isasubstantialdeparturefromearlierversions.Insteadofthinkingaboutmodels,Ecto2.0aimstoprovidedevelopersawiderangeofdata-centrictools.Therefore,inordertouseEcto2.0effectively,wemustlearnhowtowieldthosetoolsproperly.That'sthegoalofthisbook.
Thisbook,however,isnotanintroductiontoEcto.IfyouhaveneverusedEctobefore,werecommendyoutogetstartedwithEcto'sdocumentationandlearnmoreaboutrepositories,queries,schemasandchangesets.Weassumethereaderisfamiliarwiththesebuildingblocksandhowtheyrelatetoeachother.
ThefirstchaptersofthebookwillcoverthebiggestconceptualchangesinEcto2.0.Wewilltalkaboutrelationalmappersin"EctoisnotyourORM"andthenexploreSchemalessQueriesandtherelationshipbetweenSchemasandChangesets.
Afterwewilltakeadeeperlookintoqueries,discussinghowEcto2.0makesiteasiertobuilddynamicqueries,howtotargetdifferentdatabasesviaqueryprefixes,aswellasthenewaggregateandsubqueryfeatures.
Thenwewillgobacktoschemasanddiscusstheschema-relatedenhancementsthatarepartofEcto2.0,suchastheimprovedassociationsupport,many_to_manyassociationsandEcto's2.1upsertsupport.
Finally,wewillexplorebrandnewtopics,likethenewEctoSQLSandbox,thatallowsdeveloperstoruntestsagainstthedatabaseconcurrently,aswellasEcto.Multi,whichmakesworkingwithtransactionssimplerthanever.
Introduction
6
EctoisnotyourORMDependingonyourperspective,thisisaratherboldorobviousstatementtostartthisbook.Afterall,Elixirisnotanobject-orientedlanguage,soEctocan'tbeanObject-relationalMapper.However,thisstatementisslightlymorenuancedthanitlooksandthereareimportantlessonstobelearnedhere.
OisforObjectsAtitscore,objectscouplestateandbehaviourtogether.Inthesameuserobject,youcanhavedata,liketheuser.name,aswellasbehaviour,likeconfirmingaparticularuseraccountviauser.confirm().Whilesomelanguagesenforcedifferentsyntaxesbetweenaccessingdata(user.namewithoutparentheses)andbehaviour(user.confirm()withparentheses),otherlanguagesfollowtheUniformAccessPrincipleinwhichanobjectshouldnotmakeadistinctionbetweenthetwosyntaxes.EiffelandRubyarelanguagesthatfollowsuchprinciple.
Elixirfailsthe"couplingofstateandbehaviour"test.InElixir,weworkwithdifferentdatastructuressuchastuples,lists,mapsandothers.Behaviourcannotbeattachedtodatastructures.Behaviourisalwaysaddedtomodulesviafunctions.
Whenthereisaneedtoworkwithstructureddata,Elixirprovidesstructs.Structsdefineasetoffields.Astructwillbereferencedbythenameofthemodulewhereitisdefined:
defmoduleUserdo
defstruct[:name,:email]
end
user=%User{name:"JohnDoe",email:"[email protected]"}
Onceauserstructiscreated,wecanaccessitsemailviauser.email.However,structsareonlydata.Itisimpossibletoinvokeuser.confirm()onaparticularstructinawayitwillexecutecoderelatedtoe-mailconfirmation.
Althoughwecannotattachbehaviourtostructs,itispossibletoaddfunctionstothesamemodulethatdefinesthestruct:
1.EctoisnotyourORM
7
defmoduleUserdo
defstruct[:name,:email]
defconfirm(user)do
#Confirmtheuseremail
end
end
Evenwiththedefinitionabove,itisimpossibleinElixirtoconfirmagivenuserbycallinguser.confirm().Instead,theUserprefixisrequiredandtheuserstructmustbeexplicitlygivenasargument,asinUser.confirm(user).Attheendoftheday,thereisnostructuralcouplingbetweentheuserstructandthefunctionsintheUsermodule.HenceElixirdoesnothavemethods,ithasfunctions.
Withouthavingobjects,Ectocertainlycan'tbeanORM.However,ifweletgooftheletter"O"forasecond,canEctostillbearelationalmapper?
RelationalmappersAnObject-RelationalMapperisatechniqueforconvertingdatabetweenincompatibletypesystems,commonlydatabases,toobjects,andback.
Similarly,EctoprovidesschemasthatmapsanydatasourceintoanElixirstruct.Whenappliedtoyourdatabase,Ectoschemasarerelationalmappers.Therefore,whileEctoisnotarelationalmapper,itcontainsarelationalmapperaspartofthemanydifferenttoolsitoffers.
Forexample,theschemabelowtiesthefieldsname,email,inserted_atandupdated_attofieldssimilarlynamedintheuserstable:
defmoduleMyApp.Userdo
useEcto.Schema
schema"users"do
field:name
field:email
timestamps()
end
end
Theappealbehindschemasisthatyoudefinetheshapeofthedataonceandyoucanusethisshapetoretrievedatafromthedatabaseaswellascoordinatechangeshappeningonthedata:
1.EctoisnotyourORM
8
MyApp.User
|>MyApp.Repo.get!(13)
|>Ecto.Changeset.cast([name:"newname"],[:name,:email])
|>MyApp.Repo.update!
Byrelyingontheschemainformation,Ectoknowshowtoreadandwritedatawithoutextrainputfromthedeveloper.Insmallapplications,thiscouplingbetweenthedataanditsrepresentationisdesired.However,whenusedwrongly,itleadstocomplexcodebasesandsubparsolutions.
ItisimportanttounderstandtherelationshipbetweenEctoandrelationalmappersbecausesaying"EctoisnotyourORM"doesnotautomaticallysaveEctoschemasfromsomeofthedownsidesmanydevelopersassociateORMswith.
HerearesomeexamplesofissuesoftenassociatedwithORMsthatEctodevelopersmayrunintowhenusingschemas:
ProjectsusingEctomayend-upwith"GodSchemas",commonlyreferredas"GodModels","FatModels"or"CanonicalModels"insomelanguagesandframeworks.Suchschemascouldcontainhundredsoffields,oftenreflectingbaddecisionsdoneatthedatalayer.Insteadofprovidingonesingleschemawithfieldsthatspanmultipleconcerns,itisbettertobreaktheschemaacrossmultiplecontexts.Forexample,insteadofasingleMyApp.Userschemawithdozensoffields,considerbreakingitintoMyApp.Accounts.User,MyApp.Purchases.Userandsoon.Eachstructwithfieldsexclusivetoitsenclosingcontext
Developersmayexcessivelyrelyonschemaswhensometimesthebestwaytoretrievedatafromthedatabaseisintoregulardatastructures(likemapsandtuples)andnotpre-definedshapesofdatalikestructs.Forexample,whendoingsearches,generatingreportsandothers,thereisnoreasontorelyorreturnschemasfromsuchqueries,asitoftenreliesondatacomingfrommultipletableswithdifferentrequirements
Developersmaytrytousethesameschemaforoperationsthatmaybequitedifferentstructurally.Manyapplicationswouldboltfeaturessuchasregistration,accountlogin,intoasingleUserschema,whilehandlingeachoperationindividually,possiblyusingdifferentschemas,wouldleadtosimplerandclearersolutions
Inthenexttwochapters,wewanttobreakthose"badpractices"apartbyexploringhowtouseEctowithnoormultipleschemaspercontext.Bylearninghowtoinsert,delete,manipulateandvalidatedatawithandwithoutschemas,wehopedeveloperswillfeelcomfortablewithbuildingcomplexapplicationswithoutrelyingonone-size-fits-allschemas.
1.EctoisnotyourORM
9
1.EctoisnotyourORM
10
SchemalessqueriesMostqueriesinEctoarewrittenusingschemas.Forexample,toretrieveallpostsinadatabase,onemaywrite:
MyApp.Repo.all(Post)
Intheconstructabove,Ectoknowsallfieldsandtheirtypesintheschema,rewritingthequeryaboveto:
MyApp.Repo.all(frompinPost,select:%Post{title:p.title,body:p.body,...})
Interestingly,backinEcto'searlydays,therewasnosuchthingasschemas.Queriescouldonlybewrittendirectlyagainstadatabasetablebypassingthetablenameasastring:
MyApp.Repo.all(frompin"posts",select:{p.title,p.body})
Whenwritingschemalessqueries,theselectexpressionmustbeexplicitlywrittenwithallthedesiredfields.
WhiletheabovesyntaxmadeitintoEcto1.0,bythetimeEcto1.0waslaunched,mostofthedevelopmentfocusinEctohadchangedtowardsschemas.Thismeanswhiledeveloperswereabletoreaddatawithoutschemas,theywereoftentooverbose.Notonlythat,ifyouwantedtoinsertentriestoyourdatabasewithoutschemas,youwereoutofluck.
Ecto2.0levelsupthegamebyaddingmanyimprovementstoschemalessqueries,notonlybyimprovingthesyntaxforreadingandupdatingdata,butalsobyallowingalldatabaseoperationstobeexpressedwithoutaschema.
insert_allOneofthefunctionsaddedtoEcto2.0isEcto.Repo.insert_all/3.Withinsert_all,developerscaninsertmultipleentriesatonceintoarepository:
MyApp.Repo.insert_all(Post,[[title:"hello",body:"world"],
[title:"another",body:"post"]])
2.Schemalessqueries
11
Althoughinsert_allisjustaregularElixirfunction,itplaysanimportantroleinEcto2.0asitallowsdeveloperstoread,create,updateanddeleteentrieswithoutaschema.insert_allwasthelastpieceofthepuzzle.Let'sseesomeexamples.
Ifyouarewritingareportingview,itmaybecounter-productivetothinkhowyourexistingapplicationschemasrelatetothereportbeinggenerated.Itisoftensimplertowriteaquerythatreturnsonlythedatayouneed,withouttryingtofitthedataintoexistingschemas:
importEcto.Query
defrunning_activities(start_at,end_at)
MyApp.Repo.all(
fromuin"users",
join:ain"activities",
on:a.user_id==u.id,
where:a.start_at>type(^start_at,Ecto.DateTime)and
a.end_at<type(^end_at,Ecto.DateTime),
group_by:a.user_id,
select:%{user_id:a.user_id,interval:a.start_at-a.end_at,count:count(u.id
)}
)
end
Thefunctionabovedoesnotrelyonschemas.Itreturnsonlythedatathatmattersforbuildingthereport.Noticehowweusethetype/2functiontospecifywhatistheexpectedtypeoftheargumentweareinterpolating,benefitingfromthesametypecastingguaranteesaschemawouldgive.
Inserts,updatesanddeletescanalsobedonewithoutschemasviainsert_all,update_allanddelete_allrespectively:
#InsertdataintopostsandreturnitsID
[%{id:id}]=
MyApp.Repo.insert_all"posts",[[title:"hello"]],returning:[:id]
#UsetheIDtotriggerupdates
post=frompin"posts",where:[id:^id]
{1,_}=MyApp.Repo.update_allpost,set:[title:"newtitle"]
#Aswellasfordeletes
{1,_}=MyApp.Repo.delete_allpost
ItisnothardtoseehowtheseoperationsdirectlymaptotheirSQLvariants,keepingthedatabaseatyourfingertipswithouttheneedtointermediatealloperationsthroughschemas.
2.Schemalessqueries
12
SimplerqueriesBesidessupportingschemalessinserts,updatesanddeletesqueries,withvaryingdegreesofcomplexity,Ecto2.0alsomakesregularschemalessqueriesmoreexpressive.
Oneexampleistheabilitytoselectalldesiredfieldswithoutduplication.Inearlyversions,youwouldhavetowriteverboseselectexpressionssuchas:
frompin"posts",select:%{title:p.title,body:p.body}
WithEcto2.0youcansimplypassthedesiredlistoffieldsdirectly:
from"posts",select:[:title,:body]
Thetwoqueriesaboveareequivalent.Whenalistoffieldsisgiven,Ectowillautomaticallyconvertthelistoffieldstoamaporastruct.
SupportforpassingalistoffieldsorkeywordlistshasbeenaddedtoalmostallqueryconstructsinEcto2.0.Forexample,wecanuseanupdatequerytochangethetitleofagivenpostwithoutaschema:
defupdate_title(post,new_title)do
query=from"posts",where:[id:^post.id],update:[set:[title:^new_title]]
MyApp.Repo.update_all(query)
end
Theupdateconstructsupportsfourcommands:
:set-setsthegivencolumntothegivenvalues:inc-incrementsthegivencolumnbythegivenvalue:push-pushes(appends)thegivenvaluetotheendofanarraycolumn:pull-pulls(removes)thegivenvaluefromanarraycolumn
Forexample,wecanincrementacolumnatomicallybyusingthe:inccommand,withorwithoutschemas:
defincrement_page_views(post)do
query=from"posts",where:[id:^post.id],update:[inc:[page_views:1]]
MyApp.Repo.update_all(query)
end
2.Schemalessqueries
13
Byallowingregulardatastructurestobegiventomostqueryoperations,Ecto2.0makesquerieswithandwithoutschemasmoreaccessible.Notonlythat,italsoenablesdeveloperstowritedynamicqueries,wherefields,filters,orderingcannotbespecifiedupfront.Wewillexploresuchwithmoredetailsinupcomingchapters.Fornow,let'scontinueexploringschemasinthecontextofchangesets.
2.Schemalessqueries
14
SchemasandchangesetsInthelastchapterwelearnedhowtoperformalldatabaseoperations,frominsertiontodeletion,withoutusingaschema.Whilewehavebeenexploringtheabilitytowriteconstructswithoutschemas,wehaven'tdiscussedwhatschemasactuallyare.Inthischapter,wewillrectifythat.
Inthischapterwewilltakealookattheroleschemasplaywhenvalidatingandcastingdatathroughchangesets.Aswewillsee,sometimesthebestsolutionisnottocompletelyavoidschemas,butbreakalargeschemaintosmallerones.Maybeoneforreadingdata,anotherforwriting.Maybeoneforyourdatabase,anotherforyourforms.
SchemasaremappersTheEctodocumentationsays:
AnEctoschemaisusedtomapanydatasourceintoanElixirstruct.
WeputemphasisonanybecauseitisacommonmisconceptiontothinkEctoschemasmaponlytoyourdatabasetables.
Forinstance,whenyouwriteawebapplicationusingPhoenixandyouuseEctotoreceiveexternalchangesandapplysuchchangestoyourdatabase,wehavethismapping:
Database<->Ectoschema<->Forms/API
AlthoughthereisasingleEctoschemamappingtobothyourdatabaseandyourAPI,inmanysituationsitisbettertobreakthismappingintwo.Let'sseesomepracticalexamples.
Imagineyouareworkingwithaclientthatwantsthe"SignUp"formtocontainthefields"Firstname","Lastname"alongside"E-mail"andotherinformation.Youknowthereareacoupleproblemswiththisapproach.
Firstofall,noteveryonehasafirstandlastname.Althoughyourclientisdecidedonpresentingbothfields,theyareaUIconcern,andyoudon'twanttheUItodictatetheshapeofyourdata.Furthermore,youknowitwouldbeusefultobreakthe"SignUp"informationacrosstwotables,the"accounts"and"profiles"tables.
Giventherequirementsabove,howwouldweimplementtheSignUpfeatureinthebackend?
3.Schemalesschangesets
15
Oneapproachwouldbetohavetwoschemas,AccountandProfile,withvirtualfieldssuchasfirst_nameandlast_name,anduseassociationsalongsidenestedformstotietheschemastoyourUI.Oneofsuchschemaswouldbe:
defmoduleProfiledo
useEcto.Schema
schema"profiles"do
field:name
field:first_name,:string,virtual:true
field:last_name,:string,virtual:true
...
end
end
ItisnothardtoseehowwearepollutingourProfileschemawithUIrequirementsbyaddingfieldssuchfirst_nameandlast_name.IftheProfileschemaisusedforbothreadingandwritingdata,itmayend-upinanawkwardplacewhereitisnotusefulforany,asitcontainsfieldsthatmapjusttooneortheotheroperation.
Onealternativesolutionistobreakthe"Database<->Ectoschema<->Forms/API"mappingintwoparts.Thefirstwillcastandvalidatetheexternaldatawithitsownstructurewhichyouthentransformandwritetothedatabase.Forsuch,let'sdefineaschemanamedRegistrationthatwilltakecareofcastingandvalidatingtheformdataexclusively,mappingdirectlytotheUIfields:
defmoduleRegistrationdo
useEcto.Schema
embedded_schemado
field:first_name
field:last_name
field:email
end
end
Weusedembedded_schemabecauseitisnotourintenttopersistitanywhere.Withtheschemainhand,wecanuseEctochangesetsandvalidationstoprocessthedata:
3.Schemalesschangesets
16
fields=[:first_name,:last_name,:email]
changeset=
%Registration{}
|>Ecto.Changeset.cast(params["sign_up"],fields)
|>validate_required(...)
|>validate_length(...)
Nowthattheregistrationchangesaremappedandvalidated,wecancheckiftheresultingchangesetisvalidandactaccordingly:
ifchangeset.valid?do
#Getthemodifiedregistrationstructoutofthechangeset
registration=Ecto.Changeset.apply_changes(changeset)
MyApp.Repo.transactionfn->
MyApp.Repo.insert_all"accounts",[Registration.to_account(registration)]
MyApp.Repo.insert_all"profiles",[Registration.to_profile(registration)]
end
{:ok,registration}
else
#AnnotatetheactionwetriedtoperformsotheUIshowserrors
changeset=%{changeset|action::registration}
{:error,changeset}
end
Theto_account/1andto_profile/1functionsinRegistrationwouldreceivetheregistrationstructandsplittheattributesapartaccordingly:
defto_account(registration)do
Map.take(registration,[:email])
end
defto_profile(%{first_name:first,last_name:last})do
%{name:"#{first}#{last}"}
end
Intheexampleabove,bybreakingapartthemappingbetweenthedatabaseandElixirandbetweenElixirandtheUI,ourcodebecomesclearerandourdatastructuressimpler.
NotewehaveusedMyApp.Repo.insert_all/2toadddatatoboth"accounts"and"profiles"tablesdirectly.Wehavechosentobypassschemasaltogether.However,thereisnothingstoppingyoufromalsodefiningbothAccountandProfileschemasandchangingto_account/1andto_profile/1torespectivelyreturn%Account{}and%Profile{}
3.Schemalesschangesets
17
structs.Oncestructsarereturned,theycouldbeinsertedthroughtheusualMyApp.Repo.insert/2operation.Doingsocanbeespeciallyusefulifthereareuniquenessorotherconstraintsthatyouwanttocheckduringinsertion.
SchemalesschangesetsAlthoughwechosetodefineaRegistrationschematouseinthechangeset,Ecto2.0alsoallowsdeveloperstousechangesetswithoutschemas.Wecandynamicallydefinethedataandtheirtypes.Let'srewritetheregistrationchangesetabovetobypassschemas:
data=%{}
types=%{first_name::string,last_name::string,email::string}
changeset=
{data,types}#Thedata+typestupleisequivalentto%Registration{}
|>Ecto.Changeset.cast(params["sign_up"],Map.keys(types))
|>validate_required(...)
|>validate_length(...)
YoucanusethistechniquetovalidateAPIendpoints,searchforms,andothersourcesofdata.Thechoiceofusingschemasdependsmostlyifyouwanttousethesamemappingindifferentplacesorifyoudesirethecompile-timeguaranteesElixirstructsgivesyou.Otherwise,youcanbypassschemasaltogether,beitwhenusingchangesetsorinteractingwiththerepository.
However,themostimportantlessoninthischapterisnotwhentouseornottouseschemas,butratherunderstandwhenabigproblemcanbebrokenintosmallerproblemsthatcanbesolvedindependentlyleadingtoanoverallcleanersolution.Thechoiceofusingschemasornotabovedidn'taffectthesolutionasmuchasthechoiceofbreakingtheregistrationproblemapart.
3.Schemalesschangesets
18
DynamicqueriesEctowasdesignedfromthegrounduptohaveanexpressivequeryAPIthatleveragesElixirsyntaxtoprovidequeriesthatarepre-compiledtobeperformantandsafe.Whenbuildingqueries,wemayusethekeywordssyntax
importEcto.Query
frompinPost,
where:p.author=="José"andp.category=="Elixir",
where:p.published_at>^minimum_date,
order_by:[desc:p.published_at]
orthepipe-basedone
importEcto.Query
Post
|>where([p],p.author=="José"andp.category=="Elixir")
|>where([p],p.published_at>^minimum_date)
|>order_by([p],desc:p.published_at)
Whilemanydeveloperspreferthepipe-basedsyntax,havingtorepeatthebindingpmadeitquiteverbosecomparedtothekeywordone.Furthermore,thecompile-timeaspectofEctoquerieswasatoddswithbuildingqueriesdynamically.Imagineforexampleawebapplicationthatprovidessearchfunctionalityontopofexistingposts.Theusershouldbeabletospecifymultiplecriteria,suchastheauthorname,thepostcategory,publishinginterval,etc.
InEcto1.0,theonlywaytowritesuchfunctionalitywouldbeviaEnum.reduce/3:
4.Dynamicqueries
19
deffilter(params)do
Enum.reduce(params,Post,&filter/2)
end
defpfilter({"author",author},query)do
where(query,[p],p.author==^author)
end
defpfilter({"category",category},query)do
where(query,[p],p.category==^category)
end
defpfilter({"published_at",minimum_date},query)do
where(query,[p],p.published_at>^minimum_date)
end
defpfilter({"order_by","published_at_asc"},query)do
order_by(query,[p],asc:p.published_at)
end
defpfilter({"order_by","published_at_desc"},query)do
order_by(query,[p],desc:p.published_at)
end
defpfilter(_ignore_unknown,query)do
query
end
Whilethecodeaboveworksfine,itcouplestheprocessingoftheparameterswiththequerygeneration.Itisaverboseimplementationthatisalsohardtotestsincetheresultoffilteringandhandlingofparametersarestoreddirectlyinsidethequerystruct.
Apreferredapproachwouldbetoprocesstheparametersintoregulardatastructuresandthenbuildthequeryaslateaspossible.That'sexactlywhatEcto2.0andlaterallowustodo.
FocusingondatastructuresEcto2.0providesasimplerAPIforbothkeywordandpipebasedqueriesbymakingdatastructuresfirst-class.Let'srewritetheoriginalqueriestousedatastructureswhenpossible:
frompinPost,
where:[author:"José",category:"Elixir"],
where:p.published_at>^minimum_date,
order_by:[desc::published_at]
and
4.Dynamicqueries
20
Post
|>where(author:"José",category:"Elixir")
|>where([p],p.published_at>^minimum_date)
|>order_by(desc::published_at)
Noticehowwewereabletoditchthepselectorinmostexpressions.InEcto2.0,allconstructs,fromselectandorder_bytowhereandgroup_by,acceptdatastructuresasinput.Thedatastructurecanbespecifiedatcompile-time,asabove,andalsodynamicallyatruntime,shownbelow:
where=[author:"José",category:"Elixir"]
order_by=[desc::published_at]
Post
|>where(^where)
|>where([p],p.published_at>^minimum_date)
|>order_by(^order_by)
Theadvantageofinterpolatingdatastructuresasaboveisthatwecandecoupletheprocessingofparametersfromthequerygeneration.Howevernotallexpressionscanbeconvertedtothedatastructureshape.Sincewhereconvertsakey-valuetoakey==valuecomparison,order-basedcomparisonssuchasp.published_at>^minimum_datestillneedtobewrittenaspartofthequeryasbefore.
Luckily,theupcomingEcto2.1releasesolvesthisissue.
ThedynamicmacroForcaseswherewecannotrelyondatastructuresbutstilldesiretobuildqueriesdynamically,Ecto2.1includestheEcto.Query.dynamic/2macro.
Inordertounderstandhowthedynamicmacroworkslet'srewritethefilter/1functionfromthebeginningofthischapterusingbothdatastructuresandthedynamicmacro.NotetheexamplebelowrequiresEcto2.1:
4.Dynamicqueries
21
deffilter(params)do
Post
|>order_by(^filter_order_by(params["order_by"]))
|>where(^filter_where(params))
|>where(^filter_published_at(params["published_at"]))
end
deffilter_order_by("published_at_desc"),do:[desc::published_at]
deffilter_order_by("published_at"),do:[asc::published_at]
deffilter_order_by(_),do:[]
deffilter_where(params)do
forkey<-[:author,:category],
value=params[Atom.to_string(params)],
do:{key,value}
end
deffilter_published_at(date)whenis_binary(date),
do:dynamic([p],p.published_at>^date)
deffilter_published_at(_date),
do:true
Thedynamicmacroallowsustobuilddynamicexpressionsthatarelaterinterpolatedintothequery.dynamicexpressionscanalsobeinterpolatedintodynamicexpressions,allowingdeveloperstobuildcomplexexpressionsdynamicallywithouthassle.
Becausewewereabletobreakourproblemintosmallerfunctionsthatreceiveregulardatastructures,wecanuseallthetoolsavailableinElixirtoworkwithdata.Forhandlingtheorder_byparameter,itmaybebesttosimplypatternmatchontheorder_byparameter.Forbuildingthewhereclause,wecantraversethelistofknownkeysandconvertthemtotheformatexpectedbyEcto.Forcomplexconditions,weusethedynamicmacro.
Testingalsobecomessimpleraswecantesteachfunctioninisolation,evenwhenusingdynamicqueries:
test"filterpublishedatbasedonthegivendate"do
assertinspect(filter_published_at("2010-04-17"))==
"dynamic([p],p.published_at>^\"2010-04-17\")"
assertinspect(filter_published_at(nil))==
"true"
end
WhileattheendofthedaysomedevelopersmayfeelmorecomfortablewithusingtheEnum.reduce/3approachthewholeway,Ecto2.0andlatergivesustheoptiontochoosewhichapproachworksbest.
ThankstoMichałMuskałaforsuggestionsandfeedbackonthischapter.
4.Dynamicqueries
22
4.Dynamicqueries
23
MultitenancywithqueryprefixesEcto2.0introducestheabilitytorunqueriesindifferentprefixesusingasinglepoolofdatabaseconnections.FordatabasesenginessuchasPostgres,Ecto'sprefixmapstoPostgres'DDLschemas.ForMySQL,eachprefixisadifferentdatabaseonitsown.
Queryprefixesmaybeusefulindifferentscenarios.Forexample,multitenantappsrunningonPostgreswoulddefinemultipleprefixes,usuallyoneperclient,underasingledatabase.Theideaisthatprefixeswillprovidedataisolationbetweenthedifferentusersoftheapplication,guaranteeingeithergloballyoratthedatalevelthatqueriesandcommandsactonaspecificprefix.
Prefixesmayalsobeusefulonhigh-trafficapplicationswheredataispartitionedupfront.Forexample,agamingplatformmaybreakgamedataintoisolatedpartitions,eachgivenbyadifferentprefix.Apartitionforagivenplayeriseitherchosenatrandomforeachgameplayorcalculatedbasedontheplayerinformation.
Whilequeryprefixesweredesignedwiththetwoscenariosaboveinmind,theymayalsobeusedinothercircumstances,whichwewillexplorethroughoutthischapter.AlltheexamplesbelowassumeyouareusingPostgres.Otherdatabasesenginesmayrequireslightlydifferentsolutions.
GlobalprefixesAsastartingpoint,let'sstartwithasimplescenario:yourapplicationmustconnecttoaparticularprefixwhenrunninginproduction.Thismaybeduetoinfrastructureconditions,databaseadministrationrulesorothers.
Let'sdefinearepositoryandaschematogetstarted:
5.Multitenancywithqueryprefixes
24
#lib/repo.ex
defmoduleMyApp.Repodo
useEcto.Repo,otp_app::my_app
end
#lib/sample.ex
defmoduleMyApp.Sampledo
useEcto.Schema
schema"samples"do
field:name
timestamps
end
end
Nowlet'sconfiguretherepository:
#config/config.exs
config:my_app,MyApp.Repo,
adapter:Ecto.Adapters.Postgres,
username:"postgres",
password:"postgres",
database:"demo",
hostname:"localhost",
pool_size:10
Anddefineamigration:
#priv/repo/migrations/20160101000000_create_sample.exs
defmoduleMyApp.Repo.Migrations.CreateSampledo
useEcto.Migration
defchangedo
createtable(:samples)do
add:name,:string
timestamps()
end
end
end
Nowlet'screatethedatabase,migrateitandthenstartanIExsession:
5.Multitenancywithqueryprefixes
25
$mixecto.create
$mixecto.migrate
$iex-Smix
InteractiveElixir(1.4.0-dev)-pressCtrl+Ctoexit(typeh()ENTERforhelp)
iex(1)>MyApp.Repo.allMyApp.Sample
[]
Wehaven'tdoneanythingunusualsofar.Wecreatedourdatabaseinstance,madeituptodatebyrunningmigrationsandthensuccessfullymadeaqueryagainstthe"samples"table,whichreturnedanemptylist.
Bydefault,connectionstoPostgres'databasesrunonthe"public"prefix.Whenwerunmigrationsandqueries,theyareallrunningagainstthe"public"prefix.Howeverimagineyourapplicationhasarequirementtorunaparticularprefixinproduction,let'scallit"global_prefix".
LuckilyPostgresallowsustochangetheprefixourdatabaseconnectionsrunonbysettingtheschemasearchpath.Thebestmomenttochangetheschemasearchpathisrightafterwesetupthedatabaseconnection,ensuringallofourquerieswillrunonthatparticularprefix,throughouttheconnectionlife-cycle.
Todoso,let'schangeourdatabaseconfigurationin"config/config.exs"andspecifyan:after_connectoption.:after_connectexpectsatuplewithmodule,functionandargumentsitwillinvokewiththeconnectionprocessassoonasadatabaseconnectionisestablished:
config:my_app,MyApp.Repo,
adapter:Ecto.Adapters.Postgres,
username:"postgres",
password:"postgres",
database:"demo_dev",
hostname:"localhost",
pool_size:10,
after_connect:{Postgrex,:query!,["SETsearch_pathTOglobal_prefix",[]]}
Nowlet'strytorunthesamequeryasbefore:
$iex-Smix
InteractiveElixir(1.4.0-dev)-pressCtrl+Ctoexit(typeh()ENTERforhelp)
iex(1)>MyApp.Repo.allMyApp.Sample
**(Postgrex.Error)ERROR(undefined_table):relation"samples"doesnotexist
Ourpreviouslysuccessfulquerynowfailsbecausethereisnotable"samples"underthenewprefix.Let'strytofixthatbyrunningmigrations:
5.Multitenancywithqueryprefixes
26
$mixecto.migrate
**(Postgrex.Error)ERROR(invalid_schema_name):noschemahasbeenselectedtocreate
in
Oops.Nowmigrationsaysthereisnosuchschemaname.That'sbecausePostgresautomaticallycreatesthe"public"prefixeverytimewecreateanewdatabase.Ifwewanttouseadifferentprefix,wemustexplicitlycreateitonthedatabasewearerunningon:
$psql-ddemo_dev-c"CREATESCHEMAglobal_prefix"
Nowwearereadytomigrateandrunourqueries:
$mixecto.migrate
$iex-Smix
InteractiveElixir(1.4.0-dev)-pressCtrl+Ctoexit(typeh()ENTERforhelp)
iex(1)>MyApp.Repo.allMyApp.Sample
[]
Dataindifferentprefixesareisolated.Writingtothe"samples"tableinoneprefixcannotbeaccessedbytheotherunlesswechangetheprefixintheconnectionorusingtheEctoconvenienceswewilldiscussbelow.
Per-queryandper-structprefixesWhilestillconfiguredtoconnecttothe"global_prefix"on:after_connect,let'srunsomequeries:
iex(1)MyApp.Repo.allMyApp.Sample
[]
iex(2)MyApp.Repo.insert%MyApp.Sample{name:"mary"}
{:ok,%MyApp.Sample{...}}
iex(3)MyApp.Repo.allMyApp.Sample
[%MyApp.Sample{...}]
Nowwhathappensiftrytorunthesamplequeryonthe"public"prefix?Todoso,let'sbuildaquerystructandsettheprefixfieldmanually:
iex(4)>query=Ecto.Queryable.to_queryMyApp.Sample
#Ecto.Query<fromsinMyApp.Sample>
iex(5)>MyApp.Repo.all%{query|prefix:"public"}
[]
5.Multitenancywithqueryprefixes
27
Noticehowwewereabletochangetheprefixthequeryrunson.Backinthedefault"public"prefix,thereisnodata!
Ecto2.1alsosupportsthe:prefixoptiononallrelevantrepositoryoperations:
iex(6)>MyApp.Repo.allMyApp.Sample
[%MyApp.Sample{...}]
iex(7)>MyApp.Repo.allMyApp.Sample,prefix:"public"
[]
OneinterestingaspectofprefixesinEctoisthattheprefixinformationiscarriedalongeachstruct:
iex(8)[sample]=MyApp.Repo.allMyApp.Sample
[%MyApp.Sample{}]
iex(9)>Ecto.get_meta(sample,:prefix)
nil
Theexampleabovereturnednil,whichmeansnoprefixwasspecifiedbyEcto,andthereforethedatabaseconnectiondefaultwillbeused.Inthiscase,"global_prefix"willbeusedbecauseofthe:after_connectcallbackweaddedatthebeginningofthischapter.
Sincetheprefixdataiscarriedinthestruct,wecanusesuchtocopydatafromoneprefixtotheother.Let'scopythesampleabovefromthe"global_prefix"tothe"public"one:
iex(10)>public_sample=Ecto.put_meta(sample,prefix:"public")
%MyApp.Sample{}
iex(11)>MyApp.Repo.insertpublic_sample
{:ok,%MyApp.Sample{}}
iex(12)>MyApp.Repo.allMyApp.Sample,prefix:"public"
[%MyApp.Sample{}]
Nowwehavedatainsertedinbothprefixes.
Prefixesinqueriesandstructsalwayscascade.Forexample,ifyourunMyApp.Repo.preload(sample,[:some_association]),theassociationwillbequeriedforandloadedinthesameprefixasthesamplestruct.IfsamplehasassociationsandyoucallMyApp.Repo.insert(sample)orMyApp.Repo.update(sample),theassociateddatawillalsobeinserted/updatedinthesameprefixassample.That'sbydesigntofacilitateworkingwithgroupsofdatainthesameprefix,andespeciallybecausedataindifferentprefixesmustbekeptisolated.
Migrationprefixes
5.Multitenancywithqueryprefixes
28
SofarwehaveexploredhowtosetaglobalprefixusingPostgres'andhowtosettheprefixatthequeryorstructlevel.Whentheglobalprefixisset,italsochangestheprefixmigrationsrunon.Howeveritisalsopossibletosettheprefixthroughthecommandlineorpertableinthemigrationitself.
Forexample,imagineyouareagamingcompanywherethegameisbrokenin128partitions,named"prefix_1","prefix_2","prefix_3"upto"prefix_128".Now,wheneveryouneedtomigratedata,youneedtomigratedataonalldifferent128prefixes.Therearetwowaysofachievethat.
Thefirstmechanismistoinvokemixecto.migratemultipletimes,onceperprefix,passingthe--prefixoption:
$mixecto.migrate--prefix"prefix_1"
$mixecto.migrate--prefix"prefix_2"
$mixecto.migrate--prefix"prefix_3"
...
$mixecto.migrate--prefix"prefix_128"
Theotherapproachisbychangingeachdesiredmigrationtorunacrossmultipleprefixes.Forexample:
defmoduleMyApp.Repo.Migrations.CreateSampledo
useEcto.Migration
defchangedo
fori<-1..128do
prefix="prefix_#{i}"
createtable(:samples,prefix:prefix)do
add:name,:string
timestamps()
end
#Executethecommandsonthecurrentprefix
#beforemovingontothenextprefix
flush()
end
end
end
SchemaprefixesFinally,Ecto2.1addstheabilitytosetaparticularschematorunonaspecificprefix.Imagineyouarebuildingamulti-tenantapplication.Eachclientdatabelongstoaparticularprefix,suchas"client_foo","client_bar"andsoforth.Yetyourapplicationmaystillrelyona
5.Multitenancywithqueryprefixes
29
setoftablesthatissharedacrossallclients.OneofsuchtablesmaybeexactlythetablethatmapstheClientIDtoitsdatabaseprefix.Let'sassumewewanttostorethisdatainaprefixnamed"main":
defmoduleMyApp.Mappingdo
useEcto.Schema
@schema_prefix"main"
schema"mappings"do
field:client_id,:integer
field:db_prefix
timestamps
end
end
NowrunningMyApp.Repo.allMyApp.Mappingwillbydefaultrunonthe"main"prefix,regardlessofthevalueconfiguredgloballyonthe:after_connectcallback.Similarwillhappentoinsert,update,andsimilaroperations,unlessthe:prefixisexplicitlychangedviaEcto.put_meta/2orbypassingthe:prefixoptiontotherepositoryoperation.
Keepinmind,however,thatqueriesrunonasingleprefix.Forexample,ifMyApp.Mappingonprefix"main"dependsonaschemanamedMyApp.Otheronprefix"another",aquerystartingwithMyApp.Mappingwillalwaysrunonthe"main"prefix.Bydesignitisnotpossibletoperformqueryjoinsacrossprefixes.Ifdatabelongstodifferentprefixes,itisbesttonotcouplethemstructurallynorviaqueries,inordertokeepdataindifferentprefixesisolated.
SummingupEcto2.0providesmanyconveniencesforworkingwithqueryingprefixes.ThoseconvenienceshavebeenfurtherimprovedinEcto2.1,allowingdeveloperstoconfigureprefixwithdifferentlevelofgranularity:
globalprefixes>schemaprefix>query/structprefixes
Thisallowsdeveloperstotackledifferentscenarios,fromproductionrequirementstomulti-tenantapplications.Ourjourneyonexploringthenewqueryconstructsisalmostover.Thenextandlastquerychapterisonaggregatesandsubqueries.
5.Multitenancywithqueryprefixes
30
AggregatesandsubqueriesThelastfeatureswewilldiscussregardingEctoqueriesareaggregatesandsubqueries.Aswewilllearn,thoseareintrinsicallyrelated.
AggregatesEcto2.0includesaconveniencefunctioninrepositoriestocalculateaggregates.
Forexample,tofindtheaveragenumberofvisitsacrossallposts:
MyApp.Repo.aggregate(MyApp.Post,:avg,:visits)
#=>#Decimal<1743>
Behindthescenes,thequeryabovetranslatesto:
MyApp.Repo.one(frompinMyApp.Post,select:avg(p.visits))
Theaggregate/3functionsupportsanyoftheaggregatefunctionslistedintheEctoQueryAPI.
Atfirst,itlooksliketheimplementationofaggregate/3isquitestraight-forward.YoucouldevenstarttowonderwhyitwasaddedtoEctointhefirstplace.However,complexitiesstarttoariseonqueriesthatrelyonlimit,offsetordistinctclauses.
Imaginethatinsteadofcalculatingtheaverageofallposts,youwanttheaverageofonlythetop10.Youmaytrytoitasfollows:
MyApp.Repo.one(frompinMyApp.Post,
order_by:[desc::visits],
limit:10,
select:avg(p.visits))
#=>#Decimal<1743>
Oops.Thequeryabovereturnedthesamevalueasthequeriesbefore.Theoptionlimit:10hasnoeffectheresinceitislimitingtheaggregatedresultandquerieswithaggregatesreturnonlyasinglerowanyway.Inordertoretrievethecorrectresult,wewouldneedtofirstfindthetop10postsandonlythenaggregate.That'sexactlywhataggregate/3does:
6.Aggregatesandsubqueries
31
query=fromMyApp.Post,order_by:[desc::visits],limit:10
MyApp.Repo.aggregate(query,:avg,:visits)#=>#Decimal<4682>
Whenlimit,offsetordistinctisspecifiedinthequery,aggregate/3automaticallywrapsthegivenqueryinasubquery.Thequeryexecutedbyaggregate/3abovewouldbeequivalentto:
query=fromMyApp.Post,order_by:[desc::visits],limit:10
MyApp.Repo.one(fromqinsubquery(query),select:avg(q.visits))
Let'stakeacloserlookatsubqueries.
SubqueriesIntheprevioussectionwehavealreadylearnedsomequeriesthatwouldbehardtoexpresswithoutsupportforsubqueries.That'soneofmanyexamplesthatledtosubquerysupportbeingaddedtoEcto.
SubqueriesinEctoarecreatedbycallingEcto.Query.subquery/1.ThefunctionreceivesanydatastructurethatcanbeconvertedtoaqueryviatheEcto.Queryableprotocolandreturnsasubqueryconstruct(whichisalsoqueryable).
InEcto2.0,itisallowedforasubquerytoselectawholetable(p)orafield(p.field).Allfieldsselectedinasubquerycanbeaccessedfromtheparentquery.Let'srevisittheaggregatequerywesawintheprevioussection:
query=fromMyApp.Post,order_by:[desc::visits],limit:10
MyApp.Repo.one(fromqinsubquery(query),select:avg(q.visits))
Becausethequerydoesnotspecifya:selectclause,itwillreturnselect:pwherepiscontrolledbyMyApp.Postschema.SincethequerywillreturnallfieldsinMyApp.Post,whenweconvertittoasubquery,allofthefieldsfromMyApp.Postwillbeavailableontheparentquery,suchasq.visits.Infact,Ectowillkeeptheschemapropertiesacrossqueries.Forexample,ifyouwriteq.field_that_does_not_exist,yourEctoquerywon'tcompile.
Ecto2.1furtherimprovessubqueriesbyallowinganElixirmaptobereturnedfromasubquery,makingthemapfieldsdirectlyavailabletotheparentquery.
Let'sseeonelastexample.Imagineyoumanagealibrary(anactuallibraryintherealworld)andthereisatablethatlogseverytimethelibrarylendsabook.The"lendings"tableusesauto-incrementingindexesandcanbebackedbythefollowingschema:
6.Aggregatesandsubqueries
32
defmoduleLibrary.Lendingdo
useEcto.Schema
schema"lendings"do
belongs_to:book,MyApp.Book#definesbook_id
belongs_to:visitor,MyApp.Visitor#definesvisitor_id
end
end
Nowconsiderwewanttoretrievethenameofeverybookalongsidethenameofthelastpersonthelibraryhaslentitto.Todoso,weneedtofindthelastlendingIDofeverybook,andthenjoinonthebookandvisitortables.Withsubqueries,that'sstraight-forward:
last_lendings=
fromlinMyApp.Lending,
group_by:l.book_id,
select:%{book_id:l.book_id,last_lending_id:max(l.id)}
fromlinLending,
join:lastinsubquery(last_lendings),
on:last.last_lending_id==l.id,
join:binassoc(l,:book),
join:vinassoc(l,:visitor),
select:{b.name,v.name}
SubqueriesareanimportantimprovementtoEctowhichmakesitpossibletoexpressqueriesthatwerenotpossiblebefore.Ontopofthat,wewereabletoaddfeaturessuchasaggregates,whichprovideusefulfunctionalitywhileshieldingtheuserfromcornercases.
6.Aggregatesandsubqueries
33
ImprovedassociationsandfactoriesEcto2.0largelyimprovedhowassociationswork.Tounderstandwhyandhow,let'stalkaboutEcto'soriginaldesigngoals.
EctofirststartedasaSummerofCodeprojectfromEricMeadows-Jönsson,todaypartoftheElixirteamandcreatorofHex,withJoséValim,creatorofElixir,asmentor.Thiswasbackin2013andElixirwasstillatversion0.9!
Similartomanyprojectsatthetime,oneofthegoalsbehindEctowastovalidateElixiritselfasaprogramminglanguage.Oneofthequestionsitaimedtoanswerwas:"canElixirbeusedtocreateadatabasewrapperthatisperformantandsecure?".Bybecomingastable,performantandsecurefoundation,Ectowouldbeabletoaddsyntaxsugar,conveniencesanddynamismlateron-whiletheoppositedirectionwouldhavebeenexceptionallyhardintheexperienceoftheEctoteam.
Ecto1.0becamethissecureandperformantfoundation.Asaconsequence,itfeltrigidinmanyaspectsandthosewerefrequentlyreportedaslimitationsbythecommunity.
Ecto2.0improvesonthemistakesmadebyEcto1.0andthenbuildsontopofitsfoundationbyaddingtheflexibilitiesusershavelongedfor.Wehaveexploredmanyoftheminthefirstchaptersofthisbook:schemalessqueries,schemalesschangesets,dynamicqueriesandmore.Inthenextthreechapters,wewillexploretheenhancementsdonetoschemasandassociations.
Inthisparticularchapter,wewilllearnhowEctoiscapableofinsertingcomplexdatastructureswithouttheneedtousechangesetsandhowtousethisfeaturetomanagecomplexdata,incasessuchasyourapplicationtestsuite,withoutaneedtorelyonexternalprojects.
LesschangesetsAtthesametimeEcto2.0bringsmanyfeaturestochangesets,itmakeschangesetslessnecessarythroughoutEctoAPIs.Forexample,inEcto1.0,Ecto.Repo.insert/2requiredchangesets.Thismeansthat,inordertoinsertanyentrytothedatabase,suchasapost,wehadtowrapitinachangesetfirst:
%Post{title:"helloworld"}
|>Ecto.Changeset.change
|>Repo.insert!()
7.Improvedassociationsandfactories
34
ThisreflectedthroughoutEcto1.0APIs.Ifyouwantedtocreateapostwithsomecomments,youhadtowrapeachcommentinachangesetandthenputitinthepostchangeset:
comment1=%Comment{body:"excellentarticle"}|>Ecto.Changeset.change
comment2=%Comment{body:"Ilearnedsomethingnew"}|>Ecto.Changeset.change
%Post{title:"helloworld"}
|>Ecto.Changeset.put_assoc(:comments,[comment1,comment2])
|>Repo.insert!()
Furthermore,whenhandlingassociations,Ecto1.0forcedyoutoalwayswritetheparentchangesetfirstandthenthechildren.Sotheexampleabovewhereinsertapost(theparent)withmultiplecomments(children)workedbutthefollowingexamplewouldnot:
post=%Post{title:"helloworld"}|>Ecto.Changeset.change()
%Comment{body:"excellentarticle"}
|>Ecto.Changeset.put_assoc(:post,[post])
|>Repo.insert!()
Ecto2.0goesawaywithallofthoselimitations.YoucannowpassstructstotherepositoryandchangesetsandEctowilltakecareofbuildingthechangesetsforyoubehindthescenes.InEcto2.0,apostwithcommentscanbeinserteddirectlyasfollows:
Repo.insert!%Post{
title:"helloworld",
comments:[
%Comment{body:"excellentarticle"},
%Comment{body:"Ilearnedsomethingnew"}
]
}
Youarealsoabletoinsertandupdateassociationsfromanydirection,beitfromparenttochildorchildtoparent:
Repo.insert!%Comment{
body:"excellentarticle",
post:%Post{title:"helloworld"}
}
Thisfeatureisnotonlyusefulwhenwritingourapplicationsbutalsowhentestingthem,aswewillseenext.
7.Improvedassociationsandfactories
35
TestfactoriesManyprojectsdependonexternallibrariestobuildtheirtestdata.Someofthoselibrariesarecalledfactoriesbecausetheyprovideconveniencefunctionsforbuildingdifferentgroupsofdata.However,givenEcto2.0isabletomanagecomplexdatatrees,wecanimplementsuchfunctionalitywithoutrelyingonthird-partyprojects.
Togetstarted,let'screateafileat"test/support/factory.ex"withthefollowingcontents:
defmoduleMyApp.Factorydo
aliasMyApp.Repo
#Factories
defbuild(:post)do
%MyApp.Post{title:"helloworld"}
end
defbuild(:comment)do
%MyApp.Comment{body:"goodpost"}
end
defbuild(:post_with_comments)do
%MyApp.Post{
title:"hellowithcomments",
comments:[
build(:comment,body:"first"),
build(:comment,body:"second")
]
}
end
defbuild(:user)do
%MyApp.User{
email:"hello#{System.unique_integer()}",
username:"hello#{System.unique_integer()}"
}
end
#ConvenienceAPI
defbuild(factory_name,attributes)do
factory_name|>build()|>struct(attributes)
end
definsert!(factory_name,attributes\\[])do
Repo.insert!build(factory_name,attributes)
end
end
7.Improvedassociationsandfactories
36
Ourfactorymoduledefinesfour"factories"asdifferentclausestothebuildfunction::post,:comment,:post_with_commentsand:user.Eachclausedefinestructswiththefieldsthatrequiredbythedatabase.Incertaincases,thegeneratedstructalsoneedtogenerateuniquefields,suchastheuser'semailandusername.WedidsobycallingElixir'sSystem.unique_integer()-youcouldcallSystem.unique_integer([:positive])ifyouneedastrictlypositivenumber.
Attheend,wedefinedtwofunctions,build/2andinsert!/2,whichareconveniencesforbuildingstructswithspecificattributesorforinsertingdatadirectlyintherepository.
That'sliterallyallthatisnecessaryforbuildingourfactories.Wearenowreadytousefactoriesinourtests.First,openupyour"mix.exs"fileandlet'smakesurethe"test/support/factory.ex"fileiscompiled:
defprojectdo
[...,
elixirc_paths:elixirc_paths(Mix.env),
...]
end
defpelixirc_paths(:test),do:["lib","test/support"]
defpelixirc_paths(_),do:["lib"]
Nowinanyoftheteststhatneedtogeneratedata,wecanimporttheMyApp.Factorymoduleanduseitsfunctions:
importMyApp.Factory
build(:post)
#=>%MyApp.Post{id:nil,title:"helloworld",...}
build(:post,title:"customtitle")
#=>%MyApp.Post{id:nil,title:"customtitle",...}
insert!(:post,title:"customtitle")
#=>%MyApp.Post{id:...,title:"customtitle"}
BybuildingthefunctionalityweneedontopofEctocapabilities,weareabletoextendandimproveourfactoriesonwhateverwaywedesire,withoutbeingconstrainedtothird-partylimitations.
7.Improvedassociationsandfactories
37
ManytomanyandcastingBesidesbelong_to,has_many,has_oneand:throughassociations,Ecto2.0alsoincludesmany_to_many.many_to_manyrelationships,asthenamesays,allowsXtohavemanyassocaitedentriesYandvice-versa.Althoughmany_to_manyassociationscanbewrittenasahas_many:through,usingmany_to_manymayconsiderablysimplifysomeworkflows.
Inthischapter,wewilltalkaboutpolymorphicassociationsandhowmany_to_manycanconsiderablyremoveboilerplatefromcertainapproachescomparedtohas_many:through.
Todolistsv65131Thewebhasseenitsshareoftodolistapplications.Butthatwon'tstopusfromcreatingourown!
Inourcase,thereisoneaspectoftodolistapplicationsweareinterestedon,whichistherelationshipthetodolistwhichhasmanytodoitems.WehaveexploredthisexactscenarioindetailinanarticlewepostedonPlataformatec'sblogaboutnestedassociationsandembeds.Let'srecaptheimportantpoints.
Inordertomodelatodolistapp,weneedtohavetwoschemas,Todo.ListandTodo.Item:
defmoduleMyApp.TodoListdo
useEcto.Schema
schema"todo_lists"do
field:title
has_many:todo_items,MyApp.TodoItem
timestamps()
end
end
defmoduleMyApp.TodoItemdo
useEcto.Schema
schema"todo_items"do
field:description
timestamps()
end
end
8.Manytomanyandcasting
38
OneofthewaystointroduceatodolistwithmultipleitemsintothedatabaseistocoupleourUIrepresentationtoourschemas.That'stheapproachwetookintheblogpostabovewithPhoenix.Roughly:
<%=form_for@todo_list_changeset,todo_list_path(@conn,:create),fnf->%>
<%=text_inputf,:title%>
<%=inputs_forf,:todo_items,fni->%>
...
<%end%>
<%end%>
WhensuchaformissubmittedinPhoenix,itwillsendparameterswiththefollowingshape:
%{"todo_list"=>%{
"title"=>"shippinglist",
"todo_items"=>%{
0=>%{"description"=>"bread"},
1=>%{"description"=>"eggs"},
}
}}
WecouldthenretrievethoseparametersandpassittoanEctochangesetandEctowouldautomaticallyfigureoutwhattodo:
#InMyApp.TodoList
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast(params,[:title])
|>Ecto.Changeset.cast_assoc(:todo_list_items,required:true)
end
#AndtheninMyApp.TodoItem
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast(params,[:description])
end
BycallingEcto.Changeset.cast_assoc/3,Ectowilllookfora"todo_items"keyinsidetheparametersgivenoncast,andcomparethoseparameterswiththeitemsstoredinthetodoliststructs.Ectowillautomaticallygenerateinstructionstoinsert,updateordeletetodoitemssuchthat:
ifatodoitemsentasparameterhasanIDanditmatchesanexistingassociatedtodoitem,weconsiderthattodoitemshouldbeupdatedifatodoitemsentasparameterdoesnothaveanID(noramatchingID),weconsider
8.Manytomanyandcasting
39
thattodoitemshouldbeinsertedifatodoitemiscurrenetlyassociatedbutitsIDwasnotsentasparameter,weconsiderthetodoitemisbeingreplacedandweactaccordingtothe:on_replacecallback.Bydefault:on_replacewillraisesoyouchooseabehaviourbetweenreplacing,deleting,ignoringornilifyingtheassociation
Theadvantageofusingcast_assoc/3isthatEctoisabletodoallofthehardworkofkeepingtheentriesassociated,aslongaswepassthedataexactlyintheformatthatEctoexpects.However,aswelearnedinthefirstthreechaptersofthisbook,suchapproachisnotalwayspreferrableandinmanysituationsitisbettertodesignourassociationsdifferentlyordecoupleourUIsfromourdatabaserepresentation.
PolymorphictodoitemsToshowanexampleofwhereusingcast_assoc/3isjusttoocomplicatedtobeworthit,let'simagineyouwantyour"todoitems"tobepolymorphic.Forexample,youwanttobeabletoaddtodoitemsnotonlyto"todolists"buttomanyotherpartsofyourapplication,suchasprojects,dates,younameit.
Firstofall,itisimportanttorememberEctodoesnotprovidethesametypeofpolymorphicassociationsknowninframeworkssuchasRailsandLaravel.Insuchframeworks,apolymorphicassociationusestwocolumns,theparent_idandparent_type.Forexample,onetodoitemwouldhaveparent_idof1withparent_typeof"TodoList"whileanotherwouldhaveparent_idof1withparent_typeof"Project".
Theissuewiththedesignaboveisthatitbreaksdatabasereferences.Thedatabaseisnolongercapableofguaranteeingtheitemyouassociatetoexistsorwillcontinuetoexistinthefuture.Thisleadstoaninconsistentdatabasewhichend-uppushingworkaroundstoyourapplication.
Thedesignaboveisalsoextremelyinefficient.Inthepastwehaveworkedwithalargeclientonremovingsuchpolymorphicreferencesbecausefrequentpolymorphicqueriesweregrindingthedatabasetoahaltevenafteraddingindexesandoptimizingthedatabase.
Luckily,thedocumentationforthebelongs_tomacroincludesexamplesonhowtodesignsaneandperformantassociations.Oneofthoseapproachesconsistsinusingmanyjointables.Besidesthe"todo_lists"and"projects"tablesandthe"todo_items"table,wewouldcreate"todo_list_items"and"project_items"toassociatetodoitemstotodolistsandtodoitemstoprojectsrespectively.Intermsofmigrations,wearelookingatthefollowing:
8.Manytomanyandcasting
40
createtable("todo_lists")do
add:title
timestamps()
end
createtable("projects")do
add:name
timestamps()
end
createtable("todo_items")do
add:description
timestamps()
end
createtable("todo_lists_items")do
add:todo_item_id,references(:todo_items)
add:todo_list_id,references(:todo_lists)
timestamps()
end
createtable("projects_items")do
add:todo_item_id,references(:todo_items)
add:project_id,references(:projects)
timestamps()
end
Firstlet'sseehowimplementthisfunctionalityusingahas_many:throughandthenusemany_to_manytoremovealotoftheboilerplatewewereforcedtointroduce.
Polymorphismwithhas_many:throughGivenwewantourtodoitemstobepolymorphic,wecannolongerassociateatodolisttotodoitemsdirectly.InsteadwewillcreateanintermediateschematotieMyApp.TodoListandMyApp.TodoItemtogether.
8.Manytomanyandcasting
41
defmoduleMyApp.TodoListdo
useEcto.Schema
schema"todo_lists"do
field:title
has_many:todo_list_items,MyApp.TodoListItem
has_many:todo_items,through:[:todo_list_items,:todo_item]
timestamps()
end
end
defmoduleMyApp.TodoListItemdo
useEcto.Schema
schema"todo_list_items"do
belongs_to:todo_list,MyApp.TodoList
belongs_to:todo_item,MyApp.TodoItem
timestamps()
end
end
defmoduleMyApp.TodoItemdo
useEcto.Schema
schema"todo_items"do
field:description
timestamps()
end
end
AlthoughweintroducedMyApp.TodoListItemasanintermediateschema,has_many:throughallowsustoaccessalltodoitemsforanytodolisttransparently:
todo_lists|>Repo.preload(:todo_items)
Thetroubleisthat:throughassociationsareread-onlysinceEctodoesnothaveenoughinformationtofillintheintermediateschema.Thismeansthat,ifwestillwanttousecast_assoctoinsertatodolistwithmanytodoitemsdirectlyfromtheUI,wewouldneedtofirstcast_assoc(:todo_list_items)fromTodoListandthencallcast_assoc(:todo_item)fromtheTodoListItemschema:
8.Manytomanyandcasting
42
#InMyApp.TodoList
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast(params,[:title])
|>Ecto.Changeset.cast_assoc(:todo_list_items,required:true)
end
#AndthenintheMyApp.TodoListItem
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast_assoc(:todo_item,required:true)
end
#AndtheninMyApp.TodoItem
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast(params,[:description])
end
Tofurthercomplicatethings,remembercast_assocexpectsaparticularshapeofdatathatreflectsyourassociations.Inthiscase,becauseoftheintermediateschema,thedatasentthroughyourformsinPhoenixwouldhavetolookasfollows:
%{"todo_list"=>%{
"title"=>"shippinglist",
"todo_list_items"=>%{
0=>%{"todo_item"=>%{description"=>"bread"}},
1=>%{"todo_item"=>%{description"=>"eggs"}},
}
}}
Tomakemattersworse,youwouldhavetoduplicatethislogicforeveryintermediateschema,bydefiningMyApp.TodoListItemfortodolists,MyApp.ProjectItemforprojects,etc.
Luckily,many_to_manyallowsustoremoveallofthisboilerplate.
Polymorphismwithmany_to_manyInaway,theideabehindmany_to_manyassociationsisthatitallowsustoassociatetwoschemasviaanintermediateschemaswhileautomaticallytakingcareofalldetailsabouttheintermediateschema.Let'srewritetheschemasabovetousemany_to_many:
8.Manytomanyandcasting
43
defmoduleMyApp.TodoListdo
useEcto.Schema
schema"todo_lists"do
field:title
many_to_many:todo_items,join_through:MyApp.TodoListItem
timestamps()
end
end
defmoduleMyApp.TodoListItemdo
useEcto.Schema
schema"todo_list_items"do
belongs_to:todo_list,MyApp.TodoList
belongs_to:todo_item,MyApp.TodoItem
timestamps()
end
end
defmoduleMyApp.TodoItemdo
useEcto.Schema
schema"todo_items"do
field:description
timestamps()
end
end
NoticeMyApp.TodoListnolongerneedstodefineahas_manyassociationpointingtotheMyApp.TodoListItemschemaandinsteadwecanjustassociateto:todo_itemsusingmany_to_many.
Differentlyfromhas_many:through,many_to_manyassociationsarealsowriteable.Thismeanswecansenddatathroughourformsexactlyaswedidatthebeginningofthischapter:
%{"todo_list"=>%{
"title"=>"shippinglist",
"todo_items"=>%{
0=>%{"description"=>"bread"},
1=>%{"description"=>"eggs"},
}
}}
Andwenolongerneedtodefineachangesetfunctionintheintermediateschema:
8.Manytomanyandcasting
44
#InMyApp.TodoList
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast(params,[:title])
|>Ecto.Changeset.cast_assoc(:todo_items,required:true)
end
#AndtheninMyApp.TodoItem
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast(params,[:description])
end
Inotherwords,wecanuseexactlythesamecodewehadinthe"todolistshas_manytodoitems"case.Soevenwhenexternalconstraintsrequireustouseajointable,many_to_manyassociationscanautomaticallymanagethemforus.Everythingyouknowaboutassociationswilljustworkwithmany_to_manyassociations,includingtheimprovementswediscussedinthepreviouschapter.
Finally,eventhoughwehavespecifiedaschemaasthe:join_throughoptioninmany_to_many,many_to_manycanalsoworkwithoutintermediateschemasaltogetherbysimplygivingitatablename:
defmoduleMyApp.TodoListdo
useEcto.Schema
schema"todo_lists"do
field:title
many_to_many:todo_items,join_through:"todo_list_items"
timestamps()
end
end
Inthiscase,youcancompletelyremovetheMyApp.TodoListItemschemafromyourapplicationandthecodeabovewillstillwork.Theonlydifferenceisthatwhenusingtables,anyautogeneratedvaluethatisfilledbyEctoschema,suchastimestamps,won'tbefilled(aswenolongerhaveaschema).Tosolvethis,youcaneitherdropthosefieldsfromyourmigrationsorsetadefaultatthedatabaselevel.
SummaryInthischapterweusedmany_to_manyassociationstodrasticallyimproveapolymorphicassociationdesignthatreliedonhas_many:through.Ourgoalwastoallow"todo_items"toassociatetodifferententitiesinourcodebase,suchas"todo_lists"and"projects".Wehave
8.Manytomanyandcasting
45
donethisbycreatingintermediatetablesandbyusingmany_to_manyassociationstoautomaticallymanagethosejointables.
Attheend,ourschemasmaylooklike:
defmoduleMyApp.TodoListdo
useEcto.Schema
schema"todo_lists"do
field:title
many_to_many:todo_items,join_through:"todo_list_items"
timestamps()
end
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast(params,[:title])
|>Ecto.Changeset.cast_assoc(:todo_items,required:true)
end
end
defmoduleMyApp.Projectdo
useEcto.Schema
schema"todo_lists"do
field:name
many_to_many:todo_items,join_through:"project_items"
timestamps()
end
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast(params,[:name])
|>Ecto.Changeset.cast_assoc(:todo_items,required:true)
end
end
defmoduleMyApp.TodoItemdo
useEcto.Schema
schema"todo_items"do
field:description
timestamps()
end
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast(params,[:description])
end
end
8.Manytomanyandcasting
46
Andthedatabasemigration:
createtable("todo_lists")do
add:title
timestamps()
end
createtable("projects")do
add:name
timestamps()
end
createtable("todo_items")do
add:description
timestamps()
end
#Primarykeyandtimestampsarenotrequiredifusingmany_to_manywithoutschemas
createtable("todo_lists_items",primary_key:false)do
add:todo_item_id,references(:todo_items)
add:todo_list_id,references(:todo_lists)
#timestamps()
end
#Primarykeyandtimestampsarenotrequiredifusingmany_to_manywithoutschemas
createtable("projects_items",primary_key:false)do
add:todo_item_id,references(:todo_items)
add:project_id,references(:projects)
#timestamps()
end
Overallourcodelooksstructurallythesameashas_manywould,althoughatthedatabaselevelourrelationshipsareexpressedwithjointables.
Whileinthischapterwechangedourcodetocopewiththeparameterformatrequiredbycast_assoc,inthenextchapterwewilldropcast_assocaltogetheranduseput_assocwhichbringsmoreflexibilitieswhenworkingwithassociations.
8.Manytomanyandcasting
47
ManytomanyandupsertsInthepreviouschapterwehavelearnedaboutmany_to_manyassociationsandhowtomapexternaldatatoassociatedentrieswiththehelpofEcto.Changeset.cast_assoc/3.Whileinthepreviouschapterwewereabletofollowtherulesimposedbycast_assoc/3,doingsoisnotalwayspossiblenordesired.
Inthischapter,wearegoingtolookatEcto.Changeset.put_assoc/4incontrasttocast_assoc/3andexploresomeexamples.WewillalsopeekattheupsertfeaturescominginEcto2.1.
put_assocvscast_assocImaginewearebuildinganapplicationthathasblogpostsandsuchpostsmayhavemanytags.Notonlythat,agiventagmayalsobelongtomanyposts.Thisisaclassicscenariowherewewouldusemany_to_manyassociations.Ourmigrationswouldlooklike:
createtable(:posts)do
add:title
add:body
timestamps()
end
createtable(:tags)do
add:name
timestamps()
end
createunique_index(:tags,[:name])
createtable(:posts_tags,primary_key:false)do
add:post_id,references(:posts)
add:tag_id,references(:tags)
end
Noteweaddedauniqueindextothetagnamebecausewedon'twanttohaveduplicatedtagsinourdatabase.Itisimportanttoaddanindexatthedatabaselevelinsteadofusingavalidationsincethereisalwaysachancetwotagswiththesamenamewouldbevalidatedandinsertedsimultaneously,passingthevalidationandleadingtoduplicatedentries.
9.Manytomanyandupserts
48
Nowlet'salsoimaginewewanttheuserinputsuchtagsasalistofwordssplitbycomma,suchas:"elixir,erlang,ecto".Oncethisdataisreceivedintheserver,wewillbreakitapartintomultipletagsandassociatethemtothepost,creatinganytagthatdoesnotyetexistinthedatabase.
Whiletheconstraintsabovesoundreasonable,that'sexactlywhatputusintroublewithcast_assoc/3.Rememberthecast_assoc/3changesetfunctionwasdesignedtoreceiveexternalparametersandcomparethemwiththeassociateddatainourstructs.Todosocorrectly,Ectorequirestagstobesentasalistofmaps.Howeverhereweexpecttagstobesentinastringseparatedbycomma.
Furthermorecast_assoc/3reliesontheprimarykeyfieldforeachtagsentinordertodecideifitshouldbeinserted,updatedordeleted.Again,becausetheuserissimplypassingastring,wedon'thavetheIDinformationathand.
Whenwecan'tcopewithcast_assoc/3,itistimetouseput_assoc/4.Input_assoc/4,wegiveEctostructsorchangesetsinsteadofparameters,givingustheabilitytomanipulatethedataaswewant.Let'sdefinetheschemaandthechangesetfunctionforapostwhichmayreceivetagsasastring:
9.Manytomanyandupserts
49
defmoduleMyApp.Postdo
useEcto.Schema
schema"posts"do
add:title
add:body
many_to_many:tags,MyApp.Tag,join_through:"posts_tags"
timestamps()
end
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast(struct,[:title,:body])
|>Ecto.Changeset.put_assoc(:tags,parse_tags(params))
end
defpparse_tags(params)do
(params["tags"]||"")
|>String.split(",")
|>Enum.map(&String.trim/1)
|>Enum.reject(&&1=="")
|>Enum.map(&get_or_insert_tag/1)
end
defpget_or_insert_tag(name)do
Repo.get_by(MyApp.Tag,name:name)||
Repo.insert!(MyApp.Tag,%Tag{name:name})
end
end
Inthechangesetfunctionabove,wemovedallthehandlingoftagstoaseparatefunction,calledparse_tags/1,whichchecksfortheparameter,breaksitsentriesapartviaString.split/2,thenremovesanyleftoverwhitespacewithString.trim/1,rejectsanyemptystringandfinallychecksifthetagexistsinthedatabaseornot,creatingoneincasenoneexists.
Theparse_tags/1functionisgoingtoreturnalistofMyApp.Tagstructswhicharethenpassedtoput_assoc/3.Bycallingput_assoc/3,wearetellingEctothoseshouldbethetagsassociatedtothepostfromnowon.Incaseaprevioustagwasassociatedtothepostandnotgiveninput_assoc/3,Ectowillalsotakecareofremovingtheassociationbetweenthepostandtheremovedtagfromthedatabase.
Andthat'sallweneedtousemany_to_manyassociationswithput_assoc/3.put_assoc/3workswithhas_many,belongs_toandallothersassociationtypes.However,ourcodeisnotyetreadyforproduction.Let'sseewhy.
9.Manytomanyandupserts
50
ConstraintsandraceconditionsRememberweaddedauniqueindextothetag:namecolumnwhencreatingthetagstable.Wedidsotoprotectusfromhavingduplicatetagsinthedatabase.
Byaddingtheuniqueindexandthenusingget_bywithainsert!togetorinsertatag,weintroducedapotentialerrorinourapplication.Iftwopostsaresubmittedatthesametimewithasimilartag,thereisachancewewillcheckifthetagexistsatthesametime,leadingbothsubmissionstobelievethereisnosuchtaginthedatabase.Whenthathappens,onlyoneofthesubmissionswillsucceedwhiletheotheronewillfail.That'saracecondition:yourcodewillerrorfromtimetotime,onlywhencertainconditionsaremet.Andthoseconditionsaretimesensitive.
Manydevelopershaveatendencytothinksucherrorswon'thappeninpracticeor,iftheyhappened,theywouldbeirrelevant.Butinpracticetheyoftenleadtoveryfrustratinguserexperiences.Ihaveheardafirst-handexamplecomingfromamobilegamecompany.Inthegame,aplayerisabletoplayquestsandoneveryquestyouhavetochooseaguestcharacterfromanotherplayeroutofashortlisttogoonthequestwithyou.Attheendofthequest,youhavetheoptiontoaddtheguestcharacterasafriend.
Originallythewholeguestlistwasrandombut,astimepassed,playersstartedtocomplainsometimesoldaccounts,ofteninactive,werebeingshownintheguestsoptionslist.Toimprovethesituation,thegamedevelopersstartedtosorttheguestlistbymostrecentlyactive.Thismeansthat,ifyouhavejustplayedrecently,thereisahigherchanceofyoutobeonsomeoneguestlists.
However,whentheydidsuchchange,manyerrorsstartedtoshowupandusersweresuddenlyfuriousinthegameforum.That'sbecausewhentheysortedplayersbyactivity,assoontwoplayersloggedin,theircharacterswouldlikelyappearoneachothersguestlist.Ifthoseplayerspickedeachotherscharacters,thefirsttoaddtheotherasfriendattheendofaquestwouldbeabletosucceedbutanerrorwouldappearwhenthesecondplayertriedtoaddthatcharacterasafriendsincetherelationshipalreadyexistedinthedatabase!Notonlythat,alltheprogressdoneinthequestwouldbelost,becausetheserverwasunabletoproperlypersistthequestresultstothedatabase.Understandably,playersstartedtofilecomplaints.
Longstoryshort:wemustaddresstheracecondition.
LuckilyEctogivesusamechanismtohandleconstrainterrorsfromthedatabase.
Checkingforconstrainterrors
9.Manytomanyandupserts
51
Sinceourget_or_insert_tag(name)functionfailswhenatagalreadyexistsinthedatabase,weneedtohandlesuchscenariosaccordingly.Let'srewriteittakingraceconditionsintoaccountinmind:
defpget_or_insert_tag(name)do
%Tag{}
|>Ecto.Changeset.change(name:name)
|>Ecto.Changeset.unique_constraint(:name)
|>Repo.insert
|>casedo
{:ok,tag}->tag
{:error,_}->Repo.get_by!(MyApp.Tag,name:name)
end
end
Insteadofinsertingthetagdirectly,weknowbuildachangeset,whichallowsustousetheunique_constraintannotation.NowiftheRepo.insertoperationfailsbecausetheuniqueindexfor:nameisviolated,Ectowon'traise,butreturnan{:error,changeset}tuple.Therefore,iftheRepo.insertsucceeds,itisbecausethetagwassaved,otherwisethetagalreadyexists,whichwethenfetchwithRepo.get_by!.
Whilethemechanismabovefixestheracecondition,itisaquiteexpensiveone:weneedtoperformtwoqueriesforeverytagthatalreadyexistsinthedatabase:the(failed)insertandthentherepositorylookup.Giventhat'sthemostcommonscenario,wemaywanttorewriteittothefollowing:
defpget_or_insert_tag(name)do
Repo.get_by(MyApp.Tag,name:name)||maybe_insert_tag(name)
end
defpmaybe_insert_tag(name)do
%Tag{}
|>Ecto.Changeset.change(name:name)
|>Ecto.Changeset.unique_constraint(:name)
|>Repo.insert
|>casedo
{:ok,tag}->tag
{:error,_}->Repo.get_by!(MyApp.Tag,name:name)
end
end
Theaboveperforms1queryforeverytagthatalreadyexists,2queriesforeverynewtagandpossibly3queriesinthecaseofraceconditions.Whiletheabovewouldperformslightlybetteronaverage,Ecto2.1hasabetteroptioninstock.
9.Manytomanyandupserts
52
UpsertsEcto2.1supportstheso-called"upsert"commandwhichisanabbreviationfor"updateorinsert".Theideaisthatwetrytoinsertarecordandincaseitconflictswithanexistingentry,forexampleduetoauniqueindex,wecanchoosehowwewantthedatabasetoactbyeitherraisinganerror(thedefaultbehaviour),ignoringtheinsert(noerror)orbyupdatingtheconflictingdatabaseentries.
"upsert"inEcto2.1isdonewiththe:on_conflictoption.Let'srewriteget_or_insert_tag(name)oncemorebutthistimeusingthe:on_conflictoption.Alsorememberthat"upsert"isanewfeatureinPostgreSQL9.5,somakesureyouareuptodate:
defpget_or_insert_tag(name)do
Repo.insert!(%MyApp.Tag{name:name},on_conflict::nothing)
end
Andthat'sit!Wetrytoinsertatagwiththegivennameandifsuchtagalreadyexists,wetellEctotonoterrorandreturnthetagwehavepassedasargumentasifitwaspersisted.Whiletheaboveiscertainlyastepupfromallsolutionssofar,itstillperformsonequerypertag.If10tagsaresent,wewillperform10queries.Canwefurtherimproveit?
Upsertsandinsert_allEcto2.1didnotonlyaddthe:on_conflictoptiontoRepo.insert/2butalsototheRepo.insert_all/3functionintroducedinEcto2.0.Thismeanswecanbuildonequerythatattemptstoinsertallmissingtagsandthenanotherquerythatfetchesallofthematonce.Let'sseehowourPostschemawilllooklikeafterthosechanges:
9.Manytomanyandupserts
53
defmoduleMyApp.Postdo
useEcto.Schema
#Schemaisthesame
schema"posts"do
add:title
add:body
many_to_many:tags,MyApp.Tag,join_through:"posts_tags"
timestamps()
end
#Changesetisthesame
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast(struct,[:title,:body])
|>Ecto.Changeset.put_assoc(:tags,parse_tags(params))
end
#Parsetagshasslightlychanged
defpparse_tags(params)do
(params["tags"]||"")
|>String.split(",")
|>Enum.map(&String.trim/1)
|>Enum.reject(&&1=="")
|>insert_and_get_all()
end
defpinsert_and_get_all([])do
[]
end
defpinsert_and_get_all(names)do
maps=Enum.map(names,&%{name:&1})
Repo.insert_allMyApp.Tag,names,on_conflict::nothing
Repo.allfromtinMyApp.Tag,where:t.namein^names
end
end
Insteadofattemptingtogetandinserteachtagindividually,thecodeaboveworkonalltagsatonce,firstbybuildingalistofmapswhichisgiventoinsert_allandthenbylookingupalltagswiththeexistingnames.Therefore,regardlessofhowmanytagsaresent,wewillperformonly2queries(unlessnotagissent,inwhichwereturnanemptylistbackpromptly).ThissolutionisonlypossibleinEcto2.1thankstothe:on_conflictoption,whichguaranteesinsert_allwon'tfailincaseagiventagnamealreadyexists.
Finally,keepinmindthatwehaven'tusedtransactionsinanyoftheexamplessofar.Suchdecisionwasdeliberate.Sincegettingorinsertingtagsisanidempotentoperation,i.e.wecanrepeatitmanytimesanditwillalwaysgiveusthesameresultback.Therefore,evenifwefailtointroducetheposttothedatabaseduetoavalidationerror,theuserwillbefreeto
9.Manytomanyandupserts
54
resubmittheformandwewilljustattempttogetorinsertthesametagsonceagain.Thedownsideofthisapproachisthattagswillbecreatedevenifcreatingthepostfails,whichmeanssometagsmaynothavepostsassociatedtothem.Incasethat'snotdesired,thewholeoperationcouldbewrappedinatransactionormodeledwiththeEcto.Multiabstractionwewilldiscussinthenextchapter.
9.Manytomanyandupserts
55
ComposabletransactionswithEcto.MultiEctoreliesondatabasetransactionswhenmultipleoperationsmustbeperformedatomically.TransactionscanbeperformedviatheRepo.transactionfunction:
Repo.transaction(fn->
mary|>Ecto.Changeset.change(balance:mary.balance-10)|>Repo.update!
john|>Ecto.Changeset.change(balance:john.balance+10)|>Repo.update!
end)
Whenweexpectbothoperationstosucceed,asabove,transactionsarequitestraight-forward.However,transactionsgetmorecomplicatediftheoperationscanfail:
Repo.transaction(fn->
casemary|>Ecto.Changeset.change(balance:mary.balance-10)|>Repo.updatedo
{:ok,mary}->
casejohn|>Ecto.Changeset.change(balance:john.balance+10)|>Repo.updatedo
{:ok,john}->
{mary,john}
{:error,changeset}->
Repo.rollback({:john,changeset})
end
{:error,changeset}->
Repo.rollback({:mary,changeset})
end
end)
Ontheotherhand,transactionsinEctocanbenestedarbitrarily.Forexample,imaginethetransactionaboveismovedintoitsownfunction,definedastransfer_money(mary,john,10),andbesidestransferringmoneywealsowanttologthetransfer:
Repo.transaction(fn->
casetransfer_money(mary,john,10)do
{:ok,{mary,john}}->
Repo.insert!(%Transfer{from:mary.id,to:john.id,amount:10})
{:error,{who,changeset}}->
Repo.rollback({who,changeset})
end
end)
Thesnippetaboverunsinatransactionandthencallstransfer_money/3thatalsorunsinatransaction.ThisworksbecauseEctoconvertsanynestedtransactionintosavepointsautomatically.Incaseaninnertransactionfails,itrollsbacktoitsspecificsavepoint.
10.ComposabletransactionswithEcto.Multi
56
Whilenestingtransactionscanbeimprovethecodereadabilitybybreakingalargetransactionsintomultiplesmallertransactions,thereisstillalotofboilerplateinvolvedinhandlingthesuccessandfailurescenarios.Furthermore,compositionisquitelimited,asalloperationsmuststillbeperformedinsidetransactionblocks.
Amoredeclarativeapproachwouldbetodefinealloperationswewanttoperforminatransactiondecoupledfromthetransactionexecution.Thiswaywewouldbeabletocomposetransactionsoperationswithoutworryingaboutitsexecutioncontextorabouteachindividualsuccess/failurescenario.That'sexactlywhatEcto.Multiallowsustobuild.
ComposingwithdatastructuresLet'srewritethesnippetsaboveusingEcto.Multi.Thefirstsnippetthattransfersmoneybetweenmaryandjohncanrewrittento:
Ecto.Multi.new
|>Ecto.Multi.update(:mary,Ecto.Changeset.change(mary,balance:mary.balance-10))
|>Ecto.Multi.update(:john,Ecto.Changeset.change(john,balance:john.balance+10))
Ecto.Multiisadatastructurethatallowsustodefinewhichoperationsmustbeperformedtogether,withoutworryingaboutwhereandhowitwillbeexecuted.Ecto.MultimirrorsmostoftheEcto.RepoAPI,withthedifferenceeachoperationmustbeexplicitlynamed.Intheexampleabove,wehavedefinedtwoupdateoperations,named:maryand:john.Aswewillseelater,thenamesareimportantwhenhandlingsuccessandfailureresults.
SinceEcto.Multiisjustadatastructure,wecanpassitasargumenttootherfunctions,aswellasreturnit.Assumingthemultiaboveismovedintoitsownfunction,definedastransfer_money(mary,john,10),wecannowaddanoperationthatlogsthetransferasfollows:
transfer_money(mary,john,10)
|>Ecto.Multi.insert(:transfer,%Transfer{from:mary.id,to:john.id,amount:10})
Thisisconsiderablysimplerthanthenestedtransactionapproachedwehaveseenearlier!Oncealloperationsaredefinedinthemulti,wecanfinallycallRepo.transaction,thistimepassingthemulti:
10.ComposabletransactionswithEcto.Multi
57
transfer_money(mary,john,10)
|>Ecto.Multi.insert(:transfer,%Transfer{from:mary.id,to:john.id,amount:10})
|>Repo.transaction()
|>casedo
{:ok,%{mary:mary,john:john,transfer:transfer}}->
#Handlesuccesscase
{:error,name,value,rolled_back_changes}->
#Handlefailurecase
end
Ifalloperationsinthemultisucceed,itreturns{:ok,map}wherethemapcontainsthenameofalloperationsaskeysandtheirsuccessvalue.Ifanyoperationinthemultifails,thetransactionisrolledbackandRepo.transactionreturns{:error,name,value,rolled_back_changes},wherenameisthenameofthefailedoperation,valueisthefailurevalueandrolled_back_changesisamapofthepreviouslysuccessfulmultioperationsthathavebeenrolledbackduetothefailure.
Inotherwords,Ecto.Multitakescareofalltheflowcontrolboilerplatewhiledecouplingthetransactiondefinitionfromitsexecution,allowingustocomposeoperationstrivially.
TestingAnotherupsideofusingEcto.Multiisthatwecanlatertraversealloperationsstoredinthemultiandusethisinformationtowritetests.Forexample,wecouldtestthemultireturnedbytransfer_money/3asfollows:
test"transfersfrommarytojohn"do
multi=transfer_money(mary,john,10)
assert[{:mary,{:update,mary_changeset,_}},
{:john,{:update,john_changeset,_}}]=Ecto.Multi.to_list(multi)
assertmary_changeset.changes.balance==mary.balance-10
assertjohn_changeset.changes.balance==mary.balance-10
end
DependentvaluesBesidesoperationssuchasinsert,updateanddelete,Ecto.Multialsoprovidesfunctionsforhandlingmorecomplexscenarios.Forexample,prependandappendcanbeusedtomergemultistogether.Andmoregenerally,theEcto.Multi.run/3canbeusedtodefineanyoperationthatdependsontheresultsofapreviousmultioperation.
10.ComposabletransactionswithEcto.Multi
58
Let'sstudyamorepracticalexamplebyrevisitingtheproblemdefinedinthepreviouschapter.Backthen,wewantedtomodifyapostwhilepossiblygivingitalistoftagsasastringseparatedbycommas.Attheendofthechapter,webuiltasolutionthatwouldinsertanymissingtagandthenfetchallofthemusingonlytwoqueries:
defmoduleMyApp.Postdo
useEcto.Schema
#Schemaisthesame
schema"posts"do
add:title
add:body
many_to_many:tags,MyApp.Tag,join_through:"posts_tags"
timestamps()
end
#Changesetisthesame
defchangeset(struct,params\\%{})do
struct
|>Ecto.Changeset.cast(struct,[:title,:body])
|>Ecto.Changeset.put_assoc(:tags,parse_tags(params))
end
#Parsetagshasslightlychanged
defpparse_tags(params)do
(params["tags"]||"")
|>String.split(",")
|>Enum.map(&String.trim/1)
|>Enum.reject(&&1=="")
|>insert_and_get_all()
end
defpinsert_and_get_all([])do
[]
end
defpinsert_and_get_all(names)do
maps=Enum.map(names,&%{name:&1})
Repo.insert_allMyApp.Tag,names,on_conflict::nothing
Repo.allfromtinMyApp.Tag,where:t.namein^names
end
end
Whileinsert_and_get_all/1isidempotent,allowingustorunitmultipletimesandgetthesameresultback,itdoesnotruninsideatransaction,soanyfailurewhileattemptingtomodifytheparentpoststructwouldend-upcreatingtagsthathasnopostsassociatedtothem.
Let'sfixtheproblemabovebyintroducingusingEcto.Multi.Let'sstartbysplittingthelogicintobothPostandTagmodulesandkeepingitfreefromside-effects:
10.ComposabletransactionswithEcto.Multi
59
defmoduleMyApp.Postdo
useEcto.Schema
schema"posts"do
add:title
add:body
many_to_many:tags,MyApp.Tag,join_through:"posts_tags"
timestamps()
end
defchangeset(struct,tags,params)do
struct
|>Ecto.Changeset.cast(struct,[:title,:body])
|>Ecto.Changeset.put_assoc(:tags,tags)
end
end
defmoduleMyApp.Tagdo
useEcto.Schema
schema"tags"do
add:name
timestamps()
end
defparse(tags)do
(tags||"")
|>String.split(",")
|>Enum.map(&String.trim/1)
|>Enum.reject(&&1=="")
end
end
Now,wheneverweneedtointroduceapostwithtags,wecancreateamultithatwrapsalloperationsandtherepositoryaccess:
10.ComposabletransactionswithEcto.Multi
60
definsert_or_update_post_with_tags(post,params)do
Ecto.Multi.new
|>Ecto.Multi.run(:tags,&insert_and_get_all_tags(&1,params))
|>Ecto.Multi.run(:post,&insert_or_update_post(&1,post,params)
|>Repo.transaction()
end
defpinsert_and_get_all_tags(_changes,params)do
caseMyApp.Tag.parse(params["tags"])do
[]->
[]
tags->
maps=Enum.map(names,&%{name:&1})
Repo.insert_all(MyApp.Tag,names,on_conflict::nothing)
Repo.all(fromtinMyApp.Tag,where:t.namein^names)
end
end
defpinsert_or_update_post(%{tags:tags},post,params)do
Repo.insert_or_updateMyApp.Post.changeset(post,tags,params)
end
IntheexampleabovewehaveusedEcto.Multi.run/3twice,albeitfortwodifferentreasons.
1. InEcto.Multi.run(:tags,...),weusedrun/3becauseweneedtoperformbothinsert_allandalloperations,andwhilethemultiexposesEcto.Multi.insert_all/4,itdoesnotyetexposeaEcto.Multi.all/3.WheneverweneedtoperformarepositoryoperationthatisnotsupposedbyEcto.Multi,wecanalwaysfallbacktorun/3
2. InEcto.Multi.run(:post,...),weusedrun/3becauseweneedtoaccessavalueofapreviousmultioperation.Thefirstargumentofthefunctiongiventorun/3isamapwiththeresultsoftheoperationsperformedsofar.Tograbthetagsreturnedinthepreviousstep,wesimplypatternmatchon%{tags:tags}oninsert_or_update_post
Whilerun/3isveryhandywhenitneedtogobeyondthefunctionalitiesprovidednativelybyEcto.Multi,ithasthedownsidethatoperationsdefinedwithEcto.Multi.run/3areopaque,andthereforetheycannotbetestedusingEcto.Multi.to_list/1aswedidintheprevioussection.Still,Ecto.Multiallowsustogreatlyreducecontrolflowlogicandboilerplatewhenworkingwithtransactions.
10.ComposabletransactionswithEcto.Multi
61
ConcurrenttestswiththeSQLSandboxOurlastchapterisaboutoneofthemostimportantfeaturesinEcto2.0:theconcurrentSQLsandbox.GivenElixir'scapabilityofusingallofthemachineresourcesavailable,theabilitytorunteststhattalktothedatabaseconcurrentlygivesdevelopersaloweffortopportunitytospeeduptheirtestsuiteby2x,4x,8xormoretimes,dependingonthenumberofcoresavailable.
WheneveryoustartanEctorepositoryinyoursupervisiontree,suchassupervisor(MyApp.Repo,[]),Ectostartsasupervisorwithaconnectionpool.Theconnectionpoolholdsmultipleopenconnectionstothedatabase.Wheneveryouwanttoperformadatabaseoperation,forexampleinawebrequest,Ectoautomaticallygetsaconnectionfromthepool,performstheoperationyourequested,andthenputstheconnectionbackinthepool.
Thismeansthat,whenwritingtestsusingEcto'sdefaultconnectionpool(andnottheSQLsandbox),eachtimeyourunaquery,youwilllikelygetadifferentconnectionfromthepool.Thisisnotgoodfortestssincewewantalloperationsinthesametesttousethesameconnection.
Notonlythat,wealsowantdataisolationbetweenthetests.IfIintroducearecordtothe"users"tableintestA,testBshouldnotseethoseentrieswhenqueryingthesame"users"table.
Ecto1.0solvedthefirstproblembysimplyforcingteststohaveonlyoneconnectioninthedatabasepool.Thesecondproblem,regardingdataisolation,wassolvedbynotallowingteststorunconcurrently!Whileitworked,wewereunabletoleverageconcurrency.
ExplicitcheckoutsEcto2.0solvedtheproblemsabovedifferently.Themainideaisthatwewillallowthepooltohavemultipleconnectionsbut,insteadoftheconnectionbeingcheckedoutimplicitlyeverytimewerunaquery,nowtheconnectionmustbeexplicitlycheckedoutbyeverytest.Thiswayweguaranteethateverytimeaconnectionisusedinatest,itisalwaysthesameconnection.
Onceaconnectionisexplicitlycheckedout,thetestnowownsthatparticularconnectionuntilthetestisover.
11.ConcurrenttestswiththeSQLSandbox
62
Let'sstartbysettingupourdatabasetotheuseEcto.Adapters.SQL.Sandboxpool.Youcansetthoseoptionsinyourconfig/config.exs(orpreferablyconfig/test.exs):
config:my_app,Repo,
pool:Ecto.Adapters.SQL.Sandbox
Bydefaultthesandboxpoolstartsin:automaticmode,whichisexactlyhowEctoworkswithouttheSQLsandboxpool.Inotherwords,theSQLsandboxisinitiallydisabled.Thisallowsustosetupthedatabase,forexamplebyrunningmigrationsorinyourtest/test_helper.exs,asusual.
Beforeourtestsstart,weneedtoconvertthepoolto:manualmode,whereeachconnectionmustbeexplicitlycheckedout.Wedosobycallingthemode/2function,typicallyattheendofthetest/test_helper.exsfile:
#Attheendofyourtest_helper.exs
#Setthepoolmodetomanualforexplicitcheckouts
Ecto.Adapters.SQL.Sandbox.mode(MyApp.Repo,:manual)
Ifyousimplyaddthelineaboveandyoudonotchangeyourteststoexplicitlycheckaconnectionoutfromthepool,allofyourtestswillnowfail.Tosolvethis,youcouldexplicitlycheckouttheconnectiononeachtestbut,toavoidrepetition,let'sdefineaExUnit.CaseTemplatethatautomaticallydoessoinsetup:
defmoduleMyApp.RepoCasedo
useExUnit.CaseTemplate
setupdo
#Explicitlygetaconnectionbeforeeachtest
:ok=Ecto.Adapters.SQL.Sandbox.checkout(MyApp.Repo)
end
end
Nowinyourtests,insteadofuseExUnit.Case,youmaywriteuseMyApp.RepoCase,async:true.Byfollowingthestepsabove,wearenowabletohavemultipletestsrunningconcurrently,eachowningaspecificdatabasetransaction.
However,youmaywonder,howdoesEctoguaranteesthatthedatageneratedinonetestdoesnotaffecttheothertests?
Transactions
11.ConcurrenttestswiththeSQLSandbox
63
ThesecondmaininsightbesidesexplicitlycheckoutsintheSQLSandboxistheideaofrunningeachexplicitlycheckedoutconnectioninsideatransactions.EverytimeyourunEcto.Adapters.SQL.Sandbox.checkout(MyApp.Repo)inatest,itdoesnotonlycheckoutaconnectionbutitalsoguaranteesthatconnectionhasopenedatransactiontothedatabase.Thisway,anyinsert,updateordeleteyouperforminyourtestswillbevisibleonlytothattest.
Furthermore,attheendofeverytest,weautomaticallyrollbackthetransaction,effectivelyrevertingallofthedatabasechangesyouhaveperformedinyourtests.Thisguaranteesatestwon'taffecttestsrunningconcurrentlynorteststhatmayrunsubsequently.
Whiletheapproachofusingmultipleconnectionswithtransactionsworksinmanycases,italsoimposessomelimitationsrelatedtohowthedatabaseenginemanagestransactionsandperformsconcurrencycontrol.Forexample,whilebothPostgreSQLandMySQLsupportSQLSandbox,onlyPostgreSQLsupportsconcurrenttestswhilerunningtheSQLSandbox.Therefore,donotuseasync:truewithMySQLasyoumayrunintodeadlocks.
ThereisalsoachanceofrunningintodeadlockswhenrunningtestswithPostgreSQLwhenitcomestosharedresources,suchasdatabaseindexes.ButthosecasesarewelldocumentedintheEcto.Adapters.SQL.Sandbox,undertheFAQsection.
OwnershipWheneveratestexplicitlychecksoutaconnectionfromtheSQLSandboxpool,wesaythetestprocessownstheconnection.Alsorememberthatifatest,oranyotherprocess,doesnotexplicitlycheckaconnectionoutofthepool,thattestwillerrorwithanerrormessagesayingithasnodatabaseconnection.
Let'sseeanexample:
useMyApp.RepoCase,async:true
test"createtwoposts,onesync,anotherasync"do
task=Task.async(fn->
Repo.insert!(%Post{title:"async"})
end)
assert%Post{}=Repo.insert!(%Post{title:"sync"})
assert%Post{}=Task.await(task)
end
Thetestabovewillfailwithanerrorsimilarto:
**(RuntimeError)cannotfindownershipprocessfor#PID<0.35.0>
11.ConcurrenttestswiththeSQLSandbox
64
OncewespawnaTask,thereisnoconnectionassignedtothetaskprocess,causingittofail.
Whilemosttimeswewantdifferentprocessestohavetheirowndatabaseconnection,sometimesatestmayneedtointeractwithmultipleprocesses,allusingthesameconnectionsotheyallbelongtothesametransaction.
Thesandboxmoduleprovidestwowaysofdoingso,viaallowancesorbyrunninginsharedmode.
Allowances
Ifaprocessexplicitlyownsaconnection,thatprocessmayalsoallowotherprocessestousethatconnection.Effectivelyallowingmultipleprocessestocollaborateoverthesameconnectionatthesametime.Let'sgiveitatry:
test"createtwoposts,onesync,anotherasync"do
parent=self()
task=Task.async(fn->
Ecto.Adapters.SQL.Sandbox.allow(Repo,parent,self())
Repo.insert!(%Post{title:"async"})
end)
assert%Post{}=Repo.insert!(%Post{title:"sync"})
assert%Post{}=Task.await(task)
end
Andthat'sit!Bycallingallow/3,weareexplicitlyassigningtheparent'sconnection(i.e.thetestprocess'connection)tothetask.
Becauseallowancesuseanexplicitmechanism,theiradvantageisthatyoucanstillrunyourtestsinasyncmode.Thedownsideisthatyouneedtoexplicitlycontrolandalloweverysingleprocess,whichisnotalwayspossible.Insuchcases,youmayresorttosharedmode.
Sharedmode
Sharedmodeallowsaprocesstoshareitsconnectionwithanyotherprocessautomatically,withoutrelyingonexplicitallowances.
Let'schangetheexampleabovetousesharedmode:
11.ConcurrenttestswiththeSQLSandbox
65
test"createtwoposts,onesync,anotherasync"do
#Settingthesharedmodemustbedoneonlyaftercheckout
Ecto.Adapters.SQL.Sandbox.mode(Repo,{:shared,self()})
task=Task.async(fn->
Repo.insert!(%Post{title:"async"})
end)
assert%Post{}=Repo.insert!(%Post{title:"sync"})
assert%Post{}=Task.await(task)
end
Bycallingmode({:shared,self()}),anyprocessthatneedstotalktothedatabasewillnowusethesameconnectionastheonecheckedoutbythetestprocess.
Theadvantageofsharedmodeisthatbycallingasinglefunction,youwillensureallupcomingprocessesandoperationswillusethatsharedconnection,withoutaneedtoexplicitlyallowthem.Thedownsideisthattestscannolongerrunconcurrentlyinsharedmode.
SummingupInthischapterwehavelearnedaboutthepowerfulconcurrentSQLsandboxandhowitleveragestransactionsandanownershipmechanismwithexplicitcheckoutsthatallowsteststorunconcurrentlyevenwhentheyneedtocommunicatetothedatabase.
Wehavealsodiscussedtwomechanismforsharingownerships:
Usingallowances-requiresexplicitallowancesviaallow/3.Testsmayrunconcurrently.
Usingsharedmode-doesnotrequireexplicitallowances.Testscannotrunconcurrently.
WhilethroughoutthebookwecoveredhowEctoisacollectionoftoolsforworkingonyourdomain,thelastchaptersalsoshowedEctoprovidestoolstobetterinteracttothedatabase,suchasEcto.MultiwhichleveragesthefunctionalpropertiesbehindElixir,aswellastheSQLSandboxwhichexploitstheconcurrencypowerbehindtheErlangVM.
Wehopeyouhavelearnedalotthroughoutthisjourneyandthatyouarereadytowriteclean,performantandmaintainableapplications.
11.ConcurrenttestswiththeSQLSandbox
66
11.ConcurrenttestswiththeSQLSandbox
67
CONTACTUS
11.ConcurrenttestswiththeSQLSandbox
68