Big Data Analytics in the Smart Grid 1 2 3White Paper #1 – Draft 4 5Topic: Big Data Analytics, Machine Learning and Artificial Intelligence in the 6Smart Grid: Introduction, Benefits, Challenges and Issues 7 8Authored by: IEEE Smart Grid Big Data Analytics, Machine Learning and 9Artificial Intelligence in the Smart Grid Working Group 10

1112

13 14

i

1

CONTRIBUTORS 2

Working Group on Big Data Analytics, Machine Learning and 3

Artificial Intelligence in the Smart Grid 4

Chair

LauraL.Pullum IEEEComputerSociety

Contributors

AnishJindal IEEECommunicationsSociety

MehdiRoopaei IEEE

AbhishekDiggewadi IEEEPowerandEnergySociety

MerlindaAndoni IEEE(Student)

AhmedZobaa

AftabAlam

AbedalsalamBani-Ahmed

YoungNgo

ShashankVyas

RajeshKumar

ValentinRobu

IEEEPowerandEnergySocietyIEEEPowerandEnergySocietyIEEEPowerandEnergySocietyIEEEPowerandEnergySocietyIEEEPowerandEnergySocietyIEEEIndustryApplicationsSociety

DavidFlynn

ii

Staff

PhyllisCaputo IEEESmartGrid

AngeliqueRajskiParashis IEEESmartGrid

1

ACKNOWLEDGEMENT 2

IEEE Smart Grid Community brings together IEEE’s broad array of technical societies 3 and organizations through collaboration to encourage the successful rollout of 4 technologically advanced, environment-friendly and secure smart-grid networks around 5 the world. As the professional community and leading provider of globally recognized 6 Smart Grid information, IEEE Smart Grid Community is intended to organize, 7 coordinate, leverage and build upon the strength of various entities within IEEE with 8 Smart Grid expertise and interest. Additional information on IEEE Smart Grid can be 9 found at http://smartgrid.ieee.org. 10 11

12

13

14

15

16

iii

1

iv

TABLE OF CONTENTS 1

ABSTRACT............................................................................................................................................12

1.IEEESMARTGRIDBIGDATAANALYTICS,MACHINELEARNINGANDARTIFICIALINTELLIGENCE3 WORKINGGROUPWHITEPAPERSERIES...........................................................................................24

2.INTRODUCTION..............................................................................................................................35

3.CONCEPTUALDEFINITIONSofBIGDATAANALYTICS,MACHINELEARNING,ANDARTIFICIAL6 INTELLIGENCE.....................................................................................................................................77 3.1 TypicalApplicationsofBigDataAnalytics,MachineLearningandArtificialIntelligenceinthe8 SmartGrid....................................................................................................................................................99

3.1.1 BigDataAnalyticsApplications..................................................................................................910 3.1.2 MachineLearningApplications.................................................................................................1111 3.1.3Multi-AgentSystemApplications..................................................................................................1112

3.2TheNeedforDataAnalyticsintheSmartGrid...................................................................................1113 3.2.1 CloudComputing�.....................................................................................................................1214 3.2.2 EdgeComputing........................................................................................................................1315

3.3 PotentialImpactofusingDataAnalyticsintheSmartGrid..........................................................1516 3.3.1 DataAcquisitionFramework.....................................................................................................1517 3.3.2 ExtractingValuebyusingDataAnalytics..................................................................................1718

3.4 SecurityintheSmartGrid.............................................................................................................1819 3.4.1 Cyber-SecurityDetection...........................................................................................................1820 3.4.2 Cyber-PhysicalTheft..................................................................................................................1921 3.4.3 InternetofThingsandtheSmartGrid......................................................................................2022

4. OverviewofBenefits,ChallengesandIssuesofusingDataAnalyticsintheSmartGrid........2223 4.1 CurrentandExpectedUse.............................................................................................................2224 4.2 BenefitsandImpacts.....................................................................................................................2325 4.3 ExpectedBarriers/Challenges/Issues............................................................................................2626

5. Conclusion................................................................................................................................2927

References........................................................................................................................................3128

Acronyms..........................................................................................................................................3529 30

31

v

TABLE OF FIGURES 1

2 1 Interactionofrolesinsmartgriddomains 43 2 Venndiagramillustratingtheinterconnectednessofstatistics,dataanalytics,4

machinelearningandartificialintelligence 85 3 Statistics,Analytics,MachineLearningandArtificialIntelligenceincontext6

oftypeofanalysisconducted 97 4 Thefeaturesof5VsBigDatamodel 108 5 Computingplatformsa)Cloudandb)Edge 149 6 FrameworkfordataacquisitionandanalysisintheSmartGrid 1610 7 ExampleofextractingvalueusingBDA 1711 8 MachineLearningandIoTIntegration:UtilityManagementand12

ControlinSmartGrid 2013 9 Theevolutionofgridanalyticsandfuturesmartgridinbigdata 3014

applications:diversedatasourcescreatehighvolumesofdataand15 createmorevalue16

17 18

TABLE OF TABLES 19

1 Smartgriddatacompliancewiththe5VsBigDatamodel 1020 2 FrameworkfordataacquisitionandanalysisintheSmartGrid 1621 3 Othersourcesofdata 1722 4 Networkmanagementissues 2423 5 NetworkmanagementissuesandbenefitsfromBDA 2624

25

1

ABSTRACT1

Theconceptofsmartgridincorporatesanetworkofgeneration,transmissionanddistribution2 componentsthatundertakepowerdeliveryfrombulkgenerationpowerplantsanddistributed3 generationtovarioustypesofloads.Thecomponentsaregovernedandmanagedbyintelligent4 devices fromgeneration to consumption, andcanbeoptimizedbasedonenvironmental and5 economicconstraints.Asmartgridallowsutilitiestoengageconsumersinpowergenerationat6 the residential and industrial level, and may implement a bidirectional power exchange. To7 enablebeing“smart”,ahugeamountofdataisexchangedbetweengridcomponentsandthe8 enterprise systems that manage these components. Based on the application, information9 exchangedenableseconomicallyoptimizedbidirectionalpower flowbetweenautilityand its10 customers. Data exchange is essential for controlling, monitoring and coordination between11 smartequipmentinasmartgridsubsystem.Foroptimalperformance,bigdataanalyticsarea12 necessity,andlocalautonomouscontrolisachievedwhenartificialintelligenceisappliedusing13 machinelearningtechniques.Thispaperreviewstheapplicationsofbigdataanalytics,machine14 learningandartificialintelligenceinthesmartgrid.Benefits,challenges,impactsandproblems15 of employing these techniques are presented. Some big data analytics approaches for16 computingandtransmittingdataaredetailed.17

18

Keywords: Smart Grid, Big Data Analytics, Machine Learning, Artificial intelligence, Cloud19 Computing,EdgeComputing,InternetofThings,DataAcquisitionFramework,Cyber-Security20

21

22

2

1.IEEESMARTGRIDBIGDATAANALYTICS,MACHINELEARNINGAND1

ARTIFICIALINTELLIGENCEWORKINGGROUPWHITEPAPERSERIES2

ThiswhitepaperisthefirstinaseriesofwhitepapersdevelopedbytheIEEESmartGridBig3 DataAnalytics,MachineLearningandArtificialIntelligence(BDA/ML/AI)workinggroup.The4 intentoftheseriesistoprovideaconciseviewintothecurrentstatusof,benefitsof,challenges5 to,bestpracticesinandstandardsforBDA/ML/AIinthesmartgrid.6 7 TheIEEESmartGridBDA/ML/AIWhitePaperSerieswillcomprisethefollowingwhitepapers:8 9 1. IntroductiontoBDA/ML/AI,Benefits,ChallengesandIssues10 2. BestPracticesinBigDataAnalyticsfortheSmartGrid11 3. BigDataAnalyticsintheSmartGrid:RecommendedStandards,ExistingFrameworksand12

FutureNeeds13 4. PotentialApplicationsandImprovements/SolutionstoIssues:Asub-seriesofapplication-14

andsolution-specificwhitepapersorganizedbyIEEESmartGriddomainandsub-domain15 categorization.Theintentistohavethissubseriesofsmartgridanalyticswhitepapers16 coverthescopeofthesedomainsandsubdomains.17

18 19

3

2.INTRODUCTION1 2 The smart grid refers to an advanced communication and information infrastructure that3 enables optimization in energy production, transmission, distribution and storage. Other4 benefits involve systemmanagementautomation,educatedplanning, lower costs andeffort,5 andelectricitysystemreliabilityimprovement[1].Thecharacteristicsofsmartgridsinvolvethe6 wholespectrumofthepowersystem,fromgeneratorsandenergysupplierstoend-consumers7 [2]. Smart grids include the ability to enable active customer participation, and facilitate8 accommodationofpowergenerationandstorageoptions.Aperspectiveviewofthesmartgrid9 showsoneentityconsistingofmultipledomains.Thesedomainscanbeviewedasachainof10 domains for power service, starting from the generation and ending with the customer.11 However,thesedomainsarecoupledwiththehelpoffunctionalsupportsystemsthatinvolve12 many aspects of data management and communications, insuring system resiliency and13 efficiencyandsubsequentlyeconomicandenvironmentalprojections.Thedomaindefinitions14 were adapted by IEEE Smart Grid based on National Institute of Standards and Technology15 (NIST)definitions [4].Aconceptualmodelof thesmartgriddomainsand their interactions is16 showninFig.1.17 18 In addition to integration of distributed energy resources (DER), the key drivers for the19 developmentofthesmartgridarerecenttechnologybreakthroughsinenergystorage,electric20 vehicles(EV)andoperationandefficiencyimprovementsrequiredtoensurenetworkresilience21 andsecurityofsupply.Futureenergysystemsshallincludethelegacypowerequipmentwithin22 the grid infrastructure, with estimates that the US electricity grid will require $2 trillion in23 networkupgradesby2030[5].AccordingtotheEuropeanCommission,thetransitiontowardsa24 more sustainable and secureenergy systemwould require an investmentof €200billionper25 yearintheEUforgeneration,networksandenergyefficiencydevelopments[3].26

4

1

2 3 Figure1:Interactionofrolesinsmartgriddomains[12]4

5 6 Thebidirectionalflowofelectricityandinformationisanessentialfieldinthesmartgrid.With7 thegrowingelectricitysupplyfromsmaller-scale,decentralizedgenerators,i.e.,windfarmsand8 residential rooftop photovoltaics (PV) panels, andmicrogrids, advanced sensor andmetering9 technologieswith integratedsecurityprotocolsallowforenvisionedadvancedfeaturesofthe10 smartgrid,suchasdemandresponse,autonomouscontrol,self-healingandself-configuration11

5

[6].Essentially,thesmartgridprovidessignificantimprovementstotraditionalpowersystems1 that include six essential buildingblocks, namely, network, user, hardware, software, servers2 anddata[7].3 4 Thesecharacteristicsmakeitchallengingfortraditionalanalysis,butidealfortheapplicationof5 artificialintelligence,machinelearningtechniquesandbigdataanalytics.Inthispaper,wewill6 usethetermBigDataAnalytics(BDA)torefertothecollectivedataanalytics,machinelearning7 and AI. The objective of BDA is to investigate the very large volumes of data produced by8 variouscomponents inthesmartgrid,andtransformthedata intomeaningful inputssuchas9 patterns of operation, alarm trends, fault detection, and control commands. For example,10 advanced machine learning applications for distribution transformers analyze the data11 aggregatedinreal-timeforeachtransformer.Theoutcomeoftheselearningapplicationsmay12 identifysomeoperatingtrendsleadingtofailurepatternsofthesedevicesandhelpanticipate13 future failures, and consequently, provide timely and accurate insights for predictive14 maintenance.15 16 Researcheffortsinsmartgriddeploymentshavefocusedonadvancedmeteringinfrastructure17 (AMI), such as smart meters, communication, information, control and energymanagement18 systemsforutilitiesandconsumer-basedequipment(e.g.,smarthomeenergycontrollersand19 building monitoring systems). Moreover, other application areas include the integration of20 automation,controlandreal-timemonitoringofadvancedsensorsandmonitoringequipment.21 Thiscanbeaccomplishedwithfielddevices,suchasphasormeasurementunitsandintelligent22 electronicdevices(IED),atthetransmissionlevelandautomatedfeederswitches,andnetwork23 protectionrelays,voltageregulators,andcapacitorcontrollersat thedistribution level.These24 actions aim to enhance power system performance and diagnostics that will lead to cost25 reduction.26 27 Asignificantportionof the smartdevicesbeingdeployed is related to themassive rolloutof28 smartmeterscurrentlytakingplaceinmanycountries.Thenumberofsmartmeterreadingsfor29 a largeutility company is expected to rise from24million a year to 220millionper day [7].30 Approximately22GBofsmartmeterdataisbeinggeneratedby2millioncustomersperday[8].31 Assumingthatanapplicationrequiresdatacollection in15minute intervals,1milliondevices32 wouldresultin35.04billiondataentrieswithatotalvolumeof2920Tbperyear[7].Theother33 portion of smart grid devices relates to cutting-edge network devices, such as IEDs being34 installed in power systemnetworks tomonitor key network parameters and generation and35 consumptioninrealtime,controlpowerflows,exchangeinformationwitheachotherandhave36 localdecision-makingcapability.37 38

6

Traditional approaches of data analysis are inadequate to cope with the high volume and1 frequencyofdatageneratedwithinthesmartgridparadigmbyvariousdistributedsources.This2 makes optimization and smartmanagement challenging and computationally intensive. Data3 aregeneratedbymultipleheterogeneoussourcesincludingsensors,IEDs,smartmeters,smart4 appliances, distribution automation data, third-party data, asset management data and5 weatherstationdataplayinganincreasinglyimportantroleformanagingintermittentDERs[7].6 Data need to be transformed into actionable insights by applying high volume data7 management and advanced analytics (i.e., BDA) [9]. Essentially, BDA represents advanced8 analytics,suchaspredictiveanalytics,datamining,statisticalanalysis,machinelearningandAI9 techniques,whichoperateonlargedatasetshavingoneormorefeaturesofbigdata[10].10 11 Local and distributed control architectures can provide solutions that can reduce the data12 transmission load and computational resources required, as opposed to centrally controlled13 decision-making.BDAtechniquescanprovidesolutionsasthecomplexityofthepowersystem14 continuestogrow[11].15 16 17

7

3.CONCEPTUALDEFINITIONSofBIGDATAANALYTICS,MACHINE1 LEARNING,ANDARTIFICIALINTELLIGENCE2 3 Beforemovingforward,weneedtounderstandseveraltermswhichareinterrelated(as4 illustratedinFig.2)andcommonlyusedindifferentcontextswhileperforminganalyticsinthe5 smartgrid.Thesetermsare:6 7 Statistics:Itisthestudyofthecollection,analysis,interpretation,presentation,and8 organizationofdata.Further,itcanalsobedefinedasthemathematicsofestimating9 parametersofpopulationsbasedondatafromdifferentrepresentativesamplesofthose10 populations.Instatistics,thestandardprocedureforstatisticiansistostartwithanull11 hypothesis(adefaultpositionthatthereisnorelationshipbetweentwoquantities)whichis12 comparedwithanalternatehypothesis(apositionthatstatesthereisarelationshipbetween13 twoquantities).Thedecisiontorejectahypothesisistakenonthebasisofvariousstatistical14 testswhichareperformedondifferentpopulationsamples.15 16 DataAnalytics:Itisthediscoveryandcommunicationofmeaningfulpatternsindata[13].Data17 analyticsisa(sometimesautomated)processusedtodiscovernovel,valid,usefuland18 potentiallyinterestingknowledgefromlargedatasourceswhichisotherwisedifficultto19 uncover.Ifstatisticsistobeconsideredabranchofmathematics,dataanalyticsisinclined20 towardsperformingthesamefunctionalityforcomputerscience.Visualtoolsandtechniques21 arethepreferredmeansofcommunicatingtheresultsofperformingdataanalytics.22 23 MachineLearning:Itistheabilityofmachines(associatedwithcomputers)tolearn24 automaticallywithoutbeingexplicitlyprogrammed.Itdealswithrepresentationand25 generalizationofdataandcreatesarepresentationofinstancesandfunctionswhichare26 evaluatedonthesedata.Generalizationistheuniquepropertythatthemachinelearning27 systemswilltrytoyield,thatis,theabilityofthesystemstoperformwellevenonunseendata28 instances.29 30 ArtificialIntelligence:Itistheintelligenceexhibitedbymachines,asopposedtonatural31 intelligenceexhibitedbyhumansoranimals.AIencompassestechniqueswhichcanendowan32 objectoraprogramwithhuman-likeintelligence.AIalsoincludesintelligentagents,entities33 thatperceivetheirenvironmentandtakeactionsbasedonthatperception.34 35 36

8

1 2 Figure2.Venndiagramillustratingtheinterconnectednessofstatistics,dataanalytics,machine3 learningandartificialintelligence.4 5 6 Thepurposeofdifferenttypesofanalyticschangeaswemovealongthecontinuumofvalue7 (Fig.3)asfollows:8 9

● Descriptiveanalyticsaimtoprovideinformationaboutwhathappenedanditcomprises10 thefirststepthattriestoidentifyusefulinformation/dataforfurtherprocessing.It11 mightincludedatavisualization,dataminingoraggregationofreports. 12

● Diagnosticanalyticsaimtounderstandthecauseofeventsandsystembehaviorand13 triestoidentifychallengesandopportunities. 14

● Predictiveanalyticsareusedtomakeprobabilisticpredictionstoidentifytrendswiththe15 aimtodeterminewhatmighthappeninthefuture. 16

● Prescriptiveanalyticsareappliedtoidentifythebestoutcometoevents,giventhe17 system’sparameters,anddrawstrategiestodealwithsimilareventsinthefuture.It18 usestoolssuchassimulationtechniquesanddecisionsupporttoexploreoptimal19 strategiestobesttakeadvantageofafutureopportunityortomitigateafuturerisk20 [14]. 21

22 23

9

1 Figure3.Statistics,Analytics,MachineLearningandArtificialIntelligenceincontextoftypeof2 analysisconducted.(Adaptedfrom[15])3 4 5 3.1 TypicalApplicationsofBigDataAnalytics,MachineLearningandArtificial6 IntelligenceintheSmartGrid7 8 Toillustratehowbigdataanalytics,machinelearningandartificialintelligenceareusedinthe9 smartgrid,thissectionprovidesexamplesoftypicalapplicationsrelevanttothesmartgrid.10 Theseexamplesareprovidedasillustrationandare,innoway,comprehensive.Section4lists11 additionalexpectedsmartgrid-relevantapplications.Atpresent,thereisalackoftypical12 applicationsofartificialintelligenceinthesmartgridbeyondtheapplicationofML.However,13 thereareagrowingnumberofnewmodels,e.g.,deeplearningandreinforcementlearning,14 thatshowpromisetowardsenablingAIuseinthesmartgrid.15 16 3.1.1 BigDataAnalyticsApplications17 Datageneratedinthesmartgridaredifficulttohandlewithtraditionalanalysistechniquesto18 produceactionable informationwithinuseful timeframes,as requiredby thenatureof smart19 grid operations. Smart grid data can be classified as big data according to the 5Vs (Volume,20

10

Velocity, Variety, Veracity and Value) model shown in Fig. 4. Smart grid data exhibits each1 featureofthemodelasdescribedinTable1.2 3

4 5

6 Figure4:Thefeaturesof5VsBigDatamodel[7]7

8 Table1Smartgriddatacompliancewiththe5VsBigDatamodel[7]9

Feature 5VsModel SmartGridVolume Numberofrecordsandrequired

storageHighvolumesofdatafromsmartmetersandadvancedsensortechnology

Velocity Frequencyofdatageneration,transferorcollection

Ifsmartmeterdataarecollectedevery15minutes,1milliondevicesresultin35.04billiondataentriesor2920Tbperyear[7].Thefrequencydataarecollectediscrucialforreal-timemonitoringandanalysis.

Variety Diversityofsources,formats,multidimensionalfields

Existenceofstructured(e.g.,relationaldata),semi-structured(e.g.,webservicedata)andunstructureddata(e.g.,videodata)

Veracity Reliabilityandqualityofdata Reliabledataarecrucialtoensuringsafesystemoperationandstability.

Value Extractingusefulbenefitsandinsights

Applicationsderivevaluefromsmartgriddata,e.g.,predictingfuturegenerationanddemand.

10 11

11

Theresultsofbigdataanalyticscanbeusedtopredictandunderstandend-consumerbehavior,1 toimprovenetworkresilienceandfaults,toenhancesecurityandmonitoring,toenhance2 performanceandtooptimizeavailableresourcesandfutureplanning.3 4 3.1.2 MachineLearningApplications5 Machinelearningalgorithmsareparticularlyusedforclusteringdatagatheredfromthesmart6 grid domain [16]. Data are clustered to form groups with similar characteristics (natural7 classification),e.g.,groupingtogetherdatapointswithsimilaractive/reactivepowerprofilesfor8 transmission system operator (TSO) studies. Other examples include identifying low voltage9 (LV) feeders with similar load patterns for distribution system operator (DSO) studies to be10 compressed or summarized into cluster prototypes (e.g., generating representative days for11 windproductionandtheirinclusionintonetworksimulations).Further,MLcanusesmartgrid12 data to understand the underlying structure, to gain useful insights, detect anomalies and13 generatehypotheses,etc. (e.g.,detect theft andunderstanduser consumptionbehaviorata14 particularfeeder).Predictionsplayasignificantroleinpowersystemsastheyaretypicallyused15 toplan future aggregatedelectricitydemand, future system supply, estimating flexibility and16 reserveservicesrequirementsoroperationalmanagementofdistributionnetworks.17 18 3.1.3Multi-AgentSystemApplications19 AnapplicationofAI in the smart grid context ismulti-agent system (MAS)modelling.As the20 power system becomes more decentralized, market-oriented, multi-variable and complex,21 controlanddecision-makingthroughacentralizedapproachbecomeschallengingasitrequires22 significantcomputationalpowertodetermineoptimaldecisionsfortheentiresystemwithout23 significantdelays.Animprovedapproachistodividethepowersystemintomoreautonomous24 units that can make some of the decisions locally, following decentralized or distributed25 control.MASapproachescanbeusedtosolvecomplexproblems inanefficient,scalableand26 distributedway[17].PotentialapplicationsofMASinthecontextofthesmartgridincludethe27 control of microgrids, fault management and disturbance diagnosis, self-healing and power28 restoration, voltage control, frequency control, demand side management based on agent29 architecture, energy consumption optimization and scheduling and coordination of storage30 devices.31 32 3.2TheNeedforDataAnalyticsintheSmartGrid33 34 Thesmartgridgathersdatafromdiversesourcesandstoresittobeconsumablebyanalytics.35 Managingsmartgridstoprovidesmartenergyrequiresadvancedmachinelearningtechniques36 tocollectaccurateinformationinanautomatedfashion,automatedecision-makingandcontrol37 eventsinatimelymanneratboththelocalandsystem-widelevel.Importantprogresshasbeen38

12

madeforusingfielddataacquiredfromsmartdevicesmountedinsubstations,feeders,and1 numerousdatabasesandmodelsacrosstheutilityenterprise.Thereareseveralsourcesofdata2 insmartgridsonmarkets,equipment,geographyandpowersystemdatawhichcanbeusedto3 predictstates,providesituationalawareness,analyzestability,detectfaultsandprovide4 advancewarning.Therefore,analytics(comprisingBDA,MLandAI)haveasignificantroleto5 makethegridmoreintelligent,efficientandproductive.Analyticscanbeappliedtosignal,6 event,state,engineeringoperations,andcustomeranalytics,insumenablinghigh-leveland7 detailedinsightsintogridsituationalawareness.Thereareseveraltypesofanalyticsmodels,8 namelydescriptive,diagnostic,predictive,andprescriptivemodels(recallFig.3).Thesecanbe9 appliedforthesmartgrid,forinstance,descriptivemodelsdescribecustomerbehaviorsfor10 demandresponseprograms.Diagnosticmodelsareusedtounderstandspecificcustomer11 behaviorsandanalyzetheirpower-relateddecisions.Eachtypeofmodelcanprovidevaluable12 inputtocreatemodelsthatpredictfuturecustomerdecisionsandhence,powerneeds.Finally,13 prescriptivemodelscanprovidehighlevelanalyticstoinfluencesmartgridmarketing,14 engagementstrategiesanddecisionmaking. 15 16 Powersystemsarerequiredtoevolvefordynamicandflexibleinteractionwithconsumers17 participatingintheelectricitymarkets,LVcontrolautomation,distributionmanagementsystem18 (DMS)integration,microgridcontrolandbalancing,proactivefaultidentification,self-healing19 andresourceoptimization.Smartgridsystemsarebecomingincreasinglycomplexand20 interconnected,exhibitingcharacteristicsofa“systemofsystems”.21 22 Asexplainedabove,energysystemsneedtoevolvetoaccountfordistributedpowergeneration23 andthedynamicprocessesofdemandmanagement,loadcontrolandenergystorage24 management.Theenergysystemiscurrentlyexperiencingsignificantchanges,duetochanges25 inregulatoryframeworksandpoliciesthatrelatetosustainability.Thishasresultedin26 significantgrowthinthevolume,varietyandvelocityofdata,asignificantincreasein27 stakeholdernumberanddiversity,butalsoprovidingnewbusinessopportunitiesforimproved28 economicsandreliability.Theneedfordataanalyticsandnoveltechnologiesisrelevantto29 everystakeholderoftheenergysystem,includingthesystemoperator,marketoperator,30 regulator,serviceprovider,consumers,transmissionanddistributionsystemoperatorsand31 serviceprovidersandgenerators.32 33 3.2.1 CloudComputing�34 35 Cloudcomputingprovidestheabilitytoconnecttosoftwareanddataonthecloud(the36 Internet)insteadofalocalcomputingnetworkoralocalharddrive.Itisthemostrecent37 successortovirtualization,clustercomputing,utilitycomputing,andgridcomputing.Cloud38

13

computingcenterslargelyontheoutsourcingofcomputingneedsandstoragetocloud1 services.Itisasystemwhereuserscanconnecttoavastnetworkofcomputingresources,data2 andserversthatareavailableusuallyontheInternet.3 4 Virtualizationisthefoundationofcloudcomputing.Cloudcomputing,asdefinedbyForrester5 [18]isgivenasapoolofabstracted,highlyscalable,managedcomputeinfrastructurecapable6 ofhostingend-customerapplicationsandbilledbyconsumption.Thekeyfeatureofcloud7 computingisthatboththesoftwareandtheinformationarestoredonthemassivenetworkof8 cloudserversratherthanonanend-user’scomputer.Cloudcomputingisareliablechoicefor9 performinganalyticsasithasabundantresourcesaccessibleanywhereandatanytime.10 Therearemanycloudcomputingplatforms,likethoseofferedbyAmazonWebServices,AT&T’s11 SynapticHosting,andtoanextent,theHP/Yahoo/IntelCloudComputingTestbed,andthe12 IBM/GoogleCloud.Thegridcanbemadetorunmoreefficientlybyusingcloudplatformsv.13 massivelocalnetworks.14 15 Theopportunitiesand challengesofemergingand future smart grids canalsobeassistedby16 cloudcomputing.Theadvantagesofusingcloudcomputingare:� 17

18 ● Self-serviceon-demand:Theusercanindividuallyprovisioncomputingcapabilitiesas19

needed.Humaninteractionwitheachserviceproviderisnotrequired,astheserviceis20 providedautomatically.� 21

● Broadnetworkaccess:Capabilitiesareavailableoverthenetwork.Itcanbeaccessed22 throughstandardinternetaccessmechanisms. 23

● Swiftelasticity:Cloudcomputingalsosupportstheelasticnatureofmemorydevices24 andstorage.Dependingonuserdemand,itcanexpandandcontract. 25

● Measuredservice:�Cloudcomputingalsooffersmeteringinfrastructuretousers.Users26 arethusabletoprovisionandpayjustfortheirconsumedresources. 27

28 29 3.2.2 EdgeComputing30 31 Inrelevancetothesmartgrid,theInternetofThings(IoT)isaconceptthatbringslarge32 amountsofdatageneratedbyanembeddedcomponentofavarietyofdevices,sensorsand33 networkedentities.Formingasubsystemofthesmartgrid,thesedevicesmaybebundledina34 cybermannerinordertoaggregatepredefineddataatdifferentdatarates,e.g.,customer35 powerusage.Manyapplicationsaredevelopedtoutilizethedatasourcedattheedge(v.36 center)ofthenetwork,andtheyprocesstheaggregateddatalocally,mitigatingunnecessary37 datatransmissiontothecloud,andleadingtoanevolutionarytransitiontotheconceptofedge38

14

computing.Itisestimatedthatby2019;45%ofIoT-createddatawillbestored,processed,1 analyzedattheedgeofthenetwork[36].Figure5illustratestheconceptsofcloudcomputing2 andedgecomputing.Thelastresortofthedataincloudcomputingisadataconsuming3 application,whileinedgecomputing,thedataareproducedandconsumedlocally.Puttingthe4 computingclosetothedatasourcesreducesresponsetimeandenergyconsumptioncompared5 toacloudcomputingsolution.Thesmartgridbenefitsofbothcloudandedgecomputing6 dependonthespecificapplicationofsame.7 8

9 a) cloudcomputingplatform10

11

12

15

b)Edgecomputingplatform1 2 Figure5.Computingplatformsa)Cloudandb)Edge[37]3 4

5 3.3 PotentialImpactofusingDataAnalyticsintheSmartGrid6 7 AccordingtoaNISTreport,thebenefitsofmodernizationofpowergridsareashighasfive8 timestheone-timedevelopmentcost[19].InitialassessmentsbytheAmericanCouncilforan9 Energy-EfficientEconomypredicttheuseofinformationandcommunicationtechnology(ICT)10 andsmartapplianceswouldsaveabout$80billioninAmerica’sannualelectricitybill[20].This11 wouldonlybepossibleifanalytics(BDA,MLandAI)areperformedondatagatheredinthe12 smartgrid.Themajorbenefitsofperforminganalyticsincludeincreasedcustomersatisfaction,13 betterresourceutilizationandimprovedqualityofservice.However,toconductanalytics,a14 properdataacquisitionframeworkisrequiredtocollect,processandanalyzethedata.15 16 3.3.1 DataAcquisitionFramework17 18 ThegeneralframeworkforcollectionandanalysisofdatainthesmartgridisdepictedinFig.6.19 Theentitiespresentinasmartgridenvironmentincludepowergenerationunit(s),transmission20 lines, distribution stations and end-users. The end-users may be industrial, commercial or21 residentialusers.Apartfromthese,end-usersalso includeEVsandplug-inhybridEVs(PHEV),22 whichhavemadetheirway into theelectricitymarketdueto theirgrowingpopularity in the23 transportationsector.24

Dataaregatheredinthecloudinfrastructureduetoitsvariousadvantagesasdiscussedabove.25 As ICT is an integral part of the smart grid, data can be easily gathered from its associated26 entitiesasshowninFig.5.Thedataofsmarthomescanbegatheredbyplacingsmartmeters27 and other sensors in smart homes. EV/PHEV data can also be collected whenever they28 communicate (with the smart grid or cloud) to get services or to exchange information. The29 access points (AP) are placed at appropriate places which help these entities exchange30 informationdirectlyor indirectlyvia thecloud.Thedata fromEV/PHEVand the smartgrid is31 exchanged directly or through the APs connected to the cloud via the internet. The data32 collectedfromsmartmetersinsmarthomesandofficesaresenttothecloudusingAPs.InFig.33 5,distributionstationsconsistofvariouschargingstationstoprovidechargingto,ordischarging34 of,EVs.Thesechargingstationsalsosendtheirconsumptiondatatothecloudwiththehelpof35 APs. The communication technologies that are generally used for transmitting such a large36 amountofdatatothecloudarementionedinTable2.Inthistable,anobjectcanbeasmart37

16

home,PHEVoranyotherend-userentity.ThetermsObject-to-access_pointandAccess_point-1 to-cloudindicatemedium-andlong-rangecommunication,respectively.2

3 4

Figure6:FrameworkfordataacquisitionandanalysisintheSmartGrid5

Table2:Communicationtechnologiesusedfordatatransmission.6

Communicationscenario

Alternatives Protocolsused Frequencybands Datarate

Object-to-access_point

DSA

IEEE802.11af[21]

470-790MHz 1Mbps

DSRC/WAVE

IEEE802.11p[22] 5.850-5.925GHz 3-27Mbps

Access_point-to-cloud

Wi-Fi

IEEE802.11a/b/g[23]

2.4-5GHz 1-54Mbps

WiMAX

IEEE802.16[24] 1.25-20MHz 30Mbps–1Gbps

LTE/LTE-A

- 20MHz–100MHz

300Mbps–3Gbps

DSA-DynamicSpectrumAccess7 DSRC-Dedicatedshort-rangecommunication8 WAVE-WirelessAccessinVehicularEnvironment9 WiMAX-WorldwideInteroperabilityforMicrowaveAccess 10 LTE-LongTermEvolution11

17

LTE-A–LongTermEvolutionAdvanced1 2 Apart from the framework depicted in Fig. 5, data can also be collected from other sources3 involvingdifferenttechnologiesasgiveninTable3.4

5 Table3:Othersourcesofdata.6

Datatypesources

Technologyinvolved

Remarks

Advancedmeteringinfrastructure

Smartmeters Duetoincreaseinadoptionofsmartmetersinhomes,datageneratedbythesemetershasincreasedsignificantly.

Distributionautomation

Gridequipment Forreal-timemonitoringandcontrolofthegrid,thesensorsaredeployedindistributionsystemswhichtakemultiplesamplesofdataperunitoftime.

Off-griddata Thirdpartydatasets

Tostudytheeffectofutilitypoliciesonconsumerbehavior,utilitiesareintegratingdatafromthirdpartysources.

7 Once thedata are gathered, data analytics techniques canbe applied to these data and the8 resultscanbecommunicatedbacktothesmartgridorutilities(andotherentities,ifnecessary)9 fordecision-making.10 11 3.3.2 ExtractingValuebyusingDataAnalytics12 13 ThemainaimofusingBDAistoextractusefulinformation(value)fromthedata.Thisvaluecan14 beextractedfromthegathereddataafterperforminganalyticsonthedataasillustratedinFig.15 7.Utilitiesandconsumersmaymakeinformeddecisionsbasedontheresultingvalue.16

17 18

19

Figure7:ExampleofextractingvalueusingBDA[25].20

18

1 AsshowninFig.7,thefirststepistocollectdataonwhichanalyticswillbeperformed.Thedata2 canbecollectedfrommanysources,e.g.,smartmeters,smartdevices,andthird-party3 datasets.Oncethedataarecollected,thenextstepisdatapre-processing.Inthisstep,thedata4 fromvarioussources(inavarietyofformatsandpossiblycontainingmissingorerroneous5 values)areextracted.Thesedatavaluesarethencleanedtoremoveerroneousvalues.Thedata6 arethentransformedtothetargetrepository’sformat,afterwhichdataareloadedintoa7 repository.Nowdataanalyticstechniquesareappliedonthepre-processeddatatoextract8 value(i.e.,information)baseduponwhichsomeinformedactionsordecisionscanbemade.9 Theanalyticsperformedtoextractvaluecanbedividedintothreebroadcategoriesviz.10 consumeranalytics,operationalanalyticsandenterpriseanalytics[25].Consumeranalytics11 includeenergyforecasting,consumptionanalysisandtheftdetection.Operationalanalytics12 includeassetmaintenance,outagemanagement,anddistributionoptimization.Enterprise13 analyticsincludereal-timegridawarenessandvisualizationofdata.14 15 3.4 SecurityintheSmartGrid16 17 Inthesmartgridenvironment,millionsofdevicesandinfrastructuresareconnectedand18 interrelatedusingcommunicationlinksandthisexposesthegridtopossiblesecurity19 vulnerabilities.Furthermore,cloudcomputingenablesapplicationstobevirtualized;however,20 sharingtheplatformwithmillionsofuserscreatessomesecurityconcerns.Besides,thesmart21 gridmustbehighlyscalableandaccessibleinrealtimeapplicationwherelowlatencymightbe22 ahugechallenge.Therefore,therearemanycyber-securitydimensionssuchassecurity23 availability,integrity,confidentialityandaccountabilitythatincreasetheriskofcompromising24 thesmartgrid.25 26 3.4.1 Cyber-SecurityDetection27 28 Theelectricityindustryhasexperiencedthefirstknownsuccessfulcyber-attackonapowergrid29 sufferedbytheUkrainiandistributioncompanyinDecemberof2015,where30substations30 wereswitchoff,andover200thousandpeoplewereleftwithoutelectricityforadurationfrom31 1to6hours[45].Followingtheincident,theinvestigationuncoveredacomplexcyberattack32 consistingofspear-phishingemailswithmalwarecompromisingthecorporatenetworks,33 seizureofthesupervisorycontrolanddataacquisition(SCADA)systemandremotelyswitching34 offthesubstations,destructionofserverfiles,anddisablingofinformationtechnology(IT)35 infrastructurecomponents.Whilethethreatlandscapecontinuestoevolve,organizationsare36 mostconcernedaboutcybercriminalsusingphishingandunknownmalwaremethodologies.If37 malwareevolvesandre-engineersitselfconstantlyasitspreads,thetraditionalmalware38

19

detectionmethodsofsearchingforspecificcodesignatureswillnotbeeffective.Giventhe1 enormousamountofhistoricalandongoingcyber-securitydatacollectionavailableforanalysis,2 AIandmachinelearningalgorithmsarepowerfultoolsavailabletoincorporateintothecyber-3 securitystrategy.TheopportunitiesforsignificantimpactincybersecuritydefenseusingAImay4 includethefollowing:earlierdetectionofneworunknownmalwarebasedonhistorical5 baselines,improvingtheefficiencyofupdatingthemalwaresignaturedatabase,andincreasing6 ITstaffproductivitythroughreductionofnumberofincidentsthatmustbeinvestigatedand7 remediated.8 9 3.4.2 Cyber-PhysicalTheft10 11 Electricitytheftinthepowergridhasbeenamajorconcernforutilitiesandcauseshuge12 economiclosses.AccordingtotheNortheastGroup,LLC,utilitiesloseupto$90billion13 worldwideasaresultofelectricitytheft[26].Generally,electricitytheftcanbecategorizedinto14 physicalandcyber-attacks.Physicalattacksarecarriedoutbymanuallytappingtheelectrical15 supplyfromaneighborhoodordirectlyfromthefeeder.However,withadvancementsin16 communicationtechnology,onecanalsoinitializeremotetheftonthegridusingtheInternet,17 knownasacyber-attack.Themostcommonlyoccurringcyber-attackonthegridiscyber-18 tampering,throughwhichtheattackermaliciouslyaltersmeterconsumptiondata,ultimately19 leadingtoareducedelectricitybillforthebuildingassociatedwiththemeter.20 21 Inparttoimprovethegrid’sresilienceagainstsuchattacks,utilitycompaniesareinvesting22 billionsofdollarsonsmartgridinfrastructure.However,infrastructurechangesalonewillnot23 preventcyber-attacksonthegrid.Toaddressthisissue,advancedMLandAItechniquescanbe24 leveragedbytheutilities.Thesetechniquesarecapableoflearninguserbehaviorfrom25 historicaldataandcaneasilyidentifyanomaliesinuserbehavior[27].Forexample,asudden26 reductionintheconsumer’susecanserveasanindicatorfortheft.Machinelearningand27 artificialintelligencetechniquesneeddatatoformulateprecisemodelsfordetectingcyber-28 attacks.Thisdatacanbegatheredfromvarioussensorsdeployedacrosstheelectricitynetwork29 andcanbeprocessedinreal-timeusingcloudservicesandedgecomputing.So,deploymentof30 machinelearningmodelstoanalyzethisdatacanhelptogenerateusefulinformationwhich31 canbeusedtoidentifyandthwartcyber-attacks.32 33 Forexample,theauthorsin[27]usedthedatagatheredfromsensorsdeployedontransmission34 lines,thedistributionstationandtransformerlevelstocheckpowerlinesthatareproneto35 physicaltapping.Theauthorsleveragedconsumers’smartmeterdataandappliedML36 techniquesonthosedatatoidentifytheconsumerswhotamperedwiththesmartmeters.37 38

20

3.4.3 InternetofThingsandtheSmartGrid1 2 In 2016, the number of connected devices used worldwide was reported as 6.4 billion by3 Gartner[28]anditisexpectedtoreach20.8billionby2020.Thisreportalsonotesthatabout4 5.5 million new devices were connected each day in 2016 alone. The smart grid, as a big5 consumerofautonomousconnecteddevices,notonlyutilizesmillionsofIoTdevices,butalso6 analyzes very large volumes of created data to make better decisions about smart grid7 networks.Ninety-onemillionsmartmetersareexpectedby2020,alongwith36millionsmart8 interacting thermostats, and 183 million IoT residential devices. The predicted number of9 annual data points for 15-minute intervals will approach 11 trillion among these categories10 alone.Thisstreamofdatawillempowerdeeperanalysisofthegridatanincreasinglygranular11 leveltoactandreactinreal-time,therebyimprovingtheefficiencyofenergyuse[29].Bigdata12 analytics and IoT are integral parts of the smart grid. Therefore, there is a critical need for13 integrationofmachine learningwith IoT sensorsanddevicesatdifferent smartgrid levels to14 analyze the whole ecosystem to optimize the cost, balance the energy resources and15 consequentlyformanintelligentsmartgridasshowninFigure8.16 17

18 Figure8.MachineLearningandIoTIntegration:UtilityManagementandControlinSmartGrid19

20 ThesmartgridisconsideredoneofthelargestbeneficiariesoftheIoT.IoTtechnologycan21 supportsmartgridsbyprovidinghighpenetrationofinformationsourcessuchaspower22 production,storage,transmission,distributionandconsumption,andenhancingconnectivity,23 automationandmonitoringofeachdevice.Themodernandintelligentsmartgridwillnotbe24

21

costeffectivewithoutIoTtechnology.Smartgridshavealreadyattainedextensiveadoptionin1 sensing,transmissionandprocessingofinformation,andnowIoTtechnologyplaysan2 importantroleingridconfigurationandconnectinglegacyequipment.Theapplicationofthe3 IoTinsmartgridsmayfallintooneormoreofthefollowingcategories:(i)IoTisappliedfor4 employingdifferentIoTsmartdevicesformonitoringequipmentstatus,(ii)IoTisappliedfor5 collectinginformationfromequipmentwiththesupportofitslinkedIoTsmartsensorsand6 devicesthroughdiversecommunicationtoolsand(iii)IoTisappliedforsupervisingthesmart7 gridacrossapplicationinterfaces.Ineachofthesecategories,machinelearningisusedto8 providereal-timedataanalyticsandgenerateinformeddecisions.9 10

11

22

4. OverviewofBenefits,ChallengesandIssuesofusingData1 AnalyticsintheSmartGrid2 3 4 4.1 CurrentandExpectedUse5 6 SeveralUSstatesareactivelyexploringhowtousetoolsandtechnologiestorealizeasmarter7 grid[30].ExampleUSprojectsinclude:8 9

● DMSPlatformbytheUniversityofHawaii[31]:Theprojectshowedbothbatteryenergy10 storagesystemanddemandresponsetechnologiescanbeeffectiveinreducingpeak11 loadsontheMauiElectricCompany(MECO)systemandofindividualsubstations.The12 experiencegainedinthisprojectwillhelpMECOintegratedistributedandrenewable13 energyresources(PV,wind)withtheoperationofitscentralgeneratorsand14 transmissionsystem.15

● PerfectPowerbyIllinoisInstituteofTechnology(IIT)[46]:TheIITandpartners16 proposedtodevelopanddemonstrateasystemandsupportingtechnologiestoachieve17 “PerfectPower”atthemaincampusofIIT.Plansareforaself-healing,learningandself-18 awaresmartgridthatidentifiesandisolatesfaults,reroutespowertoaccommodate19 loadchangesandgeneration,dispatchesgenerationandreducesdemandbasedon20 pricesignals,weatherforecasts,andlossofgridpower.21

● Someutilitiesareusingpredictivetoolsutilizingphasormeasurementdatafor22 forecastingpossiblegeomagneticdisturbancesandprotectingtransformersfrom23 damage[38].24

● Apopularapplicationisassetmanagementwhereutilitiesareutilizingpredictive25 analysistodeterminewhentheirassetsrequiremaintenance[39].26

● OklahomaGasandElectriciscouplingAMIwithtime-basedratesandin-homedisplays27 toreducepeakpowerusage.Thismayenabletheutilitytodefertheconstructionofa28 170MWpeakingpowerplant[32].29

● OnJuly5,2012aseverewindstorminChattanooga,TNresultedin80,000customers30 losingpower.Thecitywasabletorestorepowertohalfoftheaffectedresidentswithin31 2secondsusingautomatedfeederswitching[32].32

● Usingsynchrophasordataforreal-timecontrol,theWesternElectricityCoordinating33 CouncildeterminedthatitcanincreasetheenergyflowalongtheCalifornia-Oregon34 Intertieby100MWormore,withanestimatedreductioninenergycostsof$35-7535 millionover40yearswithoutadditionalhigh-voltagecapitalinvestments[32].36

● Dataminingandmachinelearning-basedanalyticsonhistoricaleventsdatahasbeen37 providedinanintegratedhardware-softwareplatformforreal-timeeventdetectionby38

23

Tollgradecommunications.Thishasdemonstratedtheuseofpredictivegrid-analytics1 ondatacollectedbythecompany’sproprietarysmart-gridsensorsforpreventing2 blackoutstooccur[43].Theplatformidentifiesanomaliesonadistributionfeeder3 beforetheymagnifyandthusenablesaproactive,predictivestrategyformanagingreal-4 timeevents,similartotheconceptenvisagedin[44]forpredictivedetectionof5 unintentionalislandinginfeeders.UtilitieslikeWesternPowerDistributionhaveused6 thisplatformandhaveexperiencedimprovedreliabilityindices.7

8 SomeexamplesmartgridprojectsfromtheUKinclude:9 10

● NetworkConstraintsEarlyWarningSystemsbyScottishPowerEnergyNetworks[40]:11 Theprojectsaimtoutilizelarge-scale,distributedsmartmeterdatatomonitorand12 detecttheriskofpowersurgesandvoltageexcursionsoutsidetechnicaloperational13 limits,indifferentpartsofthedistributionnetwork(orsubnetworks).Withmillionsof14 consumers,onecannotcollectdatainrealtimefromeverysmartmeter/consumerin15 thenetwork,thereforeanimportantquestiontoansweriswhatdatagranularityis16 neededfordifferentpartsofthenetworktodetecttheseexcursionsandprovideearly17 warningsofpotentialvulnerabilities.18

● ThamesValleyVisionbyScottishandSouthernElectricityNetworks[41]:Theproject19 wasthefirstscaleddeploymentofLVsubstationmonitoringofrealtimeelectricitydata20 andtheirintegrationintoadistributionmanagementsystem.Datawereusedfor21 substationcategorizations,aggregationandforecastingoffuturenetworkloading.22

● Home-Offshore[42]:Theprojectaimstouseadvancedroboticmonitoringandsensing23 techniquesfortheremoteinspectionandassetmanagementofoffshorewindfarms24 andtheirconnections.Datacollectedwillgenerateinsightsfordiagnosticand25 prognosticschemeswhichwillallowimprovedinformationtobestreamedintomulti-26 physicsoperationalmodelsforoffshorewindfarms.27

28 4.2 BenefitsandImpacts29 30 TheuseofBDAcanprovidenumerousbenefitstoallstakeholderswithinthepowerindustry.31 Forinstance,DSOscanachieveimprovedcoordinationofthesupplyandthedemandin32 distributionnetworks,consumerscanimprovetheirenergyefficiencyandachievesignificant33 savingsintheirenergybillsbycloselymonitoringtheirenergydemand,energysuppliersor34 retailerscangaininsightsintoconsumerbehaviorandimproveoperationalefficiencyand35 energygeneratorscanincreaseprofitsbyoptimizingtheirgenerationassetsandproduction.36 TSOscanenhancetheirlong-termplanningbystudyingthebehaviorofdifferenttypesof37 consumersingroupssuchasresidential,commercialandindustrialandbyapplyingML38 algorithmsthateffectivelycapturetheimpactofdispersedgeneration,whichisofteninvisible39

24

atatransmissionlevel.1 2 Deregulatedenergysystemshaveincreasinglyreliedonserviceproviderssuchasdemand3 aggregators,storageprovidersandvirtualpowerplants.Serviceproviders’optimaloperation4 andprofitmaximizationstrategiesaresubjecttouncertaintyandneedtoaccountforthe5 strategicbehaviorofmultipleplayersparticipatingintheenergymarket.Notonlyisthe6 decision-makingprocessmulti-variable,butitneedstoaccountforinterdependenciesofthe7 decisionvariablesatdifferenttimeframes.Forexample,longtermdecisionsmightintroduce8 constraintstomediumtermgoalsandoperation.Usefultechniquestocopewiththeseissues9 canbederivedfromthefieldofgametheory,whichissuitabletostudystrategicinteractions10 betweenmultipleactorsandtoidentifymarketequilibria.Thesecombinedwithprobabilistic11 modellingandstochasticoptimizationtools,allowserviceproviderstoreducecostsand12 maximizerevenues.Agentscanidentifyandlearnbiddingstrategiesintheenergymarkets,13 optimaldispatching,balancingandoperationoftheirassets.Inaddition,serviceprovidersneed14 toadoptpredictivemaintenance,prognosticsandhealthmanagementtoadoptmarketand15 operationstrategiesthatprolongtheusefullifetimeoftheirassets.Thesametoolsand16 principlescanbeusedformicrogridcontrolinadecentralizedordistributedfashionand17 coordinationwiththecentralizedgrid.18 19 Utilitycompanies,especiallyatadistributionlevel,needtoadoptbigdatatechniquesasthe20 energysystemcontinuestoevolve.Mostutilitiesarecurrentlyrestrictedtousingdescriptive21 anddiagnosticanalyticsthataimtoanalyzehistoricaldataoreventstounderstandthereasons22 behindtheiroutcomes,e.g.,systemfaultmanagementoroutagemanagement.However,as23 energysystemsgrowtobecomemoredecentralizedandreliantonintermittentDER,utilities24 needtodeploypredictiveanalyticstoevaluatepotentialfuturescenarios,suchasevaluating25 futuregridinvestmentrequirementsandforecastingloadimpactandassetmonitoring.26 Further,prescriptiveanalyticscanleadtoactionableinsightsforplanningofgenerationand27 transmission/distributioncapacityoroptimizationofrenewableintegration.Foramore28 detailedlookatchallengesutilitiescurrentlyfaceinnetworkmanagement,seeTable4.29 30

Table4:Networkmanagementissues31 GridManagementIssue

Example Consequence

Incompletereal-timemonitoring

Lackof● Equipmentloading

information ● Statusofswitches,

transformertapchangers

● Inefficientequipmentutilization

● Difficulttoenablecustomerstoconnectdistributedgeneratorsto

25

● Systemmomentaryfaultlocation

● Statusofdistributedresources

● Customerdemand/load

gridandmaintainsystemreliability

● Nounderstandingofautomatedoperationsonfeeder

Lackofsysteminteroperability

Non-integratedsystemsfor● Customerinformation

system ● Geographicinformation

system ● Crewmanagement ● Switchorder

management ● AMI ● SCADA

● Inefficientworkprocesses ● Redundant/inaccuratedata ● Longeroutageduration ● Possiblenon-complianceof

workprocesseswithpossiblesafetyissues

LackofDiagnostics

Lackofapplicationsfor● Faultlocation ● Restorationswitching

analysis ● Voltage/Reactivepower

control ● Distributionstate

estimation

● Longeroutagedurations ● Inefficientuseofcrew

hours ● Nochancetoreduce

customerdemandthroughvoltagecontrolatpeaktimes

● Highersystemlosses ● Increasedcustomer

complaintsforvoltageoutofrange

LackofPrognostics

● Reactive-basedmaintenanceoftransmissionanddistributionnetworkassets

● SCADAneedsimprovedscalabilitytodealwithhighvolumeofdatamonitored

● Lackofenvironmentalmonitoring

● Expensivemaintenance ● Increasedcustomer

interruption ● Assumptionthatcondition

monitoringdataarenotdependentoncircumstancedata,suchastheenvironmentalconditionswhereasensorisdeployed,leadingtopotentiallyerroneousinformation

1

26

Table5describessomeoftheissuesthatcanbeimprovedsignificantlybytheuseofBDAin1 smartgrids.2 3

Table5:NetworkmanagementissuesandbenefitsfromBDA4 Category ApplicationsConnectivitymodelimprovement

● Detectincorrectdistributiontransformerconnectivity

● Correctmeterphasing ● Auto-generatesecondarycircuitmodels

Assetmanagement ● Identificationofoverloadedorhighutilizationassets(transformerreplacement,cablemonitoring)

● Identificationofunder-loadedtransformersorstrandedassets

● Identificationoftransformervoltageissues Theftandconsumerbehavior ● Improveunderstandingofnearreal-timeload

profileofdistributionfeeder ● Identificationoftheftortamperedmeters ● UsingAMIorsmartmetersdata ● Non-intrusiveloadmonitoring

Faultlocationanddiagnosis ● Identificationoffaultoccurrencewiththeuseofdatafromsensorsandmonitoringequipment,e.g.,fromsubstations,feedersorrelaydata

● Studyfaultpatternsandtypestohelpplanmitigatingactionsuchastreetrimming,equipmentmaintenance,etc.

Reliabilityanalysis ● Healthstateestimationbydata-drivenmethodsandML

5 6 4.3 ExpectedBarriers/Challenges/Issues7 8 SignificantchallengesarisewhenapplyingBDAinthesmartgrid.Thesearesummarizedbelow:9 10

● Dataoriginatefromnumeroussourcesandcomeindifferentformats[33],suchassmart11 meters,consumerdata,webdata,weatherdata,populationdata,geographic12 informationsystemdata,DMSdata,energymanagementsystem,SCADA,etc.Data13 needtobeintegratedandinteroperabilitybetweendifferentdevicesandcontrollevels14

27

mustbeensured.Newregulationandstandardizedprocessesarerequiredfordata1 collectionandgovernance[7].Thereisalackofstandardsfordatadescriptionand2 communication,essentialforinteroperability.Moreover,dataintegrationanddata3 sharingpracticesacrossinstitutionsneedtobedefinedforthebenefitofall4 stakeholders[33]. 5

● Asignificantissueisrelatedtoincompleteormissingdata.Sensingequipmentand6 smartmetersarenotinstalledeverywhereinthegridandmightbesubjecttofailuresor7 malfunctions.Inthiscase,missingdatamayneedtobeextrapolatedfromavailable8 data.TheissuebecomesextremelyimportantwhenitcomestoMLtechniquesthatrely9 ongoodqualityandavailabilityofdataforalgorithmtraining.Althoughtherehavebeen10 significantadvancesinMLapproachesthathandletheissueofmissingdatawell,more11 needstobedone.Dataintegrityneedstobeensuredbypre-processingtechniquesand12 errordetectionsystemsthatensuredataquality. 13

● UtilitycompaniesneedtoinstallnewITequipment,suchassensorequipment,14 hardwareandsoftwaretoolsanddatastoragedevices[7],presumablyatsignificant15 installationandoperationcosts.Moreover,utilitiesareuncertainregardingthe16 transitionfromacentralizedtoadecentralizedpowernetworkandtheroleofBDAin17 futureenergysystems[34]. 18

● Researchchallengesremaininthebigdataanalyticsspace.Cloudcomputing,19 distributedfilemanagementandnewdatabasesaredeveloped,suchasNoSQL20 databasesforparallelprocessing.SeveralMLlibrarieshavebeendevelopedforbatch21 processingandnoveltechniquesarebeingdevelopedforstreamprocessing.However,22 furtherimprovementisrequiredtoadheretothetimeresponserequirementsofthe23 smartgrid[34]. 24

● Datavisualizationcanbechallengingduetothesizeofthedatasetsandtheirhigh25 dimensionality.Currentdataanalyticstoolstendtohavepoorperformancefor26 visualizationoflargedatasets,havehighresponsetimesandarenoteasilyscalable27 [34].Thescaleofdatacanbehandledsomewhatbyaggregation,however,insome28 casesaggregationmayhidethespecificityneededforinsight. 29

● Acontinuouslygrowingamountofdataneedstobestored,typicallyinthecloud.This30 canleadtodataexplosionandthescalabilityissue[34].Technicalchallengessuchasthe31 networkbandwidthcapacityanddatasecurityissuesremaintobesolved.Fig.8shows32 theincreaseindatavolumeasanalyticsevolvetowardsthesmartgridmodel. 33

● AnotherissueforMLtechniquesincludesthepotentialforinadequatetrainingdata,34 whichmaydecreaseconfidenceintheresultsofsupervisedMLmodelsforpreviously35 unwitnessedsituations.Advancesinsemi-supervisedandunsupervisedMLneedto36 continuetoaddressthisissue.37

28

● Specialcareisrequiredforprivacy,confidentialityandpersonaldataprotection.1 Significantproblemstoberesolvedincludedatasecurityprotection,intellectual2 propertyprotection,personalprivacyprotection,commercialsecurity,networksecurity3 andfinancialinformation[34]. 4

● Securityofdataandsmartgridinfrastructureiscritical.Smartgridscanbevulnerableto5 cyber-attacksorcyber-physicalattacks.Smartgridsystemfunctionalityisvitalfor6 societyoverallandcontainssensitiveinformationthatneedstobeprotectedfrom7 maliciousattacksandvulnerabilities.Thisincludesallaccesspointsandprotectionof8 equipmentwhichareinherentlydistributed,butalsodatastoragesecuritythatcanbe9 obtainedbyadoptingadvancedcryptographictechniquesandverificationmechanisms10 [7]. 11

● Energyconsumptionrequiredforpotentiallyintensivedemandoncomputationalpower12 isanotherimportantissuethatrequiresacost-efficientandsustainablesolution[35]. 13

● Finally,aswitheverysignificanttransition,thereisarequirementforawell-trained14 workforce,bothcurrentandfuturehumanresources[7]. 15

● Additionalissuesincludeanalyticsverificationandvalidation,certificationand16 regulatorycompliance;andinteroperabilityofallelementsofthesmartgrid.17

18 19

29

5. Conclusion1 2 Thiswhitepaperintroducedtheconceptsandpossibleusecasesofbigdataanalytics,machine3 learningandartificial intelligence in the smartgrid.The smartgrid includesgenerationunits,4 transmissionlinesanddistributionsystems.Distributedenergyresourcesareintegratedeither5 separately or at the end-customer and then the power flows in different directions needing6 more modern devices for monitoring, transmitting and metering all power components.7 Monitoring,collectingandtransmittingallthegridparametersinvolvesalargeamountofdata.8 Meanwhile, it is infeasibletoswiftlyandcorrectlyanalyzetheseenormousdatabytraditional9 methods,whichbrings challenges to effectively enabling the smart grid. Therefore, powerful10 data analytics will be adopted to manage the massive number of data. Big data analytics,11 machine learning and artificial intelligence are approaches employed in the smart grid to12 manage the data collected from all power meters, sensors and other appliances. By the13 combinationof these techniques, it ispossible toknowandexpect loaddemand,generation14 volumeandsystemdisturbance,andthenregulatethecontrolautomaticallyandquicklysothe15 gridcanbeimprovedandavoidinstabilities.Applyinganalyticseffectivelyinthesmartgridstill16 faces numerous difficulties. Most power utilities are still uncertain of big data analytics,17 machine learning and artificial intelligence and thus this article defined terms related to18 advanced analytics approaches used in smart grid and explained the advantages, challenges19 andproblemsofutilizingtheseapproaches.Abriefreviewwasalsogivenonsecuritymatters,20 differentimpactandcommunicationtechnologiesinsmartgrid.21 22

30

1 Figure9:Theevolutionofgridanalyticsandfuturesmartgridinbigdataapplications:diversedatasourcescreatehigh2

volumesofdataandcreatemorevalue.Adaptedfrom[14]3

4

31

References1 2 [1] U.S.DepartmentofEnergy,“SmartGridsystemreport,”2009.[Online]Available:3

https://www.smartgrid.gov/files/systems_report.pdf.4

[2] EuropeanCommission,“EuropeanSmartGridsTechnologyPlatform:VisionandStrategy5 forEurope’sElectricityNetworksoftheFuture,”2006.[Online]Available:6 http://cordis.europa.eu/pub/fp7/energy/docs/smartgrids_en.pdf.7

[3] EuropeanCommission,“Aframeworkstrategyforaresilientenergyunionwitha8 forward-lookingclimatechangepolicy,”2015.[Online]Available:9 https://www.eea.europa.eu/policy-documents/com-2015-80-final10

[4] NationalInstituteofStandardsandTechnology,“NISTFrameworkandRoadmapfor11 SmartGridInteroperabilityStandards,Release3.0,”2014,[Online]Available:12 http://dx.doi.org/10.6028/NIST.SP.1108r3.13

[5] P.Bronski,J.Creyts,M.Crowdis,S.Doig,L.Guccione,J.Mandel,B.Rader,H.Touatiand14 D.Seif,“TheEconomicsofLoadDefection:HowGrid-ConnectedSolar-Plus-Battery15 SystemsWillCompetewithTraditionalElectricService–WhyitMatters,andPossible16 PathsForward,”RockyMountainInstitute,2015.[Online]Available:17 https://rmi.org/insights/reports/economics-load-defection/.18

[6] F.Khan,A.urRehman,M.Arif,M.AftabandB.Jadoon,“Asurveyofcommunication19 technologiesforsmartgridconnectivity,”IEEEInternationalConferenceonComputing,20 ElectronicandElectricalEngineering(ICECube),Quetta,2016,pp.256-261.21

[7] S.Sagiroglu,R.Terzi,Y.CanbayandI.Colak,“Bigdataissuesinsmartgridsystems,”IEEE22 InternationalConferenceonRenewableEnergyResearchandApplications(ICRERA),23 Birmingham,2016,pp.1007-1012.24

[8] A.Mahmood,M.AamirandM.I.Anis,“DesignandimplementationofAMRSmartGrid25 System,”IEEECanadaElectricPowerConference,Vancouver,BC,2008,pp.1-6.26

[9] P.Diamantoulakis,V.KapinasandG.Karagiannidis,“Bigdataanalyticsfordynamic27 energymanagementinsmartgrids,”BigDataResearch,vol.2,no.3,pp.94-101,2015.28

[10] P.Russom,“Bigdataanalytics,”TDWIBestPracticesReport,TDWIResearch,pp.1-6,29 2011.[Online]Available:http://www.teradata.com/Resources/Analyst-Reports/TDWI-30 Best-Practices-Report-Big-Data-Analytic-(1)31

[11] U.AhsanandA.Bais,“Distributedbigdatamanagementinsmartgrid,”26thWireless32 andOpticalCommunicationConference(WOCC),Newark,NJ,2017,pp.1-6.33

[12] IEEESmartGridR&DCommittee,“IEEESmartGridSurvey-StructureofEmerging34 Technologies,”October1,2017.[Online]Available:35

32

https://smartgrid.ieee.org/images/files/white-papers/white-paper-ieee-smart-grid-1 survey-structure-of-emerging-technologies.pdf.2

[13] J.Strickland,“PredictiveModelingandAnalytics,”Lulu,Inc.,ColoradoSprings,CO,2014.3

[14] T.H.Dang-Ha,R.OlssonandH.Wang,“TheRoleofBigDataonSmartGridTransition,”4 IEEEInternationalConferenceonSmartCity/SocialCom/SustainCom(SmartCity),5 Chengdu,2015,pp.33-39.6

[15] M.Davis,“Top10MomentsfromGartner’sSupplyChainExecutiveConference,”May7 28,2013.[Online]Available:http://blogs.gartner.com/matthew-davis/top-10-moments-8 from-gartners-supply-chain-executive-conference/.9

[16] A.Jain,“Dataclustering:50yearsbeyondK-means,”PatternRecognitionLetters,vol.31,10 no.8,pp.651-666,2010.11

[17] A.L.DimeasandN.D.Hatziargyriou,“AgentbasedcontrolofVirtualPowerPlants,”IEEE12 InternationalConferenceonIntelligentSystemsApplicationstoPowerSystems,Toki13 Messe,Niigata,2007,pp.1-6.14

[18] S.Tilley,“SoftwareTestingintheCloud:PerspectivesonanEmergingDiscipline,”IGI15 Global,PA,2012.16

[19] NationalInstituteofStandardsandTechnology,“StrategicR&DOpportunitiesforthe17 SmartGrid:AdvancingMeasurementScienceandStandardsforSmartGrid18 Technologies,”pp.1-36,2013,[Online]Available:19 http://www.nist.gov/smartgrid/upload/Final-Version-22-Mar-2013-Strategic-R-D-20 Opportunities-for-the-Smart-Grid.pdf.21

[20] J.A.Laitner,M.T.McDonnellandK.E.Martinez,“TheEnergyEfficiencyandProductivity22 BenefitsofSmartAppliancesandICTEnabledNetworks:AnInitialAssessment,”23 AmericanCouncilforanEnergy-EfficiencyEconomy(ACEEE),2014.[Online]Available:24 http://aceee.org/research-report/f1402.25

[21] A.B.Flores,R.E.Guerra,E.W.Knightly,P.EcclesineandS.Pandey,“IEEE802.11af:A26 StandardforTVWhiteSpaceSpectrumSharing,”IEEECommunicationsMagazine,vol.27 51,no.10,pp.92-100,2013.28

[22] IEEEStd802.11p,“IEEEStandardforInformationTechnologyPart11:WirelessLAN29 MediumAccessControl(MAC)andPhysicalLayer(PHY)specificationsAmendment6:30 WirelessAccessinVehicularEnvironments,”2010.[Online]Available:31 https://www.ietf.org/mail-archive/web/its/current/pdfqf992dHy9x.pdf.32

[23] IEEE802.11:WirelessLANs.[Online]Available:33 http://standards.ieee.org/about/get/802/802.11.html.34

33

[24] IEEE802.16:BroadbandWirelessMetropolitanAreaNetworks(MANs).[Online]1 Available:http://standards.ieee.org/about/get/802/802.16.html.2

[25] C.L.Stimmel,“BigDataAnalyticsStrategiesfortheSmartGrid,”BocaRaton,FL,USA:3 CRCPress,2014.4

[26] PRNewswire,“WorldLoses$89.3BilliontoElectricityTheftAnnually,$58.7Billionin5 EmergingMarkets,”2014.[Online]Available:http://www.prnewswire.com/news-6 releases/world-loses-893-billion-to-electricity-theft-annually-587-billion-in-emerging-7 markets-300006515.html.8

[27] A.Jindal,A.Dua,K.Kaur,M.Singh,N.KumarandS.Mishra,“DecisionTreeandSVM-9 basedDataAnalyticsforTheftDetectioninSmartGrid,”IEEETransactionsonIndustrial10 Informatics,vol.12,no.3,pp.1005-1016,2016.11

[28] Gartner,Inc.,“GartnerSays6.4BillionConnected‘Things’WillBeinUsein2016,Up3012 PercentFrom2015,”2015.[Online]Available:13 http://www.gartner.com/newsroom/id/3165317.14

[29] NavigantConsulting,Inc.,“NavigatingtheEnergyTransformation:Buildinga15 CompetitiveAdvantageforEnergyCloud2.0,”2016.[Online]Available:16 https://www.navigant.com/insights/energy/2016/navigating-the-energy-17 transformation.18

[30] LitosStrategicCommunication,“SmartGrid:AnIntroduction,”forUSDepartmentof19 Energy,2008.[Online]Available:20 https://www.smartgrid.gov/files/The_Smart_Grid_Introduction_200804.pdf.21

[31] HawaiiNaturalEnergyInstitute,“MauiSmartGridDemonstrationProject:Managing22 DistributionSystemResourcesforImprovedServiceQualityandReliability,Transmission23 CongestionRelief,andGridSupportFunctions-FinalTechnicalReport,”forUS24 DepartmentofEnergy,2014.[Online]Available:25 https://www.hnei.hawaii.edu/projects/maui-smart-grid-demonstration.26

[32] USDepartmentofEnergy,“2014:SmartGridSystemReport,AReporttoCongress”,27 2014.[Online]Available:https://www.smartgrid.gov/files/2014-Smart-Grid-System-28 Report.pdf.29

[33] F.Luo,Z.Y.Dong,J.Zhao,X.Zhang,W.KongandY.Chen,“Enablingthebigdataanalysis30 inthesmartgrid,”IEEEPower&EnergySocietyGeneralMeeting,Denver,CO,2015,pp.31 1-5.32

[34] C.S.LaiandL.L.Lai,“ApplicationofBigDatainSmartGrid,”IEEEInternational33 ConferenceonSystems,Man,andCybernetics,Kowloon,2015,pp.665-670.34

34

[35] X.Miao,“BigDataandSmartGrid,”InternationalConferenceonBigDataScienceand1 Computing,Beijing,China,2014.2

[36] D.Evans,“TheInternetofThings:HowthenextevolutionoftheInternetischanging3 everything,”CISCOWhitePaper,vol.1,pp.1-11,2011.4

[37] W.Shi,J.Cao,Q.Zhang,Y.LiandL.Xu,“EdgeComputing:VisionandChallenges,”IEEE5 InternetofThingsJournal,3(5):637-646.6

[38] PingThings,“PredictiveGrid,”accessed10/12/2017.[Online]Available:7 http://www.pingthings.io/.8

[39] IBM,“StrategicAssetManagementtoImproveElectricityNetworkPlanningand9 Maintenance,”[Online]Available:https://www-10 935.ibm.com/industries/energy/solutions/asset-workforce-management.html.11

[40] SPEnergyNetworks,“NetworkInnovationAllowanceProgressReport,”July2017,12 [Online]Available:http://www.smarternetworks.org/Project.aspx?ProjectID=2039.13

[41] Scottish&SouthernElectricityNetworks,“ThamesValleyVision,”[Online]Available:14 https://www.ssepd.co.uk/thamesvalleyvision/.15

[42] M.Barnes,etal.,“HOME-Offshore:HolisticOperationandMaintenanceforEnergyfrom16 OffshoreWindFarms,”UniversityofManchesterTechnicalReport,[Online]Available:17 http://gow.epsrc.ac.uk/NGBOViewGrant.aspx?GrantRef=EP/P009743/1.18

[43] “BringingOurElectricGridintothe21stCentury,”2014,[Online]Available:19 https://www.engerati.com/resources/bringing-our-electric-grid-21st-century.20

[44] S.Vyas,R.KumarandR.Kavasseri,“DataAnalyticsandComputationalMethodsforAnti-21 islandingofRenewableEnergyBasedDistributedGeneratorsinPowerGrids,”22 RenewableandSustainableEnergyReviews,vol.69,March2017,pp.493-502.23

[45] KimZetter,“InsidetheCunning,UnprecedentedHackofUkraine’sPowerGrid,”Wired,24 03/03/2016,[Online]Available:https://www.wired.com/2016/03/inside-cunning-25 unprecedented-hack-ukraines-power-grid/.26

[46] IllinoisInstituteofTechnology,“Public/PrivatePartnershipCreatesOpportunityto27 FundamentallyAddressLocalEnergyCrises,”[Online]Available:28 https://web.iit.edu/mediaroom/press-releases/2008/nov/20/publicprivate-partnership-29 creates-opportunity-fundamentally.30

31

35

Acronyms1 2 5V FiveVsofBigDataAnalytics(Volume,Velocity,Variety,Veracity,Value)3 AI ArtificialIntelligence4 AMI AdvancedMeteringInfrastructure5 AP AccessPoint6 BDA BigDataAnalytics7 DER DistributedEnergyResource8 DMS DistributionManagementSystem9 DSA DynamicSpectrumAccess10 DSO DistributionSystemOperator11 DSRC DedicatedShort-RangeCommunication12 EU EuropeanUnion13 EV ElectricVehicle14 GB Gigabyte15 Gbps Gigabitspersecond16 GHz Gigahertz17 ICT InformationandCommunicationTechnologies18 IED IntelligentElectronicDevice19 IEEE InstituteofElectricalandElectronicsEngineers20 IIT IllinoisInstituteofTechnology21 IoT InternetofThings22 IT InformationTechnologies23 LLC LimitedLiabilityCorporation24 LTE Long-TermEvolution25 LTE-A Long-TermEvolution-Advanced26 LV LowVoltage27 MAS Multi-AgentSystem28 Mbps Megabitspersecond29 MECO MauiElectricCompany30 MHz Megahertz31 ML MachineLearning32 MW Megawatt33 NIST NationalInstituteofStandards&Technology34 PHEV Plug-inHybridElectricVehicle35 PV Photovoltaics36 SCADA SupervisoryControlandDataAcquisition37 SQL StructuredQueryLanguage38

36

Tb Terabyte1 TN Tennessee2 TSO TransmissionSystemOperator3 UK UnitedKingdom4 US UnitedStates5 WAVE WirelessAccessinVehicularEnvironment6 WiMAX WorldwideInteroperabilityforMicrowaveAccess7 8 9