A Survey on Non-Intrusive Load Monitoring Methodies

and Techniques for Energy Disaggregation Problem

Anthony Faustine ∗1, Nerey Henry Mvungi †2, Shubi Kaijage‡1 and Kisangiri Michael §1

1Dept. of Communication Science and Engineering, NM-AIST, Tanzania2College of Information and Communication Technologies, University of Dar es Salaam

March 13, 2017

Abstract

The rapid urbanization of developing countries coupledwith explosion in construction of high rising buildingsand the high power usage in them calls for conservationand efficient energy program. Such a programme requiremonitoring of end-use appliances energy consumptionin real-time.

The worldwide recent adoption of smart-meterin smart-grid, has led to the rise of Non-IntrusiveLoad Monitoring (NILM); which enables estimation ofappliance-specific power consumption from building’saggregate power consumption reading. NILM provideshouseholds with cost-effective real-time monitoring ofend-use appliances to help them understand their con-sumption pattern and become part and parcel of energyconservation strategy.

This paper presents an up to date overviewof NILM system and its associated methods andtechniques for energy disaggregation problem. This isfollowed by the review of the state-of-the art NILMalgorithms. Furthermore, we review several perfor-mance metrics used by NILM researcher to evaluateNILM algorithms and discuss existing benchmarkingframework for direct comparison of the state of the artNILM algorithms. Finally, the paper discuss potentialNILM use-cases, presents an overview of the publicavailable dataset and highlight challenges and futureresearch directions.

∗sambaiga@gmail.com†nhmvungi@udsm.ac.tz‡shubi.kaijage@nmaist.ac.tz§kisangiri.michael@nmaist.ac.tz

1 Introduction

Residential and commercial buildings consume approxi-mately 60% of the worlds electricity 1. In U.S. A forexample 74.9% of the produced electricity is used justto operate buildings 2 while it is 56% in Africa [1]. Itis estimated that 80% more buildings will be in placeby 20503. Hence energy saving in buildings will havesignificant impact on the reduction of overall energydemand.

Effective and efficient energy saving in residentialbuildings can be achieved through real-time monitor-ing of end-use appliances consumption and provision ofreal-time actionable feedback to households that giveinsight into what appliances and when they are used,how much power they consume and why such consump-tion. Hence, households will be actively engaged anddetermine where energy is wasted and where or howto apply the most effective energy saving measuresstimulating energy saving behaviour.

Studies report that energy consumption aware-ness coupled with real-time actionable feedback tohouseholds inspire positive behavioural change and en-gage households toward sustainable energy consumption[2, 3].

Traditionally, real-time appliance-specific break-down of energy consumption is obtained by deployingsensors (smart plugs) that monitor the consumption ofeach appliance in buildings. Deploying such a sensinginfrastructure is costly, intrusive and require propri-etary communication protocols [4, 5]. Recently, large

1The United Nations Environment Programmes SustainableBuilding and Climate Initiative (UNEP-SBCI)

2United States Energy Information Administration re-port:http://www.eia.gov/todayinenergy/detail.cfm?id=14011

3The United Nations Environment Programmes SustainableBuilding and Climate Initiative (UNEP-SBCI)

1

arX

iv:1

703.

0078

5v3

[cs

.OH

] 1

0 M

ar 2

017

scale deployments of smart meters have rekindled theinterest towards developing effective non-intrusive loadmonitoring (NILM)4 techniques [6].

NILM is the computational techniques that use ag-gregate power data monitored from single point sourcesuch as smart meter to infer the end-appliances runningin the building and estimate their respective power con-sumption. It provides households with cost-effectivereal-time monitoring of end-use appliances that facili-tate energy conservation actions. Also, NILM systemcan help policy makers evaluate the effectiveness oftheir energy-efficiency policies while utility can betterforecast demand and enable manufacturers to optimizeproduct design to meet customer need [7].

The initial NILM approach to residential energydisaggregation was proposed by Hart in the 1990s [8].Recently many researchers have published several ap-proaches on energy disaggregation that improve theinitial design [9, 10]. Despite several efforts done byprevious NILM researchers, there are still several chal-lenges which need to be addressed. This work presentsan up to date overview of NILM system and its associ-ated methods and techniques for energy disaggregationproblem.

2 Energy Disaggregation Prob-lem

Energy disaggregation is a technique that estimatesthe energy consumed by every individual appliance ina house from a single energy measurement device likea smart-meter. This technique is gaining popularitydue to large-scale smart meter deployments worldwide[6]. The advantage of this approach is that it can beused in existing buildings easily without introducingany inconvenience to householders being non-intrusive.

Specifically, the problem of energy disaggregationcan be formulated as follows: Given the sequence ofaggregate power consumption X = {X1, X2..., XT }from N active appliances at the entry point of the meterat t = {1, 2..., T}, the task of the NILM algorithm isto infer the power contribution yit of appliance i ∈{1, 2...N} at time t, such that at any point in time t,

Xt =

N∑i=1

yit + σ(t) (1)

where σ(t) represents any contribution from appliancesnot accounted for and measurement noise. The key

4Sometimes referred to as Non Intrusive Load Appliance Mon-itoring (NILAM) or energy disaggregation

challenge to energy disaggregation problem is how to de-sign efficient generalized NILM algorithm across severalbuildings that can run in real-time using smart-meter. Atypical NILM algorithm consists of the following steps:power signal acquisition, event detection, feature ex-traction and learning& inference.

2.1 Power Signal Acquisition

This is the first step for any NILM algorithm and itinvolves acquiring aggregated load measurement at anadequate rate so that distinctive load patterns can beidentified. Several power meters such as Yomo [11] andc-meter [12] have been designed to measure the aggre-gated load of the building. A cost efficient approach foracquiring aggregate power data is to use smart meterswhich are currently being deployed as the requirementof smart-grid.

The aggregate power signal from these meters canbe recorded at different sampling rate. The samplingfrequency is determined by the measurements and elec-trical characteristics used by NILM algorithm [9]. Thesampling frequency can either be high-frequency orlow-frequency.

High-frequency is when sampling rate is in a rangeof 10 MHz to 100 MHz for the quantity whose electricalcharacteristics is to be determined. Power meters forthis range are often custom-built and expensive due tosophisticated hardware [9]. Smart meters belong in thelow sampling rate of the power signal which is less than1 Hz.

2.2 Event Detection

The NILM algorithm needs to detects the appliance op-erations status (e.g ON and OFF) from the power mea-surements. The changes in power levels (like ON/OFF)is done by the detection module. It is a complex processbecause of different types of appliance in buildings andthe different status to be detected like simple ON/OFF,finite state, constantly ON, and continuously variablestatus as identified in [8]. Based on different event-detection strategies, the current NILM approaches canbe classified as event-based or state-based.

Event-Based Approaches: The event-based ap-proaches focus on the state transition edges generatedby appliances and use change detection algorithm toidentify start and end of an event [10, 13]. The task ofchange detection algorithm is to detect changes in time-series aggregate load data due to one or more appliance

2

being switched ON/OFF or changing its state. A re-view on event detection algorithms used in the NILMliterature is presented in [14].

Event-based approaches rely on the fact that powermonitored in a home is constantly changing (rising andfalling, steps) as shown in Figure 1. These steps (ifsignificant enough) can be an indication that an eventhas occurred. Then, appliance signatures such as activepower, increasing/falling edge etc are extracted. Theextracted appliances signature are analysed to classifythe event based on appliance and its power consump-tion estimated. Different classification methods such asSupport Vector Method (SVM), neural networks, fuzzylogic, Naive Bayes, k-Nearest Neighbors (kNN), HiddenMarkov Model (HMM), decision trees and many otherhybrid approaches have been used [15, 16].

The performance of event-based approach is limitedby the fixed or adaptive threshold of the change de-tection algorithm, the large measurement noise, andsimilarities among steady-state signatures. In addition,miss detection and false detection of edges may arise inevent detection methods.

State-based Approaches: State-based NILM-approaches do not rely on event detectors, insteadthey represent each appliance operation using astate machine with distinct state transition based onappliance usage pattern [17]. They are based on thefact that when appliance turns ON/OFF or changesrunning states, create different edge measurementswhich have a probability distribution that match tothat appliance. State-based NILMs are usually basedin HMM and its variants [18, 19, 20, 21] .

State-based approaches are limited by the need forexpert knowledge to set a-prior value for each appliancestate via long periods of training. Besides, they havehigh computational complexity [9, 17] and do not havea good way to handle the fact that states may stayunchanged for long time intervals [22].

Figure 1: Schematic of Edge-based [23]

2.3 Feature Selection

Effective NILM algorithm requires unique features orsignatures that characterize appliance behaviour. Allappliances type have a unique energy consumption pat-tern often termed as appliance signatures. This uniqueenergy consumption pattern is often used to uniquelyidentify and recognize appliance operations from theaggregated load measurements [9]. According to [8], ap-pliances features are measured parameter of total loadthat give information about nature or operating stateof an individual appliance in the load. It is unique con-sumption pattern intrinsic to each individual electricalappliance [24]. There are two main classes of appliancesignatures used by NILM research for appliances iden-tification namely transient features and steady-statefeatures.

Transient signatures: Are short-term fluctuationsin power or current before settling into a steady-statevalue. These features uniquely define appliance statetransitions by extracting features like shape, size, dura-tion and harmonics of the transient [9]. They requirehigh sampling rates to obtain a high degree of signaluniqueness and longer monitoring time in order to cap-ture all operation cycles [17]. This in turn, demandsa costly hardware to be installed in households sincesmart meters reports only low-frequency power. For ex-ample, Patel et al. [25] use a custom built hardware todetect the transient noise from 0.01 kHz to 100 kHz. Theauthors use the fact that each appliances in state oper-ation transmits noise back to the power line.

Steady state features: Relate to more sustainedchanges in power characteristics when an appliancechange its running states. These features include; ac-tive power [20, 26], reactive power [27], current [28],current and voltage waveforms [29]; just to mention afew. The extraction of steady-state signature does notdemand high-end metering devices and can be obtainedfrom RMS values of current and voltage. Steady-statefeatures are the most commonly used features at low fre-quency in the literature. While most of prior works suchas [19, 20, 30] use real-power for disaggregation, [28]argue that current, rather than real power, is a moreeffective steady-state feature for energy disaggregationproblem.

2.4 Learning and Inference in NILM

In this stage the extracted appliances signature areanalysed in order to classify an appliance specific states

3

and estimate its corresponding power consumption. Thelearning algorithms are used to learn the model param-eters while the inference algorithms are employed toinfer appliance states from observed aggregate powerdata and estimate their corresponding power consump-tion. The algorithm for learning in NILM can be super-vised or unsupervised.

Supervised NILM techniques require a trainingphase in which both the aggregate data and the in-dividual appliance consumption are used. In that case,sub-metered appliances data or labeled observationsmust be collected from the target building. The processof collecting these data is expensive, time-consumingand limit the scalability of NILM systems [31]. Severalexisting works have focused on supervised learning tech-niques such as Support Vector Machine(SVM)[32, 33],Nearest Neighbour(k-NN) [16] and some forms of HMM[34].

Unlike supervised NILM, unsupervised NILM tech-niques do not require pre-training and thus suitable forreal time NILM application. Unsupervised NILM ap-proaches do not require individual appliance data, themodels parameters are captured only using the aggre-gated load, without the user intervention [35]. CurrentNILM research has focused on building unsupervisedlearning models which are less costly and more reliable[9, 35].

Unsupervised NILM approaches can further begrouped into three subgroups as suggested by [15]; Firstare the unsupervised approaches that require unlabelledtraining data to build appliance model or populate ap-pliances database. They are usually based on HMM andthe appliance models are either generated manually [21]or automatically [20] during the training phase. Mostof these approaches can not be generalized into unseenbuildings.

The Second groups includes unsupervised ap-proaches that use labelled data from known house tobuild appliances models which are then used for dis-aggregation in unknown (unseen) building. These ap-proaches require sub-metered appliances data to be col-lected from the training or known house. These data areused to build generic appliance models which is thenused in unseen buildings. Most deep learning basedNILM techniques such as in [36] fall in this category.

Lastly are the unsupervised approaches that donot require training before energy disaggregation takesplace. These approaches can perform energy disaggrega-tion without the need of sub-metered data or the priorknowledge [15, 37].

3 State-of-the-arts NILM Algo-rithms

Several state-of-the-art NILM unsupervised algorithmshave been proposed using different approaches such asdifferent variants of HMM [18, 19, 20, 21], Graph SignalProcessing (GSP) [15, 38] and Deep earning [39, 40].

3.1 Hidden Markov Model

HMM is a Markov model whose states are not directlyobserved instead each state is characterised by a proba-bility distribution function modelling the observationcorresponding to that state [18, 41]. There are two vari-ables in HMM: observed variables and hidden variableswhere the sequences of hidden variables form a Markovprocess. In the context of NILM, the hidden variablesare used to model appliances states (ON,OFF, standbyetc) of individual appliances and the observed variablesare used to model the electric usage. HMMs has beenwidely used in most of the recently proposed NILMapproach because it represents well the individual ap-pliance internal states which are not directly observedin the targeted energy consumption.

A typical HMM is characterised by the following:The finite set of hidden states S (e.g ON, stand-by,OFF, etc.) of an appliance, S = {S1, S2...., SN}. Thefinite set of observable symbol Y per states (power con-sumption) observed in each state, Y = {y1, y2...., yT }.The observable symbol Y can be discrete or a continu-ous set. The transition matrix A = {aij , 1 ≤ i, j ≥ N}represents the probability of moving from state Si to Sj

such that: aij = P (qt+1 = Sj | qt = Si), with aij ≤ 0and where qt denotes the state occupied by the systemat time t. The emission matrix B= P (yt | Sj) repre-senting the probability of emission of symbol yt ε Ywhen system state is Sj . The initial state probabilitydistribution is π = {πi} indicating the probability ofeach state of the hidden variable at t = 1 such that,πi = P (q1 = si), 1 ≤ i ≥ N . The set of all HMM modelparameters is represented by λ = {π,A,B}.

When applying HMM to a real world problem, twoimportant problem must be solved. First how to learnthe model parameter λ given the sequences of observ-able variable Y . Second, given the model parameterλ and the sequences of observable variable Y how toinfer the optimal sequences of hidden state S. Theseproblems are referred to as learning and inference prob-lems. Various algorithms such as Baum-Welch algorithmand the Viterbi algorithm have been proposed to solvethese problems.

The factorial HMM (FHMM) is an extension of

4

HMM with multiple independent hidden state sequencesand each observation is dependent upon multiple hid-den variables [42]. In FHMM, if we consider Y ={y1, y2, .....yT } to be the observable sequences thenS = {S(1), S(1), .....S(M)} represents the set of hid-

den state sequences where S(i) = {S(i)1 , S

(i)2 , .....S

(i)T }

is the hidden state sequence of the chain i as shownin Figure 2. FHMM are preferred to HMMs for mod-

Figure 2: Graphical representation of FHMM

elling time series generated by the interaction of severalindependent process. However, the computational com-plexity of both learning and inference is greater forFHMMs compared to HMMs. In addition, the infer-ence techniques for FHMM based approach is highlysusceptible to local optima [9].

Several HMMs based NILM algorithms for energydisaggregation at low sampling rate has been proposedin the literature. In [18] unsupervised technique for en-ergy disaggregation using a combination of four FHMMvariants is proposed. The authors use low-frequency realpower feature and assume a binary state of appliances(ON and OFF state only). To learn model parameters,Kim’s approach uses Expectation Maximasation (EM)algorithm and employ Maximum Likelihood Estima-tion(MLE) algorithm to infer load states. The perfor-mance of Kim’s technique is limited to few number ofappliances, require appliances to be manually labelledafter disaggregation and suffer from high computationalcomplexity which makes it unsuitable for real-time ap-plications [43].

The work presented in [19] propose a new inferencealgorithm for unsupervised energy disaggregation calledAdditive Factorial Approximate MAP (AFMAP) that iscomputationally efficient and does not suffer from localoptima. The AFMAP algorithm is used to perform ap-proximate inference over the additive FHMM. However,

the model requires appliances to be manually labelledafter off-line disaggregation and have a low performancefor electronics and kitchen appliances.

Parson et al. [20], introduce an approach that usedifference HMM from [19] as Bayesian network for dis-aggregation of active power with 60 s sampling rate. Toperform inference, the authors use an extension ofviterbi algorithm and propose an EM training processto build a generic appliance model for learning themodel parameters. This generic model is then tuned tospecific appliance instances using only aggregate datafrom home in which NILM is being applied. The activetuning process requires a training window of data whereno other appliance changes state. For the cyclic typesof appliance such as fridges, this is easy since it is oftenthe only appliance running at night, but it is generallydifficult for other appliances [44].

A fully unsupervised NILM framework based onnon-parametric FHMM using low-frequency real powerfeature is presented in [37]. They use the combinationof slice sampling and Gibbs sampling to do inferencethat simultaneously detect number of appliances anddisaggregate the power signal from the composite sig-nal. However, for larger disaggregation problems thisinference algorithm becomes a limitation as it may stuckin local optima [45]. Besides it difficult to see this algo-rithm runs in real-time owing to complexity problem ofFHMM.

Makonin et al. [21] present another NILM algorithmfor low-frequency sampling rate that uses a super-stateHMM in which a combination of modeled appliancesstates is represented as one super state. The authorspropose a new variant of viterbi algorithm called sparseViterbi algorithm. This algorithm perform computa-tionally efficient exact inference instead of relying onapproximate inference method like in FHMM basedapproach. Makonin’s approach preserves dependenciesbetween appliances, can disaggregate appliances withcomplex multi-state power signatures and can run inreal-time on an inexpensive embedded processor. Al-though the reported approach can disaggregate largenumber of super-state, there is still a limitation in timeand space since number of super-states grow exponen-tially with the number of appliances.

Despite the fact that HMM-based NILM approacheshave been widely used in energy disaggregation they re-quire an expert knowledge to set a-priori values for eachappliance state. Their performance are thus limited byhow well the generated models approximate appliancetrue usage [15]. Moreover, HMM-based approaches havebetter performance for controlled multi-state applianceslike refrigerator, but their performance degrades for un-

5

controlled multi-state and variable appliances [22].

3.2 Graph Signal Processing

Graph Signal Processing (GSP) or signal processing ongraph is an emerging field that extends classical signalprocessing theory to data indexed by general graphs[46]. GSP represents a dataset using a graph signal de-fined by a set of nodes and a weighted adjacency matrix[30]. Each node in the graph corresponds to an elementin the dataset while the adjacency matrix define alldirected edges in the graph and their weights, where as-signed weights reflects degree of similarity or correlationbetween the nodes [38]. GSP is the powerful, scalableand flexible signal processing approach that is suitablefor machine learning and data mining problems. In par-ticular, GSP is suitable for data classification problemsin which training periods are short and inefficient tobuild appropriate class models [47].

Given a set of aggregate power measurement X wedefine a graph G = {V, A} where V is the set of nodescorresponding to the acquired measurements and A isthe weighted adjacency matrix of a graph which definethe edge of a graph. Each element xi ∈ X correspondsto a node vi ∈ V and each weight Aij of the edgebetween nodes vi and vj reflects the degree of relationbetween xi and xj . The weight of a node Aij is usuallydefined using gaussian kernel weighting function, themost used kernels in machine learning for expressingsimilarities between dataset defined by equation (2).

Aij = exp[− (xi − xj)2

ρ2

](2)

where ρ is a scaling factor [17]. A graph signal is thendefined as a map from the map on the graph nodesV to the set of complex number s where each elementsi ∈ s is indexed by nodes vi ∈ V [46]. In the context ofenergy disaggregation, each vertex vi ∈ V is associatedto aggregate power variation signal between adjacentpower reading one sample ∆Xt, where ∆Xt = Xt+1 −Xt. For further literature on GSP, interested readermay refer to [46].

Recently, researchers have proposed different GSP-based approach for NILM. The first GSP-NILM ap-proach that is neither state-based nor event-based waspresented in [38]. The authors, leverage on the workby [47] to perform low-complexity multi-class classifi-cation of the acquired active power readings withoutthe need for event detection to detect appliance chang-ing states. However, this approach is supervised andemploys GSP only for data classification [30].

Zhao et al. [15, 30] propose a blind, low-rate andsteady state event-based GSP approach that does notrequire any training. The proposed GSP-NILM dis-aggregate any aggregate active power dataset withoutany prior knowledge and relay upon the GSP to per-form adaptive threshold, signal clustering and patternmatching [15, 30]. Zhao’s approach work well if theaverage load of each appliance is distinct enough fromother appliances load and if the power of each loaddoes not fluctuate much. This is not a typical case inmost building and thus limit the performance of Zhao’salgorithm. Additionally, the proposed GSP approachrequire appliances to be manually labelled after disag-gregation, highly affected by noise and it’s performanceis also limited by the event detection performed viaadaptive thresholding [47].

3.3 Deep Learning

Deep learning is the machine learning approach thathas drawn heavily on the knowledge of the human brain(artificial neural networks), statistics and applied math-ematics [48]. It is the artificial neural networks (ANN)that are composed of many layers. For a comprehensivesurvey and more details on deep-learning interestedreader should refer to [48, 49].

In recent years, deep learning has made substan-tial improvements in several fields such as computervision [50], speech recognition [51] and machine trans-lation [52]. This is mainly due to more powerful com-puters, larger datasets and techniques to train deepernetworks. In addition, deep learning models are flexi-ble (enabling similar models to be used in wide rangeof problems) [49]. Recently, different deep learning ar-chitecture such as Recurrent Neural Network (RNN)[39], Convolutional Neural Network (CNN) [39, 40, 53],Auto encoder [39] and a combination of deep learningand HMM [44, 53, 54] has been employed to the energydisaggregation problem.

A deep learning novel approach for energy disaggre-gation that identifies additive sub-components of thepower signal in an unsupervised way is presented in[55]. The approach uses high-frequency measurementsof current, assume two-state appliances models andrequires buffering of all the data until inference. Barsimet al. [29] propose neural network ensembles approachto address NILM problem. The ensemble of neuralnetworks are used in appliance identification problemfrom the raw high-resolution current and voltage wave-forms. The work by Kelly et al. [36], adapted threeneural network architectures to low-frequency energydisaggregation problem. In [40], Paulo et al. present

6

a comparison of variety of CNN and RNN for energydisaggregation across a number of appliances.

The work presented by [44] uses CNN network toextract appliance features which are then used as obser-vations to a hidden semi Markov model (HSMM). Huss’smodel performed considerably better than a CNN alonewith reduced computational cost. In [22], Mauch etal. propose a novel combination of HMM with deepneural network (DNN) for load disaggregation. Mauch’sapproaches trains HMM with two emission probabili-ties, one for single load to be extracted which is mod-elled as a Gaussian distribution and other for theaggregate signal in which DNN is used. Despite thefact that Mauch’s DNN-HMM models outperformedFHMM, it was trained using few data (20.7 days REDDdata). Deep learning models require lots of data in orderto be well generalized.

Zhang et al. [53] propose a sequence-to-point learn-ing with CNN for energy disaggregation in which asingle-midpoint of an appliance window is treated asclassification outputs of a neural network with the mainswindow being the input. This differ from the work pre-sented in [39] in which a given window of the mainsequence is treated as input and a sequence of targetappliance as the output of the neural network. Theauthors further integrate CNN and AFHMM using log-arithmic opinion pool method. Zhang approach wasfound to outperform HMM based approaches [19, 56]and deep learning approach by Kelly et al. [39] withreduced computational cost.

4 Evaluating NILM Algorithms

Defining relevant evaluation standards such as perfor-mance metrics and benchmarking framework are cru-cial to enable empirically evaluation of NILM algo-rithms and get fair performance comparison betweenalgorithms.

4.1 Performance Metrics

NILM researchers use several performance metrics toevaluate energy disaggregation algorithms. To measurehow well an algorithm can predict how an appliance isrunning in each state(switching ON or OFF), severalNILM researchers use accuracy metric defined in (3).

Acc. =correct matches

total possible matches(3)

Since appliance usages in a house is a relative rare event,accuracy metric is sometimes misleading. Accuracy

metric is not descriptive for an appliance that is mainlyOFF [18, 44]. For example, if a TV is ON 10% of thetime, an algorithm that predict that the TV is alwaysoff will have 90% accuracy without any ability to predictits usage. As results, classification accuracy measures,such as F-Measure (FM ) has been used [32, 38, 57].

F-Measure is the harmonic mean of precision (PR),the positive predictive values and recall (RE) whichis the true positive rate or sensitivity defined by equa-tions (4) to (6)

PR =TP

(TP + FP )(4)

RE =TP

(TP + FN)(5)

FM =2× PR×RE(PR+RE)

(6)

where true positive (TP) presents the correctly detectedstate of ON or OFF, false positive (FP) represents anincorrect detection that is predicted appliance was ONwhile OFF, and false negative (FN) indicates that theappliance used was not identified i.e. was ON butnot detected. However, F-measure is limited to binaryappliances (OFF/ON) and not applicable for multi-stateappliances [58].

To account for multi-state appliances, Makonin etal. [58] introduce a finite-state F-Measure (FS-FM ) byadapting the work by [18]. Their approach split TPinto two: inaccurate true-positives (ITP) and accuratetrue-positives (ATP). The ITP is a partial penalizationmeasure which converts the binary nature of TP into amisclassification that is not binary in nature defined byequations (7) to (8)

ITP =

∑Tt | sit − sit |Ki

(7)

ATP = 1− ITP (8)

where sit is the estimated state from appliance i at timet, sit is the ground truth state, and Ki is the number ofstates for appliance i. To take account for these partialpenalizations, [18] redefined precision (PR) and recall(RE) stated as equations (9) to (10).

PR =ATP

(ATP + ITP + FP )(9)

RE =ATP

(ATP + ITP + FN)(10)

The FS-FM remains the harmonic mean of the newprecision and recall. Accuracy and F-Measure are called

7

classification metrics because they only measure howaccurately NILM algorithms can predict what applianceis running in each state.

To measure how well NILM algorithm is able to esti-mate and assign the power consumed by each appliancedifferent measures have been used. Root mean squareerror (RMSE) is one of such estimation accuracy mea-sure that NILM researchers use [2, 20, 59]. The RMSEbetween the estimated power yit consumed at time t forappliance i and the ground truth power yit consumedat time t for appliance i is given by equation (11)

RMSE =

√1

T

∑t

(yit − yit) (11)

where T denotes the number of samples recorded. UsingRMSE measure is hard to compare how the disaggrega-tion of one appliance performed over another since thismeasure is not normalized [60]. To address this issue,other researchers [19, 59, 61] use the normalized dis-aggregation error that provides a normalized measureof the difference between the actual and the estimatedpower consumptions of the ith appliance given by equa-tion equation (12).

de =

√√√√√√√∑t,i

||yit − yit||2∑t,i

||yit||2(12)

The estimation accuracy proposed by [62] can beused to evaluate the overall performance of the NILMalgorithm and is defined by equation (13)

EAcc =

[1−

∑Tt=1

∑Ni=1 |yit − yit|

2∑T

t=1

∑Ni=1 |yit|

](13)

where T is the time sequence or number of disaggregatedreadings and N is the number of appliances. Using thismetric, we can derive the estimation accuracy for eachappliance by eliminating the summations over N as inequation (14)

E(i)Acc =

[1−

∑Tt=1 |yit − yit|

2∑T

t=1 |yit|

](14)

The disaggregation error (de) in equation (12) and

estimation accuracy for each appliance (E(i)Acc) in equa-

tion (14) measure how well the estimated power profilesmatch the actual power profiles overtime [63]. Thelow values of the disaggregation error or RMSE (high

value of the estimation accuracy) imply an accuratedisaggregation.

The work by [63] introduce another estimation met-ric called the estimated energy fraction index (EEFI)which provides the fraction of energy assigned to theith appliance. The EEFI is defined by equation (15)

EEFI =

∑Tt=1 y

it∑T

t=1

∑Ni=1 y

it

(15)

and it should be compared with the actual energy frac-tion index (AEFI) which provides the actual fractionof energy consumed by the ith appliance defined byequation (16)

AEFI =

∑Tt=1 y

it∑T

t=1

∑Ni=1 y

it

(16)

4.2 Benchmarking

The lack of efficient benchmarking framework is anothermajor challenge in the field of NILM research. This hasgreatly attributed by the fact that there is no referencealgorithms implementation. Hence, NILM researchersuse different metrics, different datasets, and differentpre-processing steps [36, 64]. It is therefore empiricallydifficult to perform direct comparison or to reproduceresults of the state-of the art NILM algorithms. As theresult, the newly proposed approaches are rarely usethe same benchmarking algorithms and/or most of thecomparison studies are sometime misleading or favourthe work presented [44].

To address the above mentioned challenge, Batraet al. [64] and Kelly et al. [65] developed Non-IntrusiveLoad Monitoring Toolkit (NILMTK)5. NILMTK is anopen-source NILM toolkit written in Python and de-signed specifically to enable the comparison of NILMalgorithms across diverse data sets. It contains dataset parsers, data set analysis statistics, preprocessorsfor reformatting data sets, benchmark disaggregationalgorithms, accuracy metrics and rich metadata supportvia the NILM Metadata6[66].

The NILMTK toolkit provides a complete pipelinefrom data sets to accuracy metrics, thereby loweringthe entry barrier for researchers to implement a newalgorithm and compare its performance against thecurrent state of the art [64]. Several studies such as[2, 3, 39, 67, 68] have used this toolkit to implement andevaluate their NILM algorithms. The work presented

5http://nilmtk.github.io/6https://github.com/nilmtk/nilm metadata

8

in [69] use NILMTK to analyse and validate the UK-DALE dataset.

The release of NILMTK was followed by the NILM-Eval framework7 developed by ETH Zurich distributionsystem research group8. The NILM-Eval is a com-prehensive evaluation framework for NILM algorithmswritten in MATLAB. It is designed to facilitate thedesign and execution of large experiments that considerseveral different parameter settings of various NILMalgorithms [4]. The NILM-Eval framework allows newNILM researcher to replicate experiments performed byothers or evaluate an algorithm on a new dataset andfine-tune configurations to improve the performance ofan algorithm in a new setting [36]. In their study, [70]implemented the algorithms on MATLAB and evaluatedthe performance of their approach on the NILM-Evalframework.

Recently, [71] propose an approach that aim to helpNILM researchers systematically evaluate and bench-mark NILM technology across different datasets andperformance metrics, using open source technologiesand well-established performance metrics and evalua-tion techniques. However, the tool is limited to event-based approaches.

5 Non-Intrusive Load Monitor-ing Use-Cases

Despite the fact that NILM techniques promise severaluseful applications for energy conservation in build-ings, its broader applications have not been fully real-ized. This is attributed to the fact that most of NILMresearchers have focused on accurate disaggregation andnot concrete application. According to [81] NILM re-search needs to move past computing appliance-level en-ergy breakdowns with emphasize on designing new andnovel applications that lead to sustainable energy sav-ing in buildings. Converting the energy disaggregationdata into actionable feedback will improve energy effi-ciency in residential buildings and engage consumers inthe path toward sustainable energy in buildings. NILMresearchers should put much of their emphasis in de-signing new and novel applications rather than seekingincremental improvements in algorithms accuracy.

One of the most important applications of NILMis the provision of real-time actionable energy feed-back or recommendations to households that couldlead to sustainable energy saving. This information

7https://github.com/beckel/nilm-eval8http://vs.inf.ethz.ch/

could help households identify unnecessary consump-tion, identify inefficient appliances and suggest optimi-sations, raise alerts and make consumers more awareof the energy they consume. For example, using thisinformation households can detect when appliance beswitched to a more energy efficient mode [43]. Thisgreatly helps households not only understand theirconsumption pattern but also become part and par-cel of energy conservation [82]. Parson et al. present anapplication that use NILM algorithms to infer fridgeusage in UK by providing households with feedbackon energy-money trade-offs of shifting to new energy-efficient fridges [20]. Another application of NILM toa large number of households smart meter data is pre-sented in [83]. The authors propose an approach bywhich the energy efficiency of fridge and freezers areestimated from an aggregate load and calculate the timeuntil the energy savings of replacing such applianceshave offset the cost of the replacement appliance.

Temporal pattern such as unusual power consump-tion in NILM data can be used in detection of faultappliances or malfunctioning appliances in residentialbuildings. This information can be further used to deter-mine time to retrofit old appliances and detect degradedperformance of appliance in buildings. In such a situa-tion, households can be provided with real-time alertfeedback by either suggesting a more energy efficiencyreplacement for less expensive appliances or a repairfor more expensive appliances. Through field test ofNILM algorithm, [8] detected a failed appliance by itsabnormal low power consumption and faulty refriger-ator which was ON almost all of the time. Batra etal. develop and demonstrate techniques that use NILMactionable feedback about a refrigerator and HVAC toresidential users [2]. Their application provide targetedactionable feedback with specific actions such as repairor fix to users with much more energy due to fridgeusage than normal or fridge that are malfunctioning ormiss configured.

NILM system can also be used to support and en-hance continues energy audits in buildings that cur-rently require multiple measurements across build-ings. Energy audit is a process by which a buildingis inspected and analysed to determine how energy isused with aim to identify opportunity for energy con-servation [84]. Detail analysis of NILM data can beused to either suggest ways of reducing consumptionand cost or confirm the energy saving resulting fromconservation measures. In [84] Berges et al. present anexperimental NILM system for supporting residentialenergy audits.

NILM can further be used to allow and verify de-

9

Table 1: Publically Available Energy Dataset Comparison

Dataset Location Duration No.ofhouses

Sensors/house Resolution Features Other Data

REDD [62] USA 3-19 days 6 24 15KHz(Aggr), 0.5Hz and 1Hz(sub)

V and P (Aggr), P (sub)

BERDS [72] USA 1 year 1 4 20sec P,Q and S climate dataBLUED [73] USA 8 days 1 Aggregated 12KHz(Aggr only) I, V and State transition label for

each appliance.Smart [74] USA 3 months 3 21-26 circuit meters 1Hz P and S (Aggr), P (Sub) electricity generation data from

on-site solar panels and wind tur-bines, outdoor weather data, tem-perature and humidity data in in-door rooms

DRED [75] Netherlands 6 months 3 12 appliances 1Hz P (Aggr & Sub) indoor temperature, outsidetemperature, wind speed,pre-cipitation, humidity andoccupancy data.

Tracebase[76]

Germany N/A 15 158 devices 1-10sec(Sub only) P

AMPDS [28] Canada 1 year 1 19 1min V, I, F P, Q, S and Pf water and natural gas,AMPds2 [77] Canada 2 1 21 1min V, I, F P, Q, S , Pf ,real energy,

reactive energy, and apparent en-ergy

water and natural gas, weatherdata and utility billing data.

UK-DALE[69]

UK 499 days, 2.5years(house 1)

5 5-54 devices 16 kHz(Aggr) and 1/6 Hz(Sub) P and switch status

iAWE [59] India 73 days 10 33 devices 1sec(Aggr) and 1sec or 6sec (Sub) V, I, F, P and phase Water and ambient conditionsREFIT [78] UK 2years 20 11 8sec P Gas and environmental dataGREEND[79]

Austria/ Italy 1year 9 9 1Hz P

ECO [4] Switzerland 8months 6 1Hz P and Q Occupancy informationIHEPCDS 9 France 4 years 1 3 1min V, I, P and QOCTES 10 Scotland,Iceland

&Finland413months 33 Aggregated 7sec P and phase

HES UK 1month(255houses),1year(26houses)

251 13-51 2min P

ACS-F1 [80] Switzerland 2, 1 hoursessions NA 100, 10 types 10sec P, Q, I, f, V and phase

Aggregte (Aggr), Sub-metering (sub), Active Power (P), Reactive Power (Q), Apparent Power (S), Energy (E), Frequency (f),

Voltage (V) and Current (I)

mand side load management control response whereusers are expected to change their use to respond tochanges in electrical energy pricing through deferringsome loads. By knowing a homes typical usage by de-vice, an energy management system can perform device-specific demand response much more effectively [85].

Detail analysis of NILM data could be used for se-lection of pricing or incentive mechanism that maximizethe effectiveness of demand response. For example, us-ing NILM data, demand response designer can identifyhighest consuming appliances and their time usageswhich can be used for deriving load shift ability duringpeak hours.

The work by Huang et al. present an HMM-basedalgorithm to estimate individual household heating us-age from aggregate smart meter data [86]. The authorsdemonstrate its application to demand response andenergy audit services for thermostatically controlledheating appliances. The work presented in [87] demon-strate the application of real-time NILM algorithm intodemand response response.

Recently [88] present novel techniques that use un-supervised NILM to predict household occupancy andstatic household properties such as age of the home,

size of the home, household income and number ofoccupants.

6 Energy Datasets

In the quest to design, test and benchmark a highperformance energy disaggregation algorithms, NILMresearchers require the availability of open-access energyconsumption datasets. These dataset record the aggre-gate demand of the whole house as well as the groundtruth demand of individual appliances and offers a realand noisy environment which can lead to more accuratealgorithms design.

Reference Energy Disaggregation Data Set (REDD)is the first public energy dataset released by MIT in 2011[62]. REDD contain high and low frequency readingsfrom 6 households in USA recorded for short period(between a few weeks and a few months). This datasetis widely used for the evaluation of NILM algorithms.

Recent years has seen the emergency of several pub-licly available datasets such as UK-DALE [69], AMPDsand AMPDs2 [28, 77], ECO dataset [4], REFIT dataset[89] and GREED dataset [79]. The comparison of vari-ous publicly available dataset with their characteristics

10

is shown in table 1. This comparison is an extensionof the proposed one in [35] and [89] with an update ofthe recent published data and additional informationincluded in some datasets.

7 Challenges and Future Re-search Directions

In the previous sections we have presented an up todate overview of NILM system and its associated meth-ods and techniques for energy disaggregation problemhighlighting the gaps and limitations. Despite severalefforts done by previous NILM researchers, there arestill several challenges which need to be addressed.

As a matter of fact, most of the prior NILM algo-rithms have been developed and tested in developedcountries. Developing countries like Tanzania offersunique characteristics such as unreliable grid whichis uncertain with blackouts and brownouts [90], dif-ferent sets of appliances such as use of second-handappliances [91] and different consumer behaviour. Ba-tra et al. observed a significant voltage fluctuation andpower outages in the data collected in India [92]. Allthese factors affect the use of energy and need to beconsidered in the design and development of NILMmethods and techniques for energy disaggregation.

Second, most of the prior NILM techniques can notperform real-time disaggregation owing to algorithmcomplexity. Practical NILM algorithms need to pro-cess on-line data and react in real-time to changes inthe power being monitored [21]. The few that providereal-time disaggregation utilize cloud services that in-troduce privacy and security concern to householdsdata. Future research should focus on real-time disag-gregation by reducing the complexity of disaggregationalgorithm. There is also a need to explore different pri-vacy and security techniques suitable for disaggregationalgorithms that utilize cloud services.

Likewise, generalizing the learned NILM models toa new building and automatically annotating applianceevents is still a problem in NILM. Previous works relyon manually labelling appliance events after disaggre-gation or assumes that sub-metered ground-truth isavailable [57]. It is very important for NILM modelto be generalised to useen buildings because it is veryrarely to find sub-metered data. Future work shouldfocus on unsupervised NILM learning algorithms thatdo not require human labeling of data and which canbe generalized across multiple buildings.

The recent study by Kelly et al. [39] demonstratedthat the use of deep learning for energy disaggregation

can be generalized well to unseen buildings. Futureworks should explore and investigate different unsuper-vised learning and deep learning algorithms for energydisaggregation. It has also been shown that the combi-nation of deep learning and probabilistic model suchas HMM have quite promising results for energy dis-aggregation problem [44, 53, 54]. Thus future worksshould also explore and investigate different hybriddeep-learning-HMM framework for energy disaggrega-tion problem.

In like manner several NILM algorithms have fo-cused on computing appliance level energy breakdownand not usability or concrete application that emphasison quantifiable energy saving in building [81]. Simplyproviding appliance-level energy breakdown is not acompelling application of NILM as it does not directlylead to quantifiable improvements in energy efficiency[93]. Thus there is a need to analyze energy disaggre-gation data and organize it into actionable feedbackthat actively stimulate energy efficiency in residentialbuilding. There is a need to expose novel NILM use-cases that use energy-disaggregation data such as howto predict electrical fires accidents or use smart-meterdata for electricity theft detection just to mention afew.

Equally important, energy disaggregation researchhave no consistent way to measure and evaluate per-formance and quality of NILM algorithms. Most of theearlier works evaluate their approaches using a differ-ent set of performance metrics which make difficultiesto fairly compare these algorithms [27]. In addition,many of these metrics are incomparable across differ-ent algorithms for the same problem variant [81] andthe numerical performance calculated by such metricscannot be compared between any two papers [64]. Fu-ture research should also focus in standardizing NILMperformance metrics.

Lastly, to develop disaggregation algorithms, re-searchers require both aggregate demand per buildingand the ground truth demand of individual appliancesdata. However, existing energy data set suffer from sev-eral problem such as incorrectly labelled sub-meters(channel labelled fridge actually records the kitchen ra-dio). There is also no ground-truth labels for importantNILM use-cases. For example, one important NILMuse-case might be to tell people when their fridge’s doorseal needs replacing11. Apart from that existing datasetis are from Europe, Canada, USA and India. There areno public datasets from developing countries such asAfrica. There is thus a need to develop more data-sets

11http://jack-kelly.com/simulating disaggregated electricity data

11

from different geographical location. However, collectionof these data is very costly and time-consuming.

Future research should consider developing a realis-tic simulators for simulating endless amount of ”near-perfect”, realistic disaggregated electricity data. Re-cently Chen et al. [94] present a publicly-availabledevice-accurate smart home energy trace generatorwhich generates energy usage traces for devices by com-bining a device energy model, capturing its pattern ofenergy usage when active and a device usage modelbased on its frequency, duration, and time of activ-ity. By leveraging on this simulator further researchcan build different statistical models for appliances indifferent geographical location and for several usagepatterns.

8 Conclusions

In this work, a review of an up to date NILM systemand its associated methods and techniques for energydisaggregation problem is presented. The review drawsseveral conclusions;

First while several NILM techniques has been pro-posed for reduction and or controlling energy consump-tion in residential building in developed countries, thereis lack of research on the use of NILM in developingcountries.

Second, despite the major leaps forward in theNILM field, the energy disaggregation is by no meanssolved. State-of-the art algorithms still leave a lot ofchallenges when it comes to real-time disaggregationthat is general enough to be deploy in any household.

Third the standardization of NILM performancemetrics, development of efficient NILM benchmarkingframework and availability of high-quality energy dataset are critical for the advancement of energy disaggre-gation research.

Lastly, despite the fact that NILM techniquespromise several useful use-cases, its broader applicationshave not been fully realized.

9 Acknowledgments

The authors would like to thank Tanzania Communica-tion Authority (TCRA) for supporting this research.

References

[1] Vincent Kitio. Promoting Energy efficiency inBuildings in East Africa. In UNEP-SBCI Fall

Symposium on Sustainable Buildings, Paris, France,2013.

[2] Nipun Batra, Amarjeet Singh, and Kamin White-house. If You Measure It, Can You Improve It?Exploring The Value of Energy Disaggregation. InIn Proceedings of the 2nd ACM International Con-ference on Embedded Systems for Energy-EfficientBuilt Environments - BuildSys ’15, pages 191–200,2015.

[3] Nipun Batra, Amarjeet Singh, and Kamin White-house. Gemello : Creating a Detailed EnergyBreakdown from just the Monthly Electricity Bill.In In Proceeding of the 22 ACM SIGKDD Con-ference on Knowledge Discovery and Data Mining,2016.

[4] Christian Beckel, Wilhelm Kleiminger, ThorstenStaake, and Silvia Santini. The ECO Data Set andthe Performance of Non-Intrusive Load MonitoringAlgorithms. In Proceedings of the 1st ACM Con-ference on Embedded Systems for Energy-EfficientBuildings, pages 80—-89, 2014.

[5] Mingjun Zhong, Nigel Goddard, and Charles Sut-ton. Interleaved Factorial Non-Homogeneous Hid-den Markov Models for Energy Disaggregation.In Proceeding of the NIPS workshop on MachineLearning for Sustainability, pages 1–5, 2013.

[6] Antonio Reyes Lua. Location-aware Energy Disag-gregation in Smart Homes. Master’s thesis, DelftUniversity of Technology, 2015.

[7] Jon Froehlich, Eric Larson, Sidhant Gupta, GabeCohn, Matthew Reynolds, and Shwetak Patel. Dis-aggregated end-use energy sensing for the smartgrid. IEEE Pervasive Computing, 10(1):28–39,2011.

[8] G.W. Hart. Nonintrusive appliance load monitor-ing. Proceedings of the IEEE, 80(12):1870–1891,1992.

[9] Ahmed Zoha, Alexander Gluhak, Muhammad AliImran, and Sutharshan Rajasegarar. Non-intrusiveLoad Monitoring approaches for disaggregated en-ergy sensing: A survey. Sensors (Switzerland),12(12):16838–16866, 2012.

[10] Karim Said Barsim, Roman Streubel, and BinYang. An Approach for Unsupervised Non-Intrusive Load Monitoring of Residential Appli-ances. In Proceedings of the 48th International Uni-

12

versities’ Power Engineering Conference (UPEC),pages 1–5, Dublin, 2013.

[11] Christoph Klemenjak, Dominik Egarter, and Wil-fried Elmenreich. YoMo: the Arduino-based smartmetering board. Computer Science - Research andDevelopment, 31(1):97–103, 2016.

[12] Stephen Makonin, William Sung, Ryan Dela Cruz,Brett Yarrow, Bob Gill, Fred Popowich, and Ivan V.Bajic. Inspiring energy conservation through opensource metering hardware and embedded real-timeload disaggregation. In Proceeding of the 5th IEEEPES Asia-Pacific Power and Energy EngineeringConference (APPEEC 2013), pages 1–6, 2013.

[13] Yung Fei Wong, Y. Ahmet Sekercioglu, Tom Drum-mond, and Voon Siong Wong. Recent approachesto non-intrusive load monitoring techniques in res-idential settings. IEEE Symposium on Compu-tational Intelligence Applications in Smart Grid,CIASG, pages 73–79, 2013.

[14] K. D. Anderson, M. E. Berges, A. Ocneanu, D. Ben-itez, and J. M. F. Moura. Event detection for NonIntrusive load monitoring. In IECON 2012 - 38thAnnual Conference on IEEE Industrial ElectronicsSociety, 2012.

[15] Bochao Zhao, Lina Stankovic, and Senior Mem-ber. On a Training-Less Solution for Non-IntrusiveAppliance Load Monitoring Using Graph SignalProcessing. IEEE Transactions on Smart Grid, 4,2016.

[16] Hana Altrabalsi, Jing Liao, Lina Stankovic, andVladimir Stankovic. A low-complexity energy dis-aggregation method : Performance and robustness.In Proceeding of the IEEE Symposium on Compu-tational Intelligence Applications in Smart Grid(CIASG),, pages pp. 1–8., Orlando, FL, 2014.

[17] Kanghang He, Lina Stankovic, Jing Liao, andVladimir Stankovic. Non-Intrusive Load Disag-gregation using Graph Signal Processing. IEEETransactions on Smart Grid, PP(99):1–1, 2016.

[18] H. Kim, M. Marwah, M. F. Arlitt, G. Lyon, andJ. Han. Unsupervised Disaggregation of Low Fre-quency Power Measurements. Proceedings of the11th SIAM International Conference on Data Min-ing, pages 747–758, 2011.

[19] Zico Kolter, Tommi Jaakkola, and J Z Kolter. Ap-proximate Inference in Additive Factorial HMMs

with Application to Energy Disaggregation. InProceedings of the International Conference on Ar-tificial Intelligence and Statistics, pages 1472–1482,2012.

[20] Oliver Parson, S Ghosh, M Weal, and A Rogers.Non-intrusive Load Monitoring using Prior Modelsof General Appliance Types. In Proceeding of thetwenty-Sixth AAAI Conference on Artificial Intel-ligence (AAAI-12), Toronto, Canada, apr 2012.

[21] Stephen Makonin, Fred Popowich, Ivan V. Bajic,Bob Gill, and Lyn Bartram. Exploiting HMM Spar-sity to Perform Online Real-Time NonintrusiveLoad Monitoring. IEEE Transactions on SmartGrid, 2015.

[22] Lukas Mauch, Karim Said Barsim, and Bin Yang.How well can HMM model load signals. In Pro-ceeding of the 3rd International Workshop on Non-Intrusive Load Monitoring (NILM 2016), 2016.

[23] Zhaoyi Kang. Electronic Thesis and Disserta-tions. PhD thesis, University of California, Berke-ley, 2015.

[24] Jian Liang, Simon K K Ng, Gail Kendall, and JohnW M Cheng. Load Signature Study Part I : BasicConcept , Structure , and Methodology. IEEETransactions on Power Delivery, 25(2):551–560,2010.

[25] Shwetak N Patel, Thomas Robertson, Julie AKientz, Matthew S Reynolds, and Gregory DAbowd. At the Flick of a Switch : Detectingand Classifying Unique Electrical Events on theResidential Power Line ( Nominated for the BestPaper Award ). In Proceeding of the UbiquitousComputing (UbiComp 07), pages 271–288. Springer,2007.

[26] Jing Liao, Georgia Elafoudi, Lina Stankovic, andVladimir Stankovic. Non-intrusive appliance loadmonitoring using low-resolution smart meter data.In Proceeding of the Smart Grid Communications(SmartGridComm), 2014 IEEE International Con-ference, pages 535–540, Venice,, 2014.

[27] Michael Zeifman, Senior Member, and Kurt Roth.Nonintrusive Appliance Load Monitoring : Reviewand Outlook. IEEE Transactions on ConsumerElectronics,, 57(1):76–84, 2011.

[28] Stephen Makonin, Fred Popowich, Lyn Bartram,Bob Gill, and Ivan V. Bajic. AMPds: A public

13

dataset for load disaggregation and eco-feedbackresearch. In Proceeding of the Annual ElectricalPower and Energy Conference (EPEC), 2013.

[29] Karim Said Barsim, Lukas Mauch, and Bin Yang.Neural Network Ensembles to Real-time Identifi-cation of Plug-level Appliance Measurements. InProceeding of the 2016 NILM workshop, 2016.

[30] Bochao Zhao, Lina Stankovic, and VladimirStankovic. Blind Non-intrusive Appliance LoadMonitoring using Graph-based Signal Processing.In IEEE GlobalSIP, pages 68–72, 2015.

[31] Karim Said Barsim and Bin Yang. Toward a Semi-Supervised Non-Intrusive Load Monitoring Systemfor Event-based Energy Disaggregation. In Pro-ceeding of the 2015 IEEE Global Conference onSignal and Information Processing (GlobalSIP),pages 58–62, Orlando, FL, USA, 2015.

[32] Hana Altrabalsi, Vladimir Stankovic, Jing Liao,and Lina Stankovic. Low-complexity energy disag-gregation using appliance load modelling. AIMSEnergy, 4(October 2015):884–905, 2016.

[33] Sean Barker. Model-Driven Analytics of EnergyMeter Data in Smart Homes. PhD thesis, 2014.

[34] Yeqing Li, Zhongxing Peng, Junzhou Huang, ZhilinZhang, and Jae Hyun Son. Energy Disaggregationvia Hierarchical Factorial HMM. In Proceeding ofthe 2014 NILM Workshop, pages 1–4, 2014.

[35] Roberto Bonfigli, Stefano Squartini, Marco Fagiani,and Francesco Piazza. Unsupervised algorithmsfor non-intrusive load monitoring: An up-to-dateoverview. 2015 IEEE 15th International Confer-ence on Environment and Electrical Engineering,EEEIC 2015 - Conference Proceedings, pages 1175–1180, 2015.

[36] Jack Daniel Kelly. Disaggregation of DomesticSmart Meter Energy Data. PhD thesis, Universityof London Imperial College of Science, Technologyand Medicine, 2016.

[37] Ruoxi Jia, Yang Gao, and Costas J Spanos. AFully Unsupervised Nonintrusive Load MonitoringFramework. In Proceeding of the IEEE Interna-tional Conference on Smart Grid Communications(SmartGridComm), pages 872–878, 2015.

[38] Vladimir Stankovic, Jing Liao, and Lina Stankovic.A graph-based signal processing approach for low-rate energy disaggregation. In Computational In-telligence for Engineering Solutions (CIES), 2014IEEE Symposium on, pages 81–87, 2014.

[39] Jack Kelly and William Knottenbelt. Neural NILM:Deep Neural Networks Applied to Energy Dis-aggregation. Proceedings of the 2nd ACM In-ternational Conference on Embedded Systems forEnergy-Efficient Built Environments - BuildSys’15, pages 55–64, 2015.

[40] Pedro Paulo. Applications of Deep Learning tech-niques on NILM. PhD thesis, Universidade Federaldo Rio de Janeiro, 2016.

[41] L.R. R Rabiner and L.R. R Rabiner. Tutorial onHidden Markov Models and Selected Applicationsin Speech Recognition. In Ieee, volume 77, pages257–286, 1989.

[42] Z. Ghahramani and M I Jordan. Factorial HiddenMarkov Models. Machine Learning, 29(2):245–273,1997.

[43] Oliver Parson. Unsupervised Training Methodsfor Non-intrusive Appliance Load Monitoring fromSmart Meter Data. PhD thesis, UNIVERSITY OFSOUTHAMPTON, 2014.

[44] Anders Huss. Hybrid Model Approach to Appli-ance Load Disaggregation. PhD thesis, KTH RoyalInstitute of Technology, 2015.

[45] Jurgen Van Gael and Zoubin Ghahramani.Bayesian Nonparametric Hidden Markov Models.PhD thesis, University of Cambridge, 2011.

[46] A. Sandryhaila and J. M. F. Moura. Big DataAnalysis with Signal Processing on Graphs: Rep-resentation and processing of massive data setswith irregular structure. IEEE Signal ProcessingMagazine, 31(5):80–90, 2014.

[47] Aliaksei Sandryhaila and Jose M F Moura. Classifi-cation via regularization on graphs. In In Proceed-ing of the 2013 IEEE Global Conference on Signaland Information Processing, GlobalSIP 2013 - Pro-ceedings, pages 495–498, 2013.

[48] Ian Goodfellow, Yoshua Bengio, and AaronCourville. Deep Learning (Adaptive Computationand Machine Learning series). MIT Press, 2016.

14

[49] Dong Yu Li Deng. Deep Learning: Methods andApplications. Technical report, may 2014.

[50] Y Taigman, M Yang, and M.A. Ranzato. Deepface:Closing the gap to humal-level performance inface verification. In In Proceeding of the 2014IEEE Conference on Computer Vision and PatternRecognition, pages 1701–1708, 2014.

[51] Dario Amodei, Rishita Anubhai, Eric Battenberg,Case Carl, Jared Casper, Bryan Catanzaro, Jing-dong Chen, Mike Chrzanowski, Adam Coates,Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan,Christopher Fougner, Tony Han, Awni Hannun,Billy Jun, Patrick LeGresley, Libby Lin, SharanNarang, Andrew Ng, Sherjil Ozair, Ryan Prenger,Jonathan Raiman, Sanjeev Satheesh, David Seeta-pun, Shubho Sengupta, Yi Wang, Zhiqian Wang,Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan,and Zhenyao Zhu. Deep-speech 2: End-to-endspeech recognition in English and Mandarin. In InProceedings of the 33rd International Conferenceon Machine Learning, New York, NY, USA, 2016,volume 48, page 28, 2015.

[52] Yonghui Wu And, Mike Schuster And,Zhifeng Chen And, Quoc V. Le And, Mo-hammad Norouzi And, Wolfgang Macherey And,Maxim Krikun And, Yuan Cao And, Qin GaoAnd, Klaus Macherey And, Jeff Klingner And,Apurva Shah And, Melvin Johnson And, Xiaob-ing Liu And, Lukasz Kaiser And, Stephan GouwsAnd, Yoshikiyo Kato And, Taku Kudo And,Hideto Kazawa And, Keith Stevens And,George Kurian And, Nishant Patil And, Wei WangAnd, Cliff Young And, Jason Smith And, Ja-son Riesa And, Alex Rudnick And, Oriol VinyalsAnd, Greg Corrado And, Macduff Hughes And,and Jeffrey Dean. Google’s Neural Machine Trans-lation System: Bridging the Gap between Humanand Machine Translation. CoRR, abs/1609.0,2016.

[53] Chaoyun Zhang, Mingjun Zhong, Zongzuo Wang,Nigel Goddard, and Charles Sutton. Sequence-to-point learning with neural networks for nonintru-sive load monitoring. (2015):1–15, 2016.

[54] Lukas Mauch and Bin Yang. A novel DNN-HMM-based approach for extracting single loads fromaggregate power signals. ICASSP, IEEE Interna-tional Conference on Acoustics, Speech and Sig-nal Processing - Proceedings, 2016-May:2384–2388,2016.

[55] Henning Lange and Mario Berges. The NeuralEnergy Decoder: Energy Disaggregation by Com-bining Binary Subcomponents. In Proceeding ofthe 3rd International Workshop on Non-IntrusiveLoad Monitoring, 2016.

[56] Mingjun Zhong, Nigel Goddard, and Charles Sut-ton. Latent Bayesian melding for integrating indi-vidual and population models. In In Proceedingsof the 28th International Conference on NeuralInformation Processing Systems, page 11, 2015.

[57] Abiodun Iwayemi and Chi Zhou. SARAA : Semi-Supervised Learning for Automated ResidentialAppliance Annotation. IEEE Transactions onSmart Grid, PP(99):1–8, 2015.

[58] Stephen Makonin and Fred Popowich. Nonintru-sive load monitoring (NILM) performance evalu-ation. Springer Energy Efficiency, 8(4):809–814,2014.

[59] Nipun Batra, Haimonti Dutta, and Amarjeet Singh.INDiC: Improved Non-intrusive Load MonitoringUsing Load Division and Calibration. In Proceedingof the 12th International Conference in MachineLearning and Applications (ICMLA), pages 79–84,2013.

[60] Stephen Makonin. Real-Time Embedded Low-Frequency Load Disaggregation by. PhD thesis,SIMON FRASER UNIVERSITY, 2014.

[61] Henning Lange and Mario Berg. Efficient Inferencein Dual-Emission FHMM for Energy Disaggrega-tion. In Proceeding of the Artificial Intelligence forSmart Grids and Smart Buildings, pages 248–254,2015.

[62] J Zico Kolter and Matthew J Johnson. REDD: A Public Data Set for Energy DisaggregationResearch. In Proceedings of the 1st KDD Work-shop on Data Mining Applications in Sustainability(SustKDD, pages 1–6, 2011.

[63] Dario Piga, Andrea Cominola, Matteo Giuliani,Andrea Castelletti, and Andrea Emilio Rizzoli.Sparse Optimization for Automated Energy EndUse Disaggregation. IEEE TRANSACTIONSON CONTROL SYSTEMS TECHNOLOGY on,24(3):1044–1051, 2016.

[64] Nipun Batra, Jack Kelly, Oliver Parson, HaimontiDutta, William Knottenbelt, Alex Rogers, Amar-jeet Singh, and Mani Srivastava. NILMTK : An

15

Open Source Toolkit for Non-intrusive Load Mon-itoring Categories and Subject Descriptors. InProceeding of the International Conference on Fu-ture Energy Systems (ACM e-Energy), 2014.

[65] Jack Kelly, Nipun Batra, Oliver Parson, HaimontiDutta, William Knottenbelt, Alex Rogers, Amar-jeet Singh, and Mani Srivastava. NILMTK V0.2:A Non-intrusive Load Monitoring Toolkit for LargeScale Data Sets: Demo Abstract. In Proceedingsof the 1st ACM Conference on Embedded Systemsfor Energy-Efficient Buildings, BuildSys ’14, pages182–183, New York, NY, USA, 2014. ACM.

[66] Oliver Parson, Grant Fisher, April Hersey, NipunBatra, Jack Kelly, Amarjeet Singh, William Knot-tenbelt, and Alex Rogers. Dataport and NILMTK: A Building Data Set Designed for Non-intrusiveLoad Monitoring. In Proceeding of the GlobalConference on Signal and Information Processing,GlobalSIP, pages 210–2014, Orlando, FL, USA,2015.

[67] Nipun Batra, Amarjeet Singh, and Kamin White-house. Exploring The Value of Energy Disaggre-gation Through Actionable Feedback. In In Pro-ceeding of the 3rd International Workshop on Non-Intrusive Load Monitoring, 2016.

[68] Nipun Batra, Oliver Parson, Mario Berges, Amar-jeet Singh, and Alex Rogers. A comparison of non-intrusive load monitoring methods for commercialand residential buildings. CoRR, abs/1408.6, 2014.

[69] Jack Kelly and William Knottenbelt. The UK-DALE dataset, domestic appliance-level electricitydemand and whole-house demand from five UKhomes. Scientific data, 2:150007, 2015.

[70] Wenjin Jason Li, Xiaoqi Tan, and Danny H KTsang. Smart Home Energy Management SystemsBased on Non-Intrusive Load Monitoring. IEEEInternational Conference on Smart Grid Commu-nications (SmartGridComm): Data Management,Grid Analytics, and Dynamic Pricing, pages 885–890, 2015.

[71] Lucas Pereira and Nuno Jardim Nunes. Towardssystematic performance evaluation of non-intrusiveload monitoring algorithms and systems. In Pro-ceeding of the 2015 Sustainable Internet and ICTfor Sustainability (SustainIT), pages 1–3, 2015.

[72] Mehdi Maasoumy. BERDS - BERkeley EneRgyDisaggregation Data Set. In Proceedings of the

Workshop on Big Learning at the Conference onNeural Information Processing Systems (NIPS),2013.

[73] Kyle Anderson, Adrian Filip Ocneanu, Diego Ben-itez, Derrick Carlson, Anthony Rowe, and MarioBerges. BLUED : A Fully Labeled Public Datasetfor Event-Based Non-Intrusive Load MonitoringResearch. In Proceeding of the 2nd KDD Work-shop on Data Mining Applications in Sustainability(SustKDD), pages 1 – 5, 2012.

[74] S. Barker, A. Mishra, D. Irwin, E. Cecchet,P. Shenoy, and J. Albrecht. Smart*: An OpenData Set and Tools for Enabling Research in Sus-tainable Homes. In Proceeding of the 2012 Work-shop on Data Mining Applications in Sustainability(SustKDD 2012), Beijing, China, 2012.

[75] Akshay Uttama Nambi S N, Antonio Reyes Lua,and R Venkatesha Prasad. LocED : Location-awareEnergy Disaggregation Framework. In In Proceed-ings of the 2Nd ACM International Conferenceon Embedded Systems for Energy-Efficient BuiltEnvironments, Seoul, South Korea, 2015. ACM.

[76] Andreas Reinhardt, Paul Bauman, DanielBurgstahler, Matthias Hollick, Hristo Chonov,Marc Werner, and Ralf Steinmetz. On the Ac-curacy of Appliance Identification Based on Dis-tributed Load Metering Data. In Proceedings ofthe 2nd IFIP Conference on Sustainable Internetand ICT for Sustainability, pages 1–9, 2012.

[77] Stephen Makonin, Bradley Ellert, Ivan V Bajic,and Fred Popowich. Data Descriptor : Electricity ,water , and natural gas consumption of a residentialhouse in Canada from 2012 to 2014. Scientific data,3(160037 —):1–12, 2016.

[78] Vladimir An, David Murray, Lina Stankovic, andVladimir Stankovic. An electrical load measure-ments dataset of United Kingdom households froma two-year longitudinal study. Scientific Data,4463:1–12, 2016.

[79] Andrea Monacchi, Dominik Egarter, Wilfried El-menreich, Salvatore D’Alessandro, and Andrea M.Tonello. GREEND: An energy consumptiondataset of households in Italy and Austria. 2014IEEE International Conference on Smart GridCommunications, SmartGridComm 2014, pages511–516, 2015.

16

[80] Christophe Gisler, Antonio Ridi, and Jean Hen-nebert. APPLIANCE CONSUMPTION SIGNA-TURE DATABASE AND RECOGNITION TESTPROTOCOLS. In Proceeding of the 8th Interna-tional Workshop on SystemsSystems, Signal Pro-cessing and their Applications (WoSSPA), pages336–341, 2013.

[81] Sean Barker, Sandeep Kalra, David Irwin, andPrashant Shenoy. NILM Redux : The Case forEmphasizing Applications over Accuracy. In Pro-ceeding of the 2014 NILM Workshop, 2014.

[82] Nipun Batra, Amarjeet Singh, Pushpendra Singh,Haimonti Dutta, Venkatesh Sarangan, and ManiSrivastava. Data Driven Energy Efficiency in Build-ings. CoRR, abs/1404.7, 2014.

[83] Oliver Parson, Mark Weal, and Alex Rogers. AScalable Non-intrusive Load Monitoring Systemfor Fridge-Freezer Energy Efficiency Estimation.In Proceeding of the 1st International Workshopon Non-Intrusive Load Monitoring (NILM 2014),pages 1–4, 2014.

[84] Mario E. Berges, Ethan Goldman, H. ScottMatthews, and Lucio Soibelman. Enhancing elec-tricity audits in residential buildings with nonintru-sive load monitoring. Journal of Industrial Ecology,14(5):844–858, 2010.

[85] D. Christensen, L. Earle, and B. Sparn. NILM Ap-plications for the Energy-Efficient Home. Techni-cal Report November, National Renewable EnergyLaboratory(NREL), 2012.

[86] Dayu Huang, Marina Thottan, and Frank Feather.Designing customized energy services based on dis-aggregation of heating usage. In Proceeding ofthe 2013 IEEE PES Innovative Smart Grid Tech-nologies Conference (ISGT), pages 1–6. IEEE, feb2013.

[87] Dawei He, Weixuan Lin, Nan Liu, Ronald G.Harley, and Thomas G. Habetler. Incorporatingnon-intrusive load monitoring into building leveldemand response. IEEE Transactions on SmartGrid, 4(4):1870–1877, 2013.

[88] Nipun Batra, Rishi Baijal, Amarjeet Singh, andKamin Whitehouse. How good is good enough?Re-evaluating the bar for energy disaggregation.CoRR, abs/1510.0, 2015.

[89] David Murray, Jing Liao, Lina Stankovic, VladimirStankovic, Charlie Wilson, Michael Coleman, andTom Kane. A data management platform for per-sonalised real-time energy feedback. Eedal, pages1–15, 2015.

[90] Office of Technology Assessment U.S. Congress.Energy in Developing Countries January 1991.OTA-E-486 (Washington, DC: U.S. GovernmentPrinting Office), 1991.

[91] Sunday Olayinka Oyedepo. Efficient Energy Uti-lization as a Tool for Sustainable Developmentin Nigeria. International Journal of Energy andEnvironmental Engineering, 3(1):1–12, 2012.

[92] Nipun Batra, Manoj Gulati, Amarjeet Singh, andMani B Srivastava. It’s Different: Insights intoHome Energy Consumption in India. In Proceed-ings of the Fifth ACM Workshop on EmbeddedSystems For Energy-Efficient Buildings, 2013.

[93] Jack Kelly and William Knottenbelt. Does dis-aggregated electricity feedback reduce domesticelectricity consumption? A systematic review ofthe literature. In Proceeding of the NILM Work-shop 2016, 2016.

[94] Dong Chen, David Irwin, and Prashant Shenoy.SmartSim : A Device-Accurate Smart Home Sim-ulator for Energy Analytics. In In Proceeding ofthe 2016 IEEE International Conference on SmartGrid Communications (SmartGridComm), 2016.

17