Water Security initiative: Customer Complaint Surveillance Vendor Integration Forum Summary

Dallas, TXFebruary 28, 2011

DisclaimerThe Water Security Division of the Office of Ground Water and Drinking Water has reviewed and approved this document for publication. Neither the United States Government nor any of its employees, contractors, or their employees make any warranty, expressed or implies, or assume any legal liability of responsibility for any third party’s use of or the results of such use of any information or process described in this report, or represents that its use by such party would not infringe on privately-owned rights. This document is not a substitute for applicable legal requirements, nor is it a regulation itself. The word “should”, as used in this document, is intended solely to recommend or suggest and does not connote a requirement. Mention of trade names or commercial products does not constitute endorsement or recommendation for use.

Any mention of trade names, companies, products, or services in this guidance does not constitute an endorsement by the U.S. Environmental Protection Agency of any non-federal entity, its products, or its services.

Questions concerning this document or its application should be addressed to:

Brian Pickard U.S. EPA Office of Ground Water and Drinking WaterU.S. EPA Headquarters, Ariel Rios Building1200 Pennsylvania Ave., N.W.Mail Code 4608TWashington, D.C. 20460Pickard.Brian@epamail.epa.gov

or

Nelson MixU.S. EPA Office of Ground Water and Drinking WaterU.S. EPA Headquarters, Ariel Rios Building1200 Pennsylvania Ave., N.W.Mail Code 4608TWashington, D.C. 20460Mix.Nelson@epamail.epa.gov

Table of ContentsSECTION 1.0: OVERVIEW....................................................................................................................................1

SECTION 2.0: CUSTOMER COMPLAINT SURVEILLANCE OVERVIEW AND APPROACH................3

SECTION 3.0: IMPLEMENTING CUSTOMER COMPLAINT SURVEILLANCE.......................................7

SECTION 4.0: VENDOR BENEFITS..................................................................................................................11

SECTION 5.0: UTILITY CASE STUDIES..........................................................................................................13

5.1 CINCINNATI PILOT, GREATER CINCINNATI WATER WORKS (GCWW) 13

5.2 PHILADELPHIA PILOT, PHILADELPHIA WATER DEPARTMENT (PWD) 13

5.3 DALLAS PILOT, DALLAS WATER UTILITIES (DWU) 16

SECTION 6.0: LOOKING FORWARD...............................................................................................................17

SECTION 7.0: REFERENCES9

List of FiguresFIGURE 2-1. CWS MONITORING COMPONENTS....................................................................................................3

FIGURE 2-2. ARCHITECTURE OF A CWS...............................................................................................................4

FIGURE 2-3. WS INITIATIVE PROGRAM OVERVIEW..............................................................................................5

FIGURE 3-1. FUNNEL, FILTER, FOCUS...................................................................................................................7

FIGURE 3-2. AET OUTPUT....................................................................................................................................8

FIGURE 3-3. CINCINNATI HYDRA MAP DISPLAY..................................................................................................9

FIGURE 3-4. CINCINNATI PILOT EXAMPLE ALERT NOTIFICATION......................................................................10

FIGURE 5-1. PWD SYSTEM ARCHITECTURE........................................................................................................14

FIGURE 5-2. EXAMPLE OF PWD EDA MATRIX..................................................................................................14

FIGURE 5-3. PWD DASHBOARD..........................................................................................................................15

FIGURE 5-4. DWU CCS ARCHITECTURE............................................................................................................16

List of Acronyms

AET Alarm Estimation ToolAWWA American Water Works AssociationCCSCSC

Customer Complaint SurveillanceComputer Sciences Corporation

CS/IMTech Customer Service/Information Management TechnologyCSR Customer Service RepresentativeCUSUM Cumulative SumCWS Contamination Warning SystemDWUEDA

Dallas Water UtilitiesEvent Detection Algorithm

EDS Event Detection SystemEPA U.S. Environmental Protection AgencyGCWW Greater Cincinnati Water WorksGIS Geographic Information SystemIT Information TechnologyIVR Interactive Voice ResponsePWD Philadelphia Water DepartmentSCADA Supervisory Control Automated Data AcquisitionSME Subject Matter ExpertSMS Short Message ServiceWS initiative Water Security initiative

vii

Section 1.0: OverviewThe Water Security (WS) initiative is a U.S. Environmental Protection Agency (EPA) program that addresses the risk of intentional contamination of drinking water distribution systems. Initiated in response to Homeland Security Presidential Directive 9, the overall goal is to establish recommendations for the design and deployment of contamination warning systems for voluntary adoption by drinking water utilities.

The EPA Customer Complaint Surveillance (CCS) Vendor Integration Forum was held on Sunday, February 28, 2011 from 12:00–3:00 p.m. at the Intercontinental Hotel in Dallas, Texas. The Forum was jointly hosted by EPA’s Office of Ground Water and Drinking Water and the American Water Works Association (AWWA) in conjunction with the AWWA 2011 Customer Service/Information Management Technology (CS/IMTech) conference. The purpose of this Forum was to meet with Vendors and Integrators to speak with them regarding the WS initiative and CCS and to create an open dialogue for product development.

The Forum was broken into two sessions. The first session provided an overview of the WS initiative and the CCS component, including aspects of implementing a CCS system. The second session consisted of presentations by three WS initiative Pilot Utility subject matter experts (SMEs), including a Question & Answer session. The report below outlines the proceedings of this forum.

1

Section 2.0: Customer Complaint Surveillance Overview and Approach

After a brief introduction from Alan Roberson of AWWA, Brian Pickard, Team Leader, EPA Office of Ground Water and Drinking Water, introduced the concept and reason behind the development of the WS initiative, the nature of contamination threats, and the Contamination Warning System (CWS).

Mr. Pickard noted that CCS is important due to distribution system vulnerabilities, including accessibility via commercial and residential service connections, fire hydrants, and finished water storage areas. He also noted it is impossible to eliminate all access to a system, but key system components can be hardened. Mr. Pickard stated that consequences of contamination include adverse impacts on public health, loss of water for public safety uses, economic damages, and loss of consumer confidence. He also stated that an attack using contaminants is likely to achieve multiple terror objectives. Such an attack would not necessarily have to produce casualties in order to be considered successful by an organization, reiterating the importance of having a CWS in place.

CWS design objectives include:

Detecting a broad spectrum of contaminant classes Achieving spatial coverage of the entire distribution system Detecting contamination in sufficient time for effective response Reliably indicating a contamination incident with a minimum number of false-positives Providing a sustainable architecture to monitor distribution system water quality Emphasizing dual-use aspects

A CWS involves the active deployment and use of monitoring technologies and strategies and enhanced surveillance activities. These activities collect, integrate, analyze, and communicate information in order to provide a timely warning of potential water contamination incidents and to initiate response actions to minimize public health and economic impacts. An integrated CWS will utilize water quality monitoring, public health surveillance, sampling and analysis, enhanced security monitoring, and CCS. While CWS requires this multi-pronged approach, as shown in Figure 2-1, CCS is the main focus of the Vendor Integration Forum.

Figure 2-1. CWS Monitoring Components

3

A CWS is not merely a collection of monitors and equipment placed throughout a water system to alert of intrusion or contamination. A CWS includes different information streams that are captured, managed, analyzed, and interpreted to recognize potential contamination incidents, with time to respond effectively. These data sources, when used concurrently, should support and augment each other such that the chances of detecting a contamination incident are better than using any one information source on its own. While a CWS will ultimately be designed by the utility, cooperation with partners, including vendors and integrators, is vital to the successful adoption of the CWS. CWS operations can be broken down into two phases: Phase I is Routine Monitoring and Surveillance; Phase II is Consequence Management and is implemented only when there has been a possible determination of contamination. Figure 2-2 represents this architecture.

Figure 2-2. Architecture of a CWS

Concerning Voluntary National Adoption of CWS, the WS initiative comprises work in three areas, or phases:

Develop a conceptual design for a system that achieves timely detection and appropriate response to drinking water contamination incidents to minimize public health and economic impacts;

Demonstrate , test, and evaluate the conceptual design in CWS Pilots at drinking water utilities;

Issue practical guidance and conduct outreach to promote voluntary national adoption of effective and sustainable drinking water CWSs.

Figure 2-3 represents these three areas.

4

Figure 2-3. WS initiative Program Overview

EPA is now implementing an “inside-out” approach to utility adoption of CWSs. EPA is urging vendors and integrators to implement this approach by including the required coding into established call management and work management software programs and systems. This approach allows for a more efficient approach to CCS adoption. The necessary mechanisms to track, transfer and analyze CCS data are already built into many existing software systems. If utilities realize that they have this capability to “plug-and-play”, it is more likely that they will adopt many, if not all, aspects of the CCS component.

5

Section 3.0: Implementing Customer Complaint Surveillance

This portion of the presentation was guided by the CSC team: Scott Weinfeld, a CSC Project Manager, Adam Haas, Lead Analyst, and Doron Shalvi, Senior Application Developer and Technical Lead for the Cincinnati Pilot.

Mr. Weinfeld began by giving a concise description of the features and characteristics of an ideal CCS. According to EPA, a “CCS encompasses the customer complaint collection process, data management, data analysis and anomaly detection of customer complaints, notification of anomalies, and investigation procedures.” It is important to keep this definition in mind when developing new CCS systems and making modifications to existing applications. Mr. Weinfeld also spoke regarding “Funnel, Filter, and Focus”. Collecting complaints into a single location, or funneling them by using Interactive Voice Response (IVR) and Asset Management Systems (including email, phone calls, and web forms) is a best practice and AWWA standard. In filtering calls, he noted the importance to recognize and filter out non-significant calls, such as those relating to rusty or dirty water, cloudy water, or water pressure. The final step is to “focus”: notify the appropriate utility personnel of detected anomalies. Figure 3-1 represents this concept.

Figure 3-1. Funnel, Filter, Focus

Mr. Weinfeld described the characteristics of contamination that can be detected by CCS. A CCS system is looking for anomalous patterns, aberrant signal-to-noise ratios, or deviations from the norm in either time or space. These contamination events involve contaminants with aesthetic properties not associated with a benign cause, and not associated with common utility water quality problems that may have similar complaint descriptions, such as rusty or dirty water, cloudy water, or water pressure. Water

7

contamination will also display temporal clustering within minutes or hours of each other, and spatial clustering, both linked by characteristics of the distribution system . Implementing a CCS system provides the possible dual-use benefit of detecting anomalies in common water quality problems in addition to those associated with possible contamination incidents.

The core functionalities that comprise a CCS component must include at a minimum:

A mechanism for a utility to baseline their data and establish alarm thresholds, Near real-time analysis of data using automated surveillance algorithms and code, Alert notification of anomalies, and most importantly, and Easy integration with other utility systems.

This last point is especially important. EPA doesn’t want to re-invent the way utilities do business, but rather integrate CCS into their existing business model. EPA thinks the best people to talk to drinking water utility personnel are the vendor/integrator community who are deeply embedded in the existing systems and processes already in place at these utilities.

Mr. Haas discussed creating a baseline for which utilities can establish a threshold. Baselining the data involves reviewing and analyzing historic data, and identifying “normal” complaint volume and setting thresholds for alarms. Mr. Haas spoke regarding how this process was completed in Cincinnati, and about the Alarm Estimation Tool (AET). The AET is a simple tool created by EPA that assists developers, integrators, and utilities in analyzing their historic water quality data and establishing thresholds. Figure 3-2 shows an example output from the tool. The tool is free, and is available on EPA’s website at: http://water.epa.gov/infrastructure/watersecurity/techtools/index.cfm.

Figure 3-2. AET Output

8

Once the utility has established their thresholds, they can build their Event Detection System (EDS). Multiple automated algorithms running in parallel executed in near real-time can detect both temporal and spatial clustering of water quality complaints. Once the utility has detected the anomalous water quality complaints, they must be able to provide actionable notifications to appropriate personnel, and provide procedure for closing out alarms. In implementing CCS at the Cincinnati Pilot, several activities occurred:

Historical data was analyzed and reviewed, Normal and anomalous complaint volumes were established, and Data resources were identified, such as:

o Contact management systems (IVR, Customer Service Representative (CSR) characterization, and work/asset management system),

o Algorithm code and servers, together comprising the EDS, o Notification system (e-mail and text messaging), and o Distribution system GIS.

Types of alarms were defined by analysis algorithms and data stream at the Cincinnati Pilot. The algorithm code (Java) was designed to detect temporal anomalies at varying thresholds utilizing 1, 2, and 7 day scans and Cumulative sum (CUSUM) and spatial anomalies by monitoring neighborhoods, zip codes, pressure zones, and other hydraulically significant areas. Spatial analysis, using Hydra MAP Geographic Information System (GIS) application, is shown in Figure 3-3. It is important that all of the different information available to the utility be displayed, for example as a dashboard.

Figure 3-3. Cincinnati Hydra Map Display

Mr. Shalvi spoke regarding notifications and notification content, investigation procedures, and information technology (IT) considerations. He stated that immediate alert notifications must be provided to appropriate personnel in near real-time. Possible communication channels are short message service (SMS), email, auto-dialers, and interface pop-ups achieved through Supervisory Control Automated Data

9

Acquisition (SCADA). It is also important that the notification contains actionable information such as details as to the type of alert, alert location, and investigation procedure instructions. At the Cincinnati Pilot, CCS alarm notifications are received by the Water Quality department in email format. The notification includes the algorithm that detected the anomaly, the dates, times, and locations of the complaints and annotated information regarding each call that led to the alert. Figure 3-4 shows an example of a notification received.

Figure 3-4. Cincinnati Pilot Example Alert Notification

Typically included on the e-mail (but not shown above) are the date, time, location and description of each complaint to aid the investigation of the alarm. In Cincinnati, theses details were later added to the e-mails, at the request of the investigating utility staff.

Investigative procedure may include:

Interviewing callers; Reviewing call data and analyzing the location and nature of the complaint(s) to

determine if calls are clustered; Reviewing plant and distribution system operations and assessing the potential for

operational changes to impact aesthetics of water; Reviewing distribution system work: Breaks and repairs may impact color, turbidity,

and odor of the water; Reviewing and analyzing available water quality data in the vicinity of the call cluster

to determine if there is a link; Closing out the alert.

When integrating a CCS, there are many considerations to take into account. First, it is important to design a flexible architecture using a standardized approach and ensuring that parameters, such as notification templates, notification recipients, and thresholds, are configurable by managers. Second, it is important to leverage existing systems and capabilities, including existing applications, the network infrastructure, programming expertise, and security. Thirdly, it is important that alarm data is stored so that the utility has the ability to go back and perform a retrospective analysis. And finally, it is important to account for any time delay in receiving alerts.

10

Section 4.0: Vendor BenefitsEPA wants to enlist vendors in a novel approach to providing CCS functionality to the water sector. EPA is taking an approach in which AWWA and EPA drive demand by encouraging voluntary adoption of CCS into their members’ operations while vendors and integrators supply the products and services that make them easy to adopt. EPA & AWWA will create the demand; vendors and integrators create the supply. Benefits are not only to the end users, AWWA and EPA, but also to the tool developers and system integrators.

There are multiple documented cases in which customer complaints have been an early indicator contamination incidents. In Milwaukee, Wisconsin, a telephone survey of 613 households provided estimates on the total number of persons in Milwaukee experiencing mild, moderate, or severe illness as a result of the cryptosporidiosis outbreak in 1993. When disease case estimates were adjusted for normal background diarrheal disease rates, investigators estimated that 403,000 residents of the five-county area experienced illness caused by the cryptosporidiosis outbreak. A high volume of customer calls could have indicated problems with the distribution system, but there was no system in place at the time to alert supervisors. Utilities will be looking for this functionality. An early warning saves a utility time and money, and has the potential to save lives. A better knowledge of water quality complaint prevalence also provides a dual-use benefit in that more knowledge can help improve normal operations. Off-the-shelf solutions offer low implementation cost for utilities, and cost is key consideration when utilities are upgrading or replacing a call or work management system.

As mentioned, AWWA and EPA are encouraging voluntary adoption of CCS, therefore creating demand. In addition to this, there is a growing body of research and guidance. EPA has invested millions of dollars into WS initiative Pilot CCS programs. Vendors and integrators stand to benefit from existing pilot implementations in Cincinnati, Dallas, New York City, Philadelphia, and San Francisco. In addition there is a dual-use, all hazards emphasis on improvements in monitoring and security, with updated federal guidelines and recommendations. EPA has already issued guidance regarding the benefits of implementing CCS, and AWWA and EPA have published articles on CCS implementation and benefits (USEPA 2005). AWWA and EPA are committed to promoting voluntary adoption over the long term.

For tool developers, CCS functionality provides a more appealing product offering with market differentiation, easy implementation (most features are already in existing tools), and easier implementation for system integrators. For system integrators, CCS functionality provides consistent implementations, efficiency of off-the-shelf solutions, and low implementation cost for the utility. Adding CCS functionality to call and work management products will allow vendors to have their product stand out from other products in the marketplace. Finally, products and services may already have the desired features – but just need to be “re-visioned” with an eye to providing a CCS component.

11

12

Section 5.0: Utility Case Studies

5.1 Cincinnati Pilot, Greater Cincinnati Water Works (GCWW)

The presentation began explaining how CCS implementation affected GCWW Operations. Mr. May stated that historically, operational staff had to be constantly aware and tuned-in to notice data trends, but now the capability to detect exists in a standardized system and there is a standard process for investigation. A standardized system helps when there is employee turnover. The general benefits to implementing the CCS are that the service-oriented architecture is accepted by GCWW, and GCWW was able to “re-use” the web service. GCWW was Generation 1, or an ‘elementary algorithm’, but now CCS is ingrained within the utility. However, if GCWW did it again, they would require more spatial intelligence, and development of the platform in .net instead of Java. GCWW is also interested in geographical RSS feeds for GIS.

Responding to a question regarding the GIS interface, Mr. May stated that work orders will automatically show up on their GIS Hydra Map display.

5.2 Philadelphia Pilot, Philadelphia Water Department (PWD)

Charles Zitomer, PWD, spoke regarding the approach taken at PWD, in implementing CCS, CCS and EDAs, and the PWD CWS dashboard. In approaching implementation of CCS, PWD opted to leverage existing resources such as their GIS investment, skilled workforce, existing data systems, and business procedures. PWD decided to improve and streamline customer response, conduct field investigations more efficiently, and integrate new systems. A new work order management system (Cityworks), and a CWS Dashboard, both of which are Web-based applications, allow remote access to assist with CCS investigations.

In implementing CCS, PWD is using Cityworks as the work order management system. It is a Web-based application with an open architecture database and a spatially compliant data structure that allows remote access. It was configured to maximize its benefits for the CWS. All water quality service requests are funneled through the Call Center and captured in Cityworks. All work activities on the distribution system will be captured in Cityworks, which will help with associating water quality alarms with potentially related distribution system activities. Figure 5-1 represents the system architecture in place at PWD.

13

Figure 5-1. PWD System Architecture

Use of EDAs allows PWD to get information to decision makers quickly, twenty four hours a day, seven days a week. There is an emphasis on GIS and attribute data. EDAs are used to continuously process water quality complaint information and determine whether a pre-defined complaint threshold has been exceeded. An EDA’s purpose is to bring a potential water quality event to the attention of a human investigator using real-time customer complaint information. At PWD, EDA thresholds were developed based on water quality complaint type(s), spatial location or clustering, leveraging GIS, and date and time. An example of an EDA, using 1-, 2-, and 7-Day Scan Windows, and monitoring multiple water quality parameters, is shown in Figure 5-2.

Figure 5-2. Example of PWD EDA Matrix

Currently under development, the CWS Dashboard and the EDAs will assist the investigator by automatically displaying only the relevant service request and work order types from the Cityworks data and alerting them to activity occurring within a specific radius of the complaint(s). The CWS Dashboard and the EDAs will also reduce the investigation and response time by promptly providing relevant

14

information during alarm conditions. The business processes for the PWD Dashboard are yet to be implemented. There are currently issues with night crews that are ‘not so much into water quality’ and can now receive training. The PWD Dashboard has the dual-use ability to show planned capital improvement projects, such as line replacement and flushing, as well as customer complaints. An example of the PWD dashboard is shown in Figure 5-3.

Figure 5-3. PWD Dashboard

Ultimately, PWD’s approach to leveraging existing GIS and Cityworks will allow the department to capture all water quality complaint calls and work performed in the distribution system on the same map. It will also allow PWD Operations to detect system issues and failures, improve customer service, and provide PWD with the opportunity to standardize business processes and protocols for water quality complaint investigation.

Responding to a question regarding CSR access to the Dashboard, Mr. Zitomer stated that CSR access to the Dashboard is limited to because there is a concern that CSRs may give too much information back to the customer about other complaints. Information regarding things such as main breaks may be made available to the CSR.

Responding to a question regarding map information being available to the laboratory, Mr. Zitomer stated that the laboratory would have a history of all the calls which would help with the issuance of notifications occurring sooner.

Responding to a question regarding the algorithm, Mr. Zitomer stated that the work order system provides customer complaint information which is then filtered based on type and is stored in the “warehouse” until it is later, when an alert is issued and it is analyzed.

15

Responding to a question regarding time of occurrence of complaints and other issues, Mr. Zitomer stated that all necessary information, including time, location, and nature of complaint, will be available on the map. If a complaint was received before a main break, that information would be available to the user.

Responding to a question regarding resolution information, Mr. Zitomer stated that lab results from LIMS are entered back into the service request in Cityworks for the complaint. There is a distinction between 311 call centers and an in-house call center.

5.3 Dallas Pilot, Dallas Water Utilities (DWU)

Cassia Sanchez, DWU, discussed DWU’s implementation of CCS. DWU is in the process of implementing SAP and changing from one water quality category to approximately ten. Implementation of this system should be completed by June 2011. DWU has shallow reservoirs. Because of this, DWU switched to ozone for treatment in 2005 and eliminated the use of carbon for treatment of taste and odor characteristics. This resulted in a significant reduction in the number of customer complaints received. DWU is educating the 311 center CSRs, making changes in organization management, as well as with water purification processes. For DWU, education coupled with system improvements represents a dual-use benefit. Figure 5-4 illustrates the CCS architecture currently in place at DWU.

Figure 5-4. DWU CCS Architecture

Responding to a question regarding system operation, Ms. Sanchez stated that the system has been developed over the past year and was not fully up and running, though it would be integrated June of 2011.

Responding to a question regarding the characteristics that would be analyzed, Ms. Sanchez stated that DWU would be analyzing complaints based on appearance, taste and odor, color, and other parameters.

Responding to a question regarding algorithm refinement, Ms. Sanchez stated that all algorithms would be reviewed this summer, following full deployment of the system.

16

Section 6.0: Looking ForwardAlan Roberson, AWWA, spoke to future collaborations, stating that looking forward there will be collaboration with AWWA, EPA, utilities, and the private sector, with input from stakeholders and their ideas. AWWA and EPA hope to start information sharing, and hold follow-up webinars in the future. As WS initiative Expansion Pilot activities near completion, there will be additional information, lessons learned and guidance generated from CCS efforts at DWU, PWD, the New York City Department of Environmental Protection, and the San Francisco Public Utilities Commission. Finally, EPA and AWWA believe that successful integration of CCS software by vendors will require a two-way conversation, and so plan to start a dialogue with these private sector partners.

17

18

Section 7.0: ReferencesU.S. Environmental Protection Agency. 2005. WaterSentinel System Architecture, EPA 817-D-05-003.

19