election systems and sotware unity 3.2.0...dec 20, 2011 · developmental testing and quality...

U.S. Election Assistance Commission

Election Systems and Software Unity 3.2.0.0

December 20, 2011

Voting System Testing and Certification Division

1201 New York Ave, NW, Suite 300

Washington, DC 20005

www.eac.gov

Formal Investigation Report

EAC Certification Number: ESSUNITY3200

Contents Executive Summary.................................................................................................iii

1. Introduction ...........................................................................................................1

2. Scope of Formal Investigation ..............................................................................1

2.1 Freeze/shutdown .......................................................................................................... 3

2.2 Failure to log ................................................................................................................ 3

2.3 Ballot Skew....................................................................................................................4

2.4 Vote miscount ...............................................................................................................4

3. Investigative process ............................................................................................4

4. Relevant facts & timeline...................................................................................... 5

5. Analysis .................................................................................................................8

5.1 Analysis of freeze/shutdown .........................................................................................8

5.2 Analysis of failure to log.............................................................................................. 10

5.3 Analysis of ballot skew .................................................................................................11

5.4 Analysis of vote miscount ............................................................................................11

6. Findings .............................................................................................................. 12

6.1 Freeze/shutdown..........................................................................................................12

6.2 Failure to log ................................................................................................................14

6.3 Ballot skew ...................................................................................................................14

6.4 Vote miscount ..............................................................................................................16

7. Recommendation.................................................................................................17

Appendix A - ES&S Technical Bulletin PRBD2000006

Appendix B - Cleveland Plain Dealer

Appendix C - ES&S Guidence to Cuyahoga County (Freeze)

Appendix D - EAC Technical Advisory ESS2010-01

Appendix E - ES&S Technical Bulletin PRBD2000008

Appendix F - Recommendation to Refer for Formal Inquiry

Appendix G - Addendum Recommendation to Refer for Formal Inquiry

Appendix H - ES&S Technical Bulletin PRBDS2000010

i

Appendix I - ES&S Technical Bulletin FYIDS2000019

Appendix J - Authorization for Formal Investigation

Appendix K - EAC Technical Advisory ESS2011-01

Appendix L - EAC Report on Cuyahoga 3.2.1.0 Upgrade

Appendix M - EAC Technical Advisory ESS2011-02

Appendix N - EAC Technical Advisory ESS2011-03

Appendix O - ES&S Technical Bulletin FYIDS2000021

Appendix P - ES&S Technical Bulletin PRBDS2000013

Appendix Q - DS200 Lockup Analysis

Appendix R - Notice of Formal Investigation

Appendix S - Interrogatory: iBeta Quality Assurance

Appendix T - Interrogatory: Wyle Laboratories

Appendix U - Interrogatory: Cuyahoga County

Appendix V - Interrogatory: ES&S

Appendix W - Expectations for Freeze Shutdown Testing

Appendix X - iBeta VSTL Withdrawal Letter

Appendix Y - Approval to move Unity 3.2.1.0 to Wyle

Appendix Z - ES&S Response to Investigation

Appendix AA - Cuyahoga 3.2.1.0 Upgrade Checklist

ii

Executive Summary

Overview

This Formal Investigation Report summarizes the Scope, Analysis, Findings andRecommendations of the EAC investigation into the Election Systems & Software(ES&S) DS200 Precinct Count Optical Scanner in the Unity 3.2.0.0 EAC certifiedvoting system. The EAC acknowledges and appreciates the level of cooperationreceived from ES&S during the course of this investigation.

The investigation was initiated by the EAC as a result of information contained in an article in The Cleveland Plain Dealer published on April 14, 2010, about a freeze/ shutdown issue experienced in Cuyahoga County, Ohio during pre-election testing.

Athough participation in the EAC certification program is voluntary for States,adherence to the program’s procedural requirements is mandatory for allparticipating voting system manufacturers.

Substantiated Anomalies

The EAC deems three of the four allegations of anomalies described in thisinvestigation to be substantiated. The substantiated anomalies are:

₋ Intermittent screen freezes, system lockups and shutdowns that prevent the voting system from operating in the manner in which it was designed.

₋ Failure to log all normal and abnormal voting system events.

₋ Skewing of the ballot resulting in a negative effect on system accuracy.

The cure to these anomalies proposed by ES&S in the form of the modifiedUnity 3.2.0.0 Rev. 2 voting system, does not correct all anomalies cited in thisinvestigation. These anomalies remain in the EAC Certified Unity 3.2.0.0 votingsystem in use today. In addition, the ES&S proposed fixes created additionalanomalies discovered in the EAC certified Unity 3.2.1.0 voting system.

Recommendations

Based on this finding, the EAC recommends the following:

1. That the EAC issue a Notice of Non-Compliance for the substantiated allegations in the EAC certified Unity 3.2.0.0 system. The Notice will inform the Manufacturer of the next steps in the process, including the Manufacturer’s opportunity to cure non-compliance and have an opportunity to be heard prior to any final decision on decertification.

Formal Investigation - ES&S Unity 3.2.0.0 www.eac.gov

iii

http:www.eac.gov

2. That a Manufacturer Site Audit be conducted by the EAC to evaluate the developmental testing and quality assurance practices of ES&S. The EAC has concerns about the Quality Assurance practices of the manufacturer based on the recurrence of certain anomalies and the fact that fixes provided for certain anomalies created additional issues with the DS200.

The Process

Because of the potential adverse impact on voting system manufacturers, election officials and the public, the EAC process for possible decertification actions is complex. The investigation process is initiated when the EAC receives information that a voting system may not be in compliance with the applicable voting system standard or procedural requirements of the EAC certification program. Upon receipt of such information, the EAC initiates an Informal Inquiry to determine the credibility of the information. If the information is credible and suggests the system is non-compliant, a Formal Investigation is initiated. If the results of the Formal Investigation demonstrate non-compliance, the Manufacturer is provided a Notice of Non-Compliance. Before a final decision on decertification is made, the Manufacturer will have the opportunity to remedy any defects identified in the voting system and present any additional information for consideration by the EAC.


iv

http:www.eac.gov

Formal Investigation - ES&S Unity 3.2.0.0 1 www.eac.gov

1. Introduction

This report summarizes the Scope, Analysis, Findings and Recommendations of the Formal Investigation into the Election Systems & Software (ES&S) DS200 Precinct Count Optical Scanner in the Unity 3.2.0.0 EAC certified voting system.

The Cleveland Plain Dealer (“Appendix B - Cleveland Plain Dealer”) published an article on April 14, 2010, about a freeze/shutdown issue experienced in Cuyahoga County, OH, during pre-election logic & accuracy testing. The EAC followed up with Cuyahoga County and ES&S about the anomaly. ES&S provided the EAC with a “DS200 Lockup Analysis” (“Appendix Q - DS200 Lockup Analysis”) on June 28, 2010. The EAC analyzed this report and responded with further questions. The EAC contacted all known jurisdictions using Unity 3.2.0.0 and the DS200 to gather information about their experiences with the system.

On October 15, 2010, the Testing and Certification Program Director submitted a “Recommendation to Refer for Formal Inquiry,” (“Appendix F - Recommendation to Refer for Formal Inquiry”) and submitted an addendum (“Appendix G -Addendum Recommendation to Refer for Formal Inquiry”) to that recommendation in December 2010. On February 25, 2011, the EAC Executive Director, issued an “Authorization for Formal Investigation” (“Appendix J - Authorization for Formal Investigation”)1. The investigation focused on these anomalies: freeze/shutdown, ballot skew, failure to log and vote miscount. The EAC issued a “Notice of Formal Investigation” (“Appendix R - Notice of Formal Investigation”) to ES&S on March 1, 2011.

As noted in the diagram below, there have been a number of modifications to the EAC certified Unity 3.2.0.0 system. Most of these modifications attempt to address the anomalies reported in this investigation or were discovered during testing of the cure for the anomalies listed in this investigation. Testing for the Unity 3.2.0.0 cure was conducted during the Unity 3.2.1.0 test campaign. The DS200 in 3.2.1.0 shares a large amount of source code with Unity 3.2.0.0, which means that anomalies found in Unity 3.2.1.0 may be present in the Unity 3.2.0.0 certified DS200.

2. Scope of Formal Investigation

The focus of the Investigation was the ES&S DS200 Precinct Count Optical Scanner (firmware version 1.3.10.0) contained in the ES&S Unity 3.2.0.0 EAC certified voting system. The scope of the Investigation was to determine if the freeze/shutdown anomaly first experienced in Cuyahoga County, Ohio during pre-election logic and accuracy testing in preparation for the May 4, 2010 Primary Election rendered the system non-conformant to the 2002 Voting Systems Standards (VSS). In addition to the freeze/shutdown anomaly, the EAC’s Informal Inquiry revealed additional anomalies with the DS200, including issues related to ballot skew, ballot insertion,

1 EAC Executive Director Thomas Wilkey retired effective November 30, 2011. Under the provisions of HAVA, the EAC General Counsel becomes acting Executive Director until the Commission selects a permanent replacement.


and unlogged errors. Further, the potentially more serious issue noted in the December 20, 2010 addendum which outlined the problem encountered when the DS200 accepted a voted test ballot without recording that ballot on its internal

ES&S Unity 3.2.0.0 Voting System Flowchart: This flowchart is a graphical representation of the relationships between the Unity 3.2.0.0 voting

system and its subsequent modifications described in this Report


counter during testing at iBeta Quality Assurance, was investigated for potential non-compliance with the 2002 VSS.

2.1 Freeze/shutdown At random intervals, the DS200 initiates the shutdown process whereupon it will complete approximately 90% of the shutdown process and then freeze. The DS200 Unit will not accept ballots in the frozen state.

The Freeze/shutdown issues indicates a system non-conformity to the 2002 VSS Volume 1 Section 2.2.1 & 3.4.3:

2.2.1.b – Provide system functions that are executable only in the intended manner and order, and only under the intended conditions.

3.4.3 – Reliability: The reliability of voting system devices shall be measured as Mean Time Between Failure (MTBF) for the system submitted for testing. MBTF is defined as the value of the ratio of operating time to the number of failures which have occurred in the specified time interval. A typical system operations scenario consist of approx. 45 hours of equipment operation, consisting of 30 hours of equipment set-up and readiness testing and 15 hours of elections operations. For the purpose of demonstrating compliance with this requirement, a failure is defined as any event which results in either the:

₋ Loss of one or more functions; or

₋ Degradation of performance such that the device is unable to perform its intended function for longer than 10 seconds

The MTBF demonstrated during certification testing shall be at least 163 hours.

2.2 Failure to log Cuyahoga County election officials provided the EAC with DS200 system logs from their May 4 and August 8, 2010 elections. The May 4 election used over 1,000 DS200s; the August 8 election was smaller, providing logs from only 12 machines. Review of these system logs identified an additional issue. The freeze/shutdown event did not result in any record of its occurrence in the system logs. In addition, records of numerous other normal and abnormal events were absent from the system logs.

The failure to log issue indicates a system non-conformity to the 2002 VSS Volume I 2.2.4 Integrity:

Integrity measures ensure the physical stability and function of the vote recording and counting processes.

2.2.4.1 - Common Standards

To ensure system integrity, all systems shall:


g. Record and report the date and time of normal and abnormal events.

i. Detect and record every event, including the occurrence of an error condition that the system cannot overcome, and time-dependent or programmed events that occur without the intervention of the voter or a polling place operator.

2.3 Ballot Skew2

When a 17” ballot is inserted incorrectly into the unit the lower left and right hand corners of the ballot are not accurately read.

The failure to accurately read all valid votes on a ballot indicates a system non-conformity to the 2002 VSS Volume I 2.2.2.1 Common Standards:

To ensure vote accuracy, all systems shall:

c. Record each vote precisely as indicated by the voter and be able to produce an accurate report of all votes cast.

2.4 Vote miscount The DS200 accepts a voted ballot but does not record the ballot on its internal counter. In addition the marks of the second ballot are not recorded.

The inaccurate vote count indicates a system non-conformity to the 2002 VSS Volume I 2.2.9 Ballot Counter:

For all voting systems, each device that tabulates ballots shall provide a counter that:

b. Records the number of ballots cast during a particular test cycle or election.

3. Investigative process

On March 1, 2011, the EAC Executive Director issued a Notice of FormalInvestigation for the ES&S Unity 3.2.0.0 voting system. Section 7.4 of the EACVoting System Testing and Certification Program Manual (Manual) outlines theprocess for conducting a formal investigation.

During the course of the investigation, EAC staff created a record of all relevant documents and information, including: initial reports about the anomalies; ES&S Lockup Analysis and documentation regarding proposed fixes to the system; ES&S Technical Advisories; EAC Product Advisories; and system documentation. The EAC created this record from documents and information received during both the Informal and Formal investigation process.

Section 7.4.5.5.4 of the Manual permits the EAC to develop written requests forinformation (referred to as interrogatories in the Manual). The EAC developed

2 As used in this report. “Ballot Skew” refers to the ballot paper being placed into the scanning feed mechanism of the DS200 not perfectly parallel to the DS200 paper guide.


several sets of interrogatories and sent them (March 30, 2011) to iBeta Quality Assurance (“Appendix S - Interrogatory: iBeta Quality Assurance”); Wyle Laboratories (“Appendix T - Interrogatory: Wyle Laboratories”); Director of Elections, Cuyahoga County, OH (“Appendix U - Interrogatory: Cuyahoga County”); and ES&S (“Appendix V - Interrogatory: ES&S”). The EAC received and reviewed the responses and documentation for these interrogatories. The information gathered through this process is part of the record of this investigation.

On April 15, 2011, ES&S received EAC approval to begin testing a modification, (Unity 3.2.0.0, Rev. 2), to cure the issues cited in the Notice of Formal Investigation. This modification integrated firmware changes, which are part of the certified Unity 3.2.1.0 system, with the Unity 3.2.0.0.

Section 7.4.5.5.2 of the Manual permits the EAC to conduct field audits of EACcertified systems. An EAC Computer Engineer traveled to Cuyahoga County toobserve the County’s upgrade to the Unity 3.2.1.0 system. While observing theupgrade installation, the EAC witnessed anomalies similar to those reported in theinformal and formal investigation for Unity 3.2.0.0 (“Appendix L - EAC Report onCuyahoga 3.2.1.0 Upgrade”).

4. Relevant facts & timeline

₋ On July 21, 2009, ES&S Unity 3.2.0.0 received EAC certification (EAC certification number: ESSUnity3200). This system contained the DS200, M650, and AutoMARK.

₋ On October 20, 2009, ES&S released Technical Bulletin PRBD2000006 (DS200 Registers Unmarked Ovals as Marks on Skewed 17 and 19 inch Ballots) to certain DS200 users.

₋ On April 14, 2010, The Cleveland Plain Dealer published an article about the freeze/shutdown issue experienced in Cuyahoga County, OH, during Logic & Accuracy Testing.

₋ On April 19, 2010, ES&S created an initial hardware analysis describing tests run on machines and initial findings on the cause of the freeze/shutdown issue.

₋ On April 23, 2010, ES&S sent guidance to Cuyahoga County on recommended procedures to handle the freeze/shutdown anomaly on Election Day (May 4, 2010).

₋ On June 22,2010, ES&S Unity 3.2.0.0 Rev.1 application for modification of Unity 3.2.0.0 was approved. This modification was intended to fix the ballot skew and allow for multiple upload stations for the DS200.

₋ On June 25, 2010, the EAC released a Technical Advisory titled “Intermittent Freeze/Shutdowns with EAC Certified ES&S Unity 3.2.0.0 System .”


₋ On June 28, 2010, ES&S provided the EAC with a “DS200 Lockup Analysis.”

₋ On July 7, 2010, ES&S released Technical Bulletin PRBDS2000008 (DS200(i) Intermittent Freeze and Shutdown).

₋ On July 9, 2010, ES&S provided the EAC with a USB trace analysis and clarification to questions about the “DS200 Lockup Analysis.”

₋ On July 22, 2010, The EAC issued an “Initial Decision on Certification” for ES&S Unity 3.2.0.0 Rev. 1 modification. This modification addressed ballot skew and multiple upload station functionality. This modification did not address all known non-conformities.

₋ On August 2-4, 2010, an EAC Computer Engineer traveled to Cuyahoga County to witness Logic & Accuracy Testing.

₋ On August 17-20, 2010, Program Director; Program Deputy Director; an EAC Technical Reviewer; visit Orange County, Florida election offices to discuss the DS200.

₋ On October 8, 2010, The EAC certified ES&S Unity 3.2.0.0 Rev. 1 (Modification).

₋ On October 15, 2010, EAC Program Director, submitted a “Recommendation to Refer for Formal Inquiry ” to EAC Executive Director.

₋ On November 2, 2010, an EAC Computer Engineer traveled to Cuyahoga County, OH, to witness Election Day activities.

₋ On November 30, 2010, iBeta Quality Assurance decided to leave the EAC program, effective December 14, 2010. All testing for Unity 3.2.1.0 was moved to Wyle Laboratories at the request of ES&S. iBeta provided Wyle with a partial test report for Unity 3.2.1.0.

₋ On December 20, 2010, EAC Program Director submitted an addendum to the “Recommendation to Refer for Formal Inquiry ” to the EAC Executive Director.

₋ On February 11, 2011, ES&S issued Technical Bulletin PRBDS2000010 (Public Counter Does Not Increment When Ballot is Dropped into Ballot Box).

₋ On February 17, 2011, ES&S issued Technical Bulletin FYIDS2000019 (DS200 Threshold Settings).

₋ On February 25, 2011 EAC Executive Director issued an “Authorization for Formal Investigation”. The investigation focused on these anomalies: freeze/ shutdown, ballot skew, failure to log and vote miscount. The EAC issued a “Notice of Formal Investigation” to ES&S on March 1, 2011.

₋ On March 3, 2011, the EAC released a Technical Advisory titled “Ballot Drop with EAC Certified ES&S Unity 3.2.0.0 and 3.2.0.0 Rev. 1 .”


₋ On March 29, 2011, the EAC certified ES&S Unity 3.2.1.0 (EAC Certification number: ESSUnity3210).

₋ On March 30, 2011, the EAC sent interrogatories for the Formal Inquiry to iBeta Quality Assurance, Wyle Laboratories, Cuyahoga County and ES&S. The EAC requested all parties to respond by April 13, 2011. All parties responded to the Interrogatories by April 14, 2011. The interrogatories and responses focused on issue notification, quality assurance testing, configuration management and root cause analysis of reported issues.

₋ On April 15, 2011, the EAC approved an application for ES&S Unity 3.2.0.0 Rev. 2 (Modification). This modification used firmware from the EAC Certified ES&S Unity 3.2.1.0 system to fix the freeze/shutdown, ballot skew, failure to log, and vote miscount as identified in the “Notice of Formal Investigation.”

₋ On June 13-15, 2011, an EAC Computer Engineer traveled to Cuyahoga to witness the County’s upgrade to the Unity 3.2.1.0 system. The EAC Engineer reported occurrences of the ballot skew anomaly, as well as several other issues.

₋ On July 28, 2011, the EAC released a Technical Advisory, titled “DS200 Unresponsive Touchscreen .”

₋ On July 28, 2011, the EAC released a Technical Advisory titled “Ballot Skew.”

₋ On July, 28, 2011, ES&S Unity 3.3.0.0 application to modify EAC Certified ES&S Unity 3.2.1.0 by adding the DS850 hardware component was approved.

₋ On July 29, 2011, the EAC approved an application for ES&S Unity 3.2.1.0 Rev.1 (Modification). This modification fixes the anomaly presented by the diagnostic log, used to analyze the freeze/shutdown issue, in EAC certified Unity 3.2.1.0.

₋ On July, 29, 2011, ES&S Unity 3.4.0.0 application to modify EAC Certified ES&S Unity 3.2.1.0 by adding landline modeming functionality to the DS200 was approved.

₋ On August 3, 2011, ES&S issued Technical Bulletin FYIDS2000021 (Mark Reported Missed during a Customer Acceptance Testing Exercise).

₋ On August 3, 2011, ES&S issued Technical Bulletin PRBDS2000013 (System Log Fills Internal Compact Flash Card Partition in DS200 Versions 1.4.3.11 (Unity 3.2.1.0) or 1.5.2.0 (4.0.0.3 Version 2 [FL]).

₋ On August 9, 2011, Unity 3.2.0.0 Rev 2. test campaign was suspended after calibration and ballot skew anomalies were found in fielded DS200’s Unity 3.2.1.0 and reported to the EAC (see June 13-25, 2011).

₋ On October 25, 2011, ES&S expressed their intent to terminate Unity 3.2.0.0 Rev 2., Unity 3.2.1.0 Rev 1., and Unity 3.3.0.0 test campaigns and merge all proposed fixes and modifications into the Unity 3.4.0.0 test campaign.

http:1.4.3.11


5. Analysis

5.1 Analysis of freeze/shutdown On April 15, 2010, the EAC became aware of an issue with the EAC certified DS200 (Unity 3.2.0.0) in Cuyahoga County, OH. The DS200 demonstrated intermittent screen freezes, system lockups and shutdowns. ES&S provided the EAC with a root cause analysis of the freeze/shutdown anomaly. The DS200 System Lockup Analysis (“Appendix Q - DS200 Lockup Analysis”) cited two causes of the freeze/ shutdown anomaly.

The first cause cited was a memory management issue. In the DS200 source code there were several instances where proper memory deallocation was not performed. The second contributing factor cited in the report was an X-Windows call , which caused a fatal fault in the system. The following is a summary of the root cause analysis performed by ES&S on the DS200, taken directly from the Lockup Analysis provided to the EAC on June 28, 2010:

“In summary, through analysis of the accumulated data, the investigating engineers have determined that the sequence of events leading up to and causing the shutdowns is as follows:

₋ A program (HAL) that uses the x-windows library makes a call to the x-windows server. The specific call listed above is XOpenDisplay().

₋ The x-call fails and this causes the pipe between the client (HAL) and the server to break and the OS raises the SIGPIPE signal.

₋ The client (ES&S code) catches the signal but the x-windows system was designed such that an xclient process must exit (the exit call is contained in the x library code) if it loses contact with the server.

₋ The client (HAL) exits.

₋ The rest of the ES&S code no longer has access to the information it needs and therefore shuts itself down.”

The excerpt below outlines the proposed changes to fix the anomalies described by ES&S in the Lockup Analysis:

“A vast majority of the changes to the ES&S firmware source code as a result of the investigation fall into 4 main categories.

The largest category is the addition of extensive diagnostic logging. Software engineers changed a function named LogIt to LogErr. The LogErr routine calls the Linux OS system function vsyslog to write entries to a log located in the log directory on the var partition of the CF card. (/var/log) This log file is named “messages” with no suffix. This file is designed to provide diagnostic information in case of a problem. ES&S software engineers felt the need to expand the diagnostics based upon a lesson learned from confronting a lack of helpful data to


use to diagnose the initial problem. It is a “rotating” file that never grows beyond a limited size.

The second category is the removal of all code that accessed the x-windows library in the fashion that exposes the apparent bug in the x-windows library. This deals primarily with queries to determine the screen and backlight state. These changes also involve making sure the backlight no longer turns off after a specified period of time. This avoids the possibility that the backlight will shut off and a user could inadvertently touch the screen on a button (hotspot) when they attempt to “wake up” the display.

The third category is the code to improve memory management problems as detailed earlier in this document. Most of this is making sure data pointers are properly set to null after a free() call.

The last category contains a relatively small amount of code to look for a ballot in the transport upon start up. If a ballot is in the transport, the DS200 will automatically back it out of the machine.”

In response to the Lockup Analysis, an EAC Computer Engineer attempted to verify the conclusions drawn by ES&S. The EAC determined the examination completed by ES&S did not provide a sufficient explanation or analysis of the root cause for the freeze/shutdown anomaly. The EAC based its determination on the fact that the Lockup Analysis did not provide adequate details or descriptions of the investigation conducted by ES&S, and that the test parameters, setup, raw data and results of the testing were insufficient for the EAC to validate the analysis.

ES&S applied their identified fixes for this anomaly to the Unity 3.2.1.0 certification effort then underway at iBeta Quality Assurance. The EAC, using the information provided by ES&S and independent EAC research, issued test expectations to iBeta Quality Assurance (“Appendix W - Expectations for Freeze Shutdown Testing”). The anomaly did not appear in testing after the ES&S fixes were applied to the system. Upon completion of testing, iBeta Quality Assurance submitted a partial test report for Unity 3.2.1.0 to the EAC. During the EAC review of the test report, iBeta withdrew from the Voting System Test Laboratory program(“Appendix X - iBeta VSTL Withdrawal Letter”). The EAC granted the ES&S request to move the system to Wyle Laboratories per Section 4.3.1.2 of the EAC Testing and Certification Program Manual (“Appendix Y - Approval to move Unity 3.2.1.0 to Wyle”) on January 11, 2011. Wyle addressed the remaining issues with the Unity 3.2.1.0 system and recommended it for certification. The EAC certified the system on April 4, 2011 (EAC certification # ESSUNITY3210). On April 15, 2011, ES&S submitted a testing application for the Unity 3.2.0.0 Rev. 2 modicication for the Unity 3.2.0.0 Rev. 1 system. This submission was to correct the anomalies noted in the Scope of Formal Investigation for the Unity 3.2.0.0 system. The fixes for Unity 3.2.0.0 Rev. 2 modification are from the EAC Certified Unity 3.2.1.0. ES&S chose to place the identical firmware from Unity 3.2.1.0 on the Unity 3.2.0.0 DS200.


In June 2011, an EAC Computer Engineer witnessed the upgrade from Unity 3.2.0.0 to Unity 3.2.1.0 in Cuyahoga County, OH. During testing of the upgrade, the EAC discovered that the DS200 touch screen interface became unresponsive at random times. Cuyahoga County provided this information to ES&S and the EAC requested that ES&S provide a root cause analysis of this latest anomaly. ES&S reported (“Appendix Z - ES&S Response to Investigation”) that the cause of the screen unresponsiveness was a diagnostic log feature ES&S added during the initial investigation of the freeze/shutdown anomaly (Now referred to as the “Sys.log file”).

In their report, ES&S stated that because the file does not “rotate”, meaning the system did not create a new file to prevent the original file from growing beyond its memory allotment, it grows without bounds. This uncontrolled growth made a section of the internal compact flash (CF) card inaccessible. This inaccessible section of the CF card also contained the calibration settings for the DS200’s touch screen interface. When this section of the CF card became inaccessible, the calibration settings were no longer available to the system and caused the screen to become unresponsive.

5.2 Analysis of failure to log During the course of the formal investigation, the EAC examined DS200 logs provided by multiple jurisdictions. Using the ES&S Software Design Specification, EAC staff extracted the logs from the election media and analyzed their content in Excel. During analysis, the EAC discovered the Unity 3.2.0.0 DS200’s were not capturing some significant system events. Events not captured in the logs included:

₋ Casting a vote

₋ Power on, power off, and

₋ Certain administrative functions.

ES&S proposed to resolve this anomaly with a modification to the Unity 3.2.1.0 system. The EAC instructed iBeta Quality Assurance to review the system logs for conformance to the 2002 VSS. (“Appendix W - Expectations for Freeze Shutdown Testing”). The laboratory recommended the Unity 3.2.1.0 system for certification stating that all VSS and VVSG requirements had been met. The EAC certified the system on April 4, 2011. On April 15, 2011, ES&S submitted a testing application for the Unity 3.2.0.0 Rev. 2 modification. This application was to correct the logging anomalies noted in the Scope of Formal Investigation for the Unity 3.2.0.0 system. The fixes for Unity 3.2.0.0 Rev. 2 modification are from the EAC Certified Unity 3.2.1.0 DS200. ES&S choose to place the identical firmware from Unity 3.2.1.0 onthe Unity 3.2.0.0 DS200.

During installation of the 3.2.1.0 upgrade in Cuyahoga County, OH, election officials provided DS200 system logs to the EAC for examination. EAC staff compared the logged actions to the actual actions of the county testers. (“Appendix AA - Cuyahoga 3.2.1.0 Upgrade Checklist”). The EAC found that the DS200 event log still did not


record all significant system events. For example, one of the steps that the testperforms is to calibrate the touch screen interface. This calibration event is notrecorded.

5.3 Analysis of ballot skew The ballot skew anomaly was discovered during the Unity 3.2.1.0 EAC testing campaign. When a 17” ballot was inserted at an angle, the DS200 did not consistently count the mark properly. The mark registered either as a different selection than intended or did not register at all. ES&S reported to the EAC that ballot skew caused this anomaly. Ballot skew is defined as instances in which the image of the ballot captured by the OS units bowed or out of perfectly straight alignment in some other way. The DS200 uses this image to create evaluation areas where the voting targets, the space a voter marks, should be. This skew caused the mark evaluation area to shift to a different location than the voting target. The DS200 codebase for Unity 3.2.1.0 is similar to the certified Unity 3.2.0.0 software that was under an informal inquiry. The EAC suspected that the anomaly would also be present in the 3.2.0.0 DS200. Due to this possibility, the EAC expanded the scope of the inquiry to include the ballot skewing issue.

ES&S submitted a fix to the skew issue to iBeta Quality Assurance. Along with this fix, ES&S submitted the internal testing that they performed to show that the anomaly was resolved. After iBeta reviewed the testing performed by ES&S, they reported the anomaly as resolved to the EAC. In June 2011, EAC staff witnessed the upgrade from Unity 3.2.0.0 to Unity 3.2.1.0 in Cuyahoga County, OH. During this upgrade and testing, a valid vote (mark) was not counted by one of the DS200s.

The EAC, while at the Cuyahoga testing facility, began an analysis of the ballot images to see if ballot skew was the cause of the anomaly. The EAC documented a noticeable amount of skewing in many of the ballot images recorded on the DS200.

ES&S subsequently submitted information describing how their patented mark recognition software works. Using this information, the EAC performed an analysis on 11”, 14”, 17” and 19” ballots. The EAC found varying degrees of ballot image distortion; with the 17” ballot having the largest degree of skew.

5.4 Analysis of vote miscount During testing of the Unity 3.2.1.0 system, one of the DS200 units accepted a voted ballot without incrementing the public counter or counting any of the marks. As a result, the laboratory test personnel found two ballots were in the DS200 ballot bin but only one ballot was counted by the DS200.

The DS200 displayed two different messages in rapid succession when this anomaly occurred. The first message informed the voter that votes for more than one party was selected. This is followed by a message advising the voter to either accept or


return the ballot. In normal operation of the system the messages would have required the user to respond before the system would continue. The system did not require response and continued on.

On November 29, 2010 iBeta laboratories submitted a letter (“Appendix X - iBeta VSTL Withdrawal Letter”) informing the EAC of their intent to resign as a VSTL. Due to the resignation, ES&S moved all testing campaigns to Wyle Laboratories. Wyle Laboratories and ES&S conducted extensive research in order to identify the root cause of the ballot counter anomaly. Wyle traced the anomaly through the voting system’s source code and found that when certain ballot exceptions (i.e., ballot overvote messages) occur while processing a ballot, a function within the scanner can incorrectly make the DS200 believe the submitted ballot is twenty-eight (28) inches long . Therefore, when a standard ballot was fed into the scanner and this error condition occurred the ballot was accepted into the ballot bin without incrementing the counter because the scanner was expecting a longer ballot. ES&S implemented improvements to the Unity 3.2.1.0 DS200 firmware in an attempt to resolve this anomaly.

6. Findings

6.1 Freeze/shutdown Based on the analysis of the freeze/shutdown issue described in Section 5.1 ofthis document, the EAC believes that the allegation, as described in the Scopeof Investigation (Section 2.1), is substantiated. The EAC believes that the fixprovided in Unity 3.2.0.0 Rev. 2 (changed to Unity 3.4.0.0. See page 15 of thisreport) will cure the anomaly.

The addition of the diagnostic log added to improve troubleshooting capabilities introduced a new anomaly noted in the above analysis. The EAC therefore finds that the DS200 is non-compliant with Volume 1 Section 2.2.1.b:

2.2.1.b . Provide system functions that are executable only in the intended manner and order, and only under the intended conditions.

And Volume 1 sections 7.2, 7.4, 7.5, & 7.6 Quality Assurance of the 2002 VSS:

7.2 General Requirements:

The voting system vendor is responsible for designing and implementing a quality assurance program to ensure that the design, workmanship, and performance requirements of this standard are achieved in all delivered systems and components.

At a minimum, this program shall:

a. Include procedures for specifying, procuring, inspecting, accepting, and controlling parts and raw materials of the requisite quality;


b. Require the documentation of the hardware and software development process;

c. Identify and enforce all requirements for:

1) In-process inspection and testing that the manufacturer deems necessary to ensure proper fabrication and assembly of hardware, and

2) Installation and operation of software (including firmware).

d. Include plans and procedures for post-production environmental screening and acceptance test; and

e. Include a procedure for maintaining all data and records required to document and verify the quality inspections and tests.

7.4 Responsibility for Tests

The manufacturer or vendor shall be responsible for:

a. Performing all quality assurance tests;

b. Acquiring and documenting test data; and

c. Providing test reports for review by the ITA, and to the purchaser upon request.

7.5 Parts & Materials Special Tests and Examinations

In order to ensure that voting system parts and materials function properly, vendors shall:

a. Select parts and materials to be used in voting systems and components according to their suitability for the intended application. Suitability may be determined by similarity of this application to existing standard practice, or by means of special tests;

b. Design special tests, if needed, to evaluate the part or material under conditions accurately simulating the actual operating environment; and

c. Maintain the resulting test data as part of the quality assurance program documentation.

7.6 Quality Conformance Inspections

The vendor performs conformance inspections to ensure the overall quality of the voting system and components delivered to the ITA for testing and to the jurisdiction for implementation. To meet the conformance inspection requirements the vendor or manufacturer shall:

a. Inspect and test each voting system or component to verify that it meets all inspection and test requirements for the system; and


b. Deliver a record of tests, or a certificate of satisfactory completion, with each system or component.

6.2 Failure to log Based on the analysis of the DS200 logs in the Unity 3.2.0.0 described in Section 5.2 of this document, the EAC finds that the allegation that the DS200 is non-compliant with Volume 1 section 2.2.4 of the 2002 Voting System Standards issubstantiated.

Integrity measures ensure the physical stability and function of the vote recording and counting processes.

2.2.4.1 - Common Standards

To ensure system integrity, all systems shall:

g. Record and report the date and time of normal and abnormal events.

i. Detect and record every event, including the occurrence of an error condition that the system cannot overcome, and time-dependent or programmed events that occur without the intervention of the voter or a polling place operator.

6.3 Ballot skew Based on the analysis of the DS200 ballot images in the Unity 3.2.0.0 and 3.2.1.0 as described in Section 5.3 of this document, the EAC finds that the allegation that the DS200 is non-compliant with Volume 1 Section 2.2.2.1.c of the 2002 Voting System Standards is substantiated.

c. Record each vote precisely as indicated by the voter and be able to produce an accurate report of all votes cast

In addition, during the analysis the EAC discovered that the DS200 is also in non-compliance with Volume 1 Section 3.2.1.a & c Accuracy:

Voting system accuracy addresses the accuracy of data for each of the individual ballot positions that could be selected by a voter, including the positions that are not selected.

For a voting system, accuracy is defined as the ability of the system to capture, record, store, consolidate and report the specific selections and absence of selections, made by the voter for each ballot position without error. Required accuracy is defined in terms of an error rate that for testing purposes represents the maximum number of errors allowed while processing a specified volume of data. This rate is set at a sufficiently stringent level such that the likelihood of voting system errors affecting the outcome of an election is exceptionally remote even in the closest of elections.

The error rate is defined using a convention that recognizes differences in how vote data is processed by different types of voting systems. Paper-based and


DRE systems have different processing steps. Some differences also exist between precinct count and central count systems. Therefore, the acceptable error rate applies separately and distinctly to each of the following functions:

a. For all paper-based systems:

i. Scanning ballot positions on paper ballots to detect selections for individual candidates and contests;

ii. Conversion of selections detected on paper ballots into digital data;

c. For precinct-count systems (paper-based and DRE):

Consolidation of vote selection data from multiple precinct-based systems to generate jurisdiction-wide vote counts, including storage and reporting of the consolidated vote data; and

For testing purposes, the acceptable error rate is defined using two parameters: the desired error rate to be achieved, and the maximum error rate that should be accepted by the test process.

For each processing function indicated above, the system shall achieve a target error rate of no more than one in 10,000,000 ballot positions, with a maximum acceptable error rate in the test process of one in 500,000 ballot positions.

And Volume 1 sections 7.2, 7.4, 7.5, & 7.6 Quality Assurance of the 2002 Voting System Standard:

7.2 General Requirements:

The voting system vendor is responsible for designing and implementing a quality assurance program to ensure that the design, workmanship, and performance requirements of this standard are achieved in all delivered systems and components.

At a minimum, this program shall:

a. Include procedures for specifying, procuring, inspecting, accepting, and controlling parts and raw materials of the requisite quality;

b. Require the documentation of the hardware and software development process;

c. Identify and enforce all requirements for:

1) In-process inspection and testing that the manufacturer deems necessary to ensure proper fabrication and assembly of hardware, and

2) Installation and operation of software (including firmware).


d. Include plans and procedures for post-production environmental screening and acceptance test; and

e. Include a procedure for maintaining all data and records required to document and verify the quality inspections and tests.

7.4 Responsibility for Tests

The manufacturer or vendor shall be responsible for:

a. Performing all quality assurance tests;

b. Acquiring and documenting test data; and

c. Providing test reports for review by the ITA, and to the purchaser upon request.

7.5 Parts & Materials Special Tests and Examinations

In order to ensure that voting system parts and materials function properly, vendors shall:

a. Select parts and materials to be used in voting systems and components according to their suitability for the intended application. Suitability may be determined by similarity of this application to existing standard practice, or by means of special tests;

b. Design special tests, if needed, to evaluate the part or material under conditions accurately simulating the actual operating environment; and

c. Maintain the resulting test data as part of the quality assurance program documentation.

7.6 Quality Conformance Inspections

The vendor performs conformance inspections to ensure the overall quality of the voting system and components delivered to the ITA for testing andto the jurisdiction for implementation. To meet the conformance inspectionrequirements the vendor or manufacturer shall:

a. Inspect and test each voting system or component to verify that it meets all inspection and test requirements for the system; and

b. Deliver a record of tests, or a certificate of satisfactory completion, with each system or component.

6.4 Vote miscount During the course of the investigation ES&S proposed a fix to the DS200. This fix was tested in the Unity 3.2.1.0 test campaign. Based on the testing and subsequent June 11, 2011 field audit, the EAC finds that the allegation described in Section 5.4


of this document that the DS200 is non-compliant with Volume 1 Section 2.1.8.b of the 2002 Voting System Standards is unsubstantiated.

For all voting systems, each piece of voting equipment that tabulates ballots shall provide a counter that:

b. Records the number of ballots cast during a particular test cycle or election.

7. Recommendation

For the reasons noted in findings section of this document, the EAC deems three of the four allegations described in this investigation to be substantiated. The cure presently proposed by ES&S, Unity 3.2.0.0 Rev. 2, as demonstrated in the upgrade in Cuyahoga County does not correct all anomalies cited in this investigation. These anomalies remain in the EAC Certified Unity 3.2.0.0 system in use today. In addition, the proposed fixes created additional anomalies discovered in the Unity 3.2.1.0 system.

Based on the findings discussed above and the fact that the fix proposed in Unity 3.2.0.0 Rev. 2 is entirely based on Unity 3.2.1.0, which demonstrated some of the anomalies that prompted this investigation, the Program Director recommends a Manufacturer Site Audit to evaluate the developmental testing and quality assurance practices of ES&S (See Section 8.5 of the EAC Voting System Testing and Certification Program Manual). The EAC has noted outstanding issues related to the Quality Assurance practices of the manufacturer, based on the recurrence of certain anomalies and the fact that fixes provided for certain anomalies created additional anomalies with the DS200. The EAC intends to conduct a Manufacturer Site Audit to review policies, procedures and practices related to Quality Assurance and Configuration Management, to ensure that the Manufacturer meets EAC standards.

On November 9, 2011, the EAC received an application modification from ES&S requesting to combine the scope of testing for Unity 3.2.0.0 Rev 2, Unity 3.2.1.0 Rev 1, and Unity 3.3.0.0 into the revised ES&S Unity 3.4.0.0 test campaign. ES&S requested that all test activities cease on the three above mentioned test campaigns and noted that separate letters withdrawing these systems from certification testing would be submitted separately to the EAC. On November 17, 2011 the EAC accepted this modified application for the ES&S Unity 3.4.0.0 test campaign.

In addition, the Program Director recommends that the EAC issue a Notice of Non-Compliance (See Sections 7.6-7.7.3 of the Manual) for the substantiated allegations in the EAC certified Unity 3.2.0.0 system. The Notice of Non-Compliance will be sent to the Manufacturer regarding the substantiated allegations. The Notice will inform the Manufacturer of the next steps in the process, including the Manufacturer’s opportunity to cure non-compliance and have an opportunity to be heard, based on Sections 7.6-7.7.3 of the Manual.


Per Section 7.2 of the Manual, voting systems certified by the EAC may be decertified if they are shown not to meet an applicable voting system standard or standards. Systems may only be decertified after the completion of an Informal Inquiry and Formal Investigation.

Decertification is the process the EAC uses to revoke a certification granted for a voting system. It serves to ensure that program requirements are followed and that certified voting systems fielded for use maintain the same level of quality as those presented for testing.

Decision Authority Signoff I have reviewed this Formal Investigation Report and concur with the recommendations outlined in Section Seven ( 7) of the Report.

Mark Robbins, General Counsel and Acting Executive Director

United States Election Assistance Commission

12/20/11

Date


http:www.eac.gov

Appendix A - ES&S Technical Bulletin PRBD2000006

Appendix B - Cleveland Plain Dealer

10 percent of Cuyahoga County's voting machines fail pre-election tests

By Joan Mazzolini, The Plain Dealer

April 14, 2010, 4:00AM

Scott Shaw, The Plain Dealer

Election workers return optical scanners to storage after the November 2008 elections. Tests before this year's primaries showed problems with about 10 percent of the devices. CLEVELAND, Ohio -- About 10 percent of Cuyahoga County's voting machines checked so far have failed a pre-election test, once again challenging public confidence in the election system.

With just weeks until the May 4 primary election, the system's manufacturer, Election Systems & Software Inc., has been unable to find what is causing the machines to arbitrarily power down and lock up during a test.

The problem showed up about 10 days ago when the Cuyahoga County Board of Elections began a series of routine tests on the voting machines. The machines froze during a specific test done to ensure the optical scanners were reading paper ballots correctly. At different points in that test, the machines simply started powering down, then freezing.

The Cuyahoga County Board of Elections has 1,200 voting machines. Of the 279 that have been tested, 28 have had this problem.

But it is hard to tell how pervasive the problem is. Board members were told at their meeting Tuesday that some machines that first passed the test, later failed when the procedure was repeated.

http://connect.cleveland.com/user/jmazzoli/index.htmlhttp://www.essvote.com/HTML/home.htmlhttp://boe.cuyahogacounty.us/http://boe.cuyahogacounty.us/

Employees stopped the test after 19 of the 279 machines had problems. Those 19 were put aside, and the staff started the tests on the others all over again. By the time they had retested 200 machines, nine of those that initially passed had failed the second time around.

With optical-scan machines, voters fill in ovals next to candidates' names on a printed paper ballot. The votes are tallied when the ballot is fed into the machine at the polling location.

"A 10 percent failure rates is high," said board member Inajo Davis Chappell. "We've never had this rate of failure. We don't want the public to have a lack of confidence."

The problem "is completely unacceptable," board Chairman Jeff Hastings told an ES&S representative. "I hope the secretary of state knows about this."

Kyle Weber, ES&S's project manager for Cuyahoga County, said Tuesday that officials were working hard on trying to find the problem and determine how to fix it before the election.

It isn't the first time the board has had problems with its election system.

The current board was installed in 2007 after Ohio Secretary of State Jennifer Brunner fired the former members because of mismanagement. The new members pledged to restore voter confidence, which was damaged by several events, including a series of problems with the then-new touch screen voting machines.

Those machines, which cost about $20 million, were shelved at Brunner's request and the county went to a new optical scan machine that up to now had been working well. A five-year deal with ES&S cost $13.4 million.

And officials say the failure rate in an actual election is very low; about four scanner machines have to be swapped out because of a problem in most elections.

Pat McDonald, elections board deputy director, said that if ES&S can't figure out the problem and a fix before the primary May 4, there will be a contingency plan that will include sending replacement machines to precincts.

McDonald said after the meeting that if a large number of machines freeze up, workers could bring a locked box of paper ballots from the failed machines to board headquarters and count them with high-speed scanners.

The high-speed scanners could, if necessary, scan the ballots for the entire election, which would slow down the counting process. While the board attempts to get results out by about midnight on election night, state law requires boards of election only to have tabulation complete by the next day.

Board member Eben "Sandy" McNair said that because the votes are now marked on paper ballots, voters and others don't have to worry that they will be lost or not counted.

http://blog.cleveland.com/metro/2007/03/cuyahoga_board_of_elections_or.htmlhttp://blog.cleveland.com/metro/2007/03/elections_chief_cuyahoga_votin.htmlhttp://blog.cleveland.com/metro/2007/03/elections_chief_cuyahoga_votin.htmlhttp://blog.cleveland.com/metro/2007/12/state_elections_chief_votes_to.html

With the touch-screen machines, votes were kept on memory cards that could be lost or erased. And while there was a paper trail similar to a cash register tape, it was hard to use to count votes and in some cases jammed during the election, making the trail non-existent.

While officials are concerned about the Election Day process, increasing numbers of Cuyahoga County residents have been choosing to vote absentee since the law changed allowing anyone to vote that way.

In the 2009 May primary, more votes were cast in Cuyahoga County through absentee ballots than at the polling locations. Fifty nine percent of the 72,447 votes cast, or 42,582, came in through absentee ballots.

So far this year, 174,594 voters have requested an absentee ballot for the May primary.

Appendix C - ES&S Guidence to Cuyahoga County (Freeze)

MEMORANDUM

TO: Jane Platten ‐ Director, Cuyahoga County Board of Elections

CC: Ken Carbullido, ES&S Senior Vice‐President Aldo Tesi, ES&S President & CEO

FROM: Kyle Weber, ES&S Project Manager

DATE: 23 April 2010

SUBJECT: DS200(i) – Recommended Procedures for Election Day

ES&S Recommended Procedures for Election Day

If during the Cuyahoga County May 4th Primary Election a call is received by election central in reference to the DS200 shutdown issue, the following steps are recommended for your consideration and final approval. These instructions can then be utilized by election day call center personnel.

ES&S on-site support staff is available to discuss and/or demonstrate each of these recommended contingency instructions with you and your staff. We have presented the various scenarios that may be encountered.

Please see the attached page for recommended help desk staff instructions to assist poll workers.

.

“Maintaining Voter Confidence. Enhancing the Voting Experience.” 11208 John Galt Boulevard • Omaha, NE 68137 USA • Phone: 402.593.0101 • Toll-Free: 1.800.247.8683 • Fax: 402.593.8107 • www.essvote.com

http:www.essvote.com

Recommended Procedures for Election Day

If the DS200 shuts down during opening/closing of the polls…

Press and hold the “Power” button until it turns red.

Once the red light goes off and the machine is completely shut down, press the “Power” button to restart the DS200.

Continue with the opening/closing polls process according to the poll worker manual.

If the DS200 shuts down during voting…

Determine if there is a ballot attached to the back of the DS200 by instructing the poll workers to do the following:

-Go to the rear of the machine where the power cord is located.

-Look through the clear plastic windows on the back of the ballot bin below the power supply.

-If a ballot is still attached, they will see the white of the paper. If not, it will be dark inside the ballot box.

If no ballot is attached to the back of the DS200

Press and hold the “Power” button until it turns red.

Once the red light goes off and the machine is completely shut down, press the “Power” button to restart the DS200 and continue voting.

If necessary, scan any ballots that may have been inserted into the emergency slot.

Continue voting by having voters insert ballots directly into the DS200.

If there is a ballot attached to the back of the DS200

Instruct the poll workers to begin using the emergency ballot box slot on the front of the ballot bin.

Deploy a roving technician to the location with the backup memory stick for that precinct.

Once onsite, have the tech remove the ballot attached to the back of the DS200 by doing the following:

-Remove the seal and unlock the ballot bin front flap

-Flip the ballot bin front flap down and slide the DS200 forward

-Gently remove the ballot from the back of the DS200

-Slide the DS200 back into place and lock and seal the ballot bin front flap.

CAUTION: When sliding the DS200 back into place, make sure the power cord does not block the ballot path.

Press and hold the “Power” button until it turns red. Wait until the red light goes off and the DS200 completely shuts down.

Replace the election day memory stick with the backup memory stick.

Press the “Power” button to restart the DS200.

With a bi-partisan team, rescan all ballots in the precinct, including those in the ballot bin, any that may have been inserted into the emergency slot, and the ballot removed from the back of the DS200.


Appendix D - EAC Technical Advisory ESS2010-01

U.S. Election Assistance Commission Voting System ESS# 2010‐01Testing and Certification Program

1201 New York Avenue, NW, Suite 300 Washington, DC. 20005

Publication Date: June 25, 2010

Voting System Technical Advisory Intermittent Freeze/Shutdowns with EAC Certified

ES&S Unity 3.2.0.0 System

System(s) Affected: ES&S Unity 3.2.0.0 System Component(s) Affected: DS200 Version(s) Affected: Firmware v. 1.3.10.0; Hardware v. 1.1, v.1.2; COTS

Operating System v. 2.6.16.27 Notification Date: May 21, 2010 Summary: Intermittent freezes, lockups, and shutdowns

Advisory: Counties and jurisdictions with this product should be aware of “power down” and “freeze” issue experienced during Logic & Accuracy (L&A) testing and Election Day.

Status: EAC has launched an Informal Investigation and is working with ES&S to find a root cause to these issues.

Overview: The DS200 precinct count optical scan voting device fielded in Cuyahoga County,Ohio is part of the EAC Certified Unity 3.2.0.0 voting system. During pre‐election logic and accuracy (L&A) testing prior to the May 4, 2010 Primary Election the DS200 demonstrated intermittent screen freezes, system lockups and shutdowns. These issues were conveyed to the voting system manufacturer, Election Systems & Software (ES&S). ES&S provided the county with initial information on what theybelieve had occurred during L&A testing and during the subsequent election. EAC was notified of the anomaly and has contacted Cuyahoga County and otherjurisdictions that use the same system, as well as ES&S to gather information. AnInformal Investigation into these issues has been launched by the EAC.

Issue Descriptions: Cuyahoga County enhanced and expanded the Logic & Accuracy (L&A) tests that were initially provided by ES&S. L&A testing was initiated 3 to 4 weeks prior to the May 4th election. Cuyahoga County has 1068 precincts with ≈1200 machines to fulfill

training and election needs. During the course of L&A testing a “power down” and “freezing” anomaly occurred on some DS200 machines. This anomaly appeared a total of 89 times during L&A testing without a distinguishable pattern in the timing or actions taken to cause the freeze/shutdown issue. In addition, another 8 of 108 new DS200’s failed Cuyahoga County’s acceptance and independent verification and validation testing which is conducted on newly received systems prior to acceptance. These systems were not deployed in the election.

During the May 4th Election, poll workers reported four DS200 shutdowns to theCounty. The poll workers tried to troubleshoot the DS200 machine failures and were able to restore the systems for use during the election period by rebooting the machines. Cuyahoga County officials asked poll workers who experiences this issue to reboot the DS200 and check for “hanging” or “stuck” ballots prior to allowing voters to use the machine. The county also completed a hand count in the precincts in which the shutdown occurred to make sure votes were not lost. Although the machine failures were encountered less frequently on Election Day than during L&A testing, the anomaly still presented itself in a number of machines.

Root Cause: The EAC is working with ES&S in order to help determine a root cause.

ES&S Recommended Procedures for Election Day: If the DS200 shuts down during opening/closing of the polls… Press and hold the “Power” button until it turns red. Once the red light goes off and the machine is completely shut down, press the

“Power” button to restart the DS200. Continue with the opening/closing polls process according to the poll worker

manual.

If the DS200 shuts down during voting… Determine if there is a ballot attached to the back of the DS200 by instructing

the poll workers to do the following: ‐Go to the rear of the machine where the power cord is located. ‐Look through the clear plastic windows on the back of the ballot bin below the power supply. ‐If a ballot is still attached, they will see the white of the paper. If not, it will be dark inside the ballot box.

If no ballot is attached to the back of the DS200 Press and hold the “Power” button until it turns red. Once the red light goes off and the machine is completely shut down, press the

“Power” button to restart the DS200 and continue voting. If necessary, scan any ballots that may have been inserted into the emergency

slot. Continue voting by having voters insert ballots directly into the DS200.

2

If there is a ballot attached to the back of the DS200 Instruct the poll workers to begin using the emergency ballot box slot on the

front of the ballot bin. Deploy a roving technician to the location with the backup memory stick for that

precinct. Once onsite, have the tech remove the ballot attached to the back of the DS200

by doing the following: ‐Remove the seal and unlock the ballot bin front flap ‐Flip the ballot bin front flap down and slide the DS200 forward ‐Gently remove the ballot from the back of the DS200 ‐Slide the DS200 back into place and lock and seal the ballot bin front flap. CAUTION: When sliding the DS200 back into place, make sure the power cord does not block the ballot path.

Press and hold the “Power” button until it turns red. Wait until the red light goes off and the DS200 completely shuts down.

Replace the Election Day memory stick with the backup memory stick. Press the “Power” button to restart the DS200. With a bi‐partisan team, rescan all ballots in the precinct, including those in the

ballot bin, any which may have been inserted into the emergency slot, and the ballot removed from the back of the DS200.


3

Appendix E - ES&S Technical Bulletin PRBD2000008

Appendix F - Recommendation to Refer for Formal Inquiry

U. S. ELECTION ASSISTANCE COMMISSION VOTING SYSTEM TESTING AND CERTIFICATION PROGRAM 1201 New York Avenue, NW, Suite 300 Washington, DC. 20005

To: Tom Wilkey, Executive Director/Decision Authority

From: Brian Hancock, Director of Voting System Testing and Certification

Cc: Matthew V. Masterson, Deputy Director, Voting System Testing and

Certification

Date: October 15, 2010

Subject: Recommendation to Refer for Formal Inquiry

As required under §7.3.5 of the EAC’s Voting System Testing and Certification Program Manual, (Manual) this memorandum constitutes a recommendation that you refer the ES&S DS200 Informal Inquiry for Formal Investigation under § 7.4 of the Manual.

The focus of the Investigation shall be the ES&S DS200 Precinct Count Optical Scanner (Firmware Version 1.3.10.0) contained in the ES&S Unity 3.2.0.0 EAC certified voting system. The scope of the Investigation shall include a conclusive determination of the root cause of the freeze /shutdown anomaly first experienced in Cuyahoga County, Ohio during pre-election logic and accuracy testing in preparation for the May 4, 2010 Primary Election.

The facts and findings from our Informal Investigation into this matter are contained in the attached Informal Inquiry Report. In summary, the report concludes that the claims made by Cuyahoga County are credible and may therefore serve as a basis for decertification of the Unity 3.2.0.0 voting system by the EAC. In addition to the freeze/shutdown anomaly, the Inquiry revealed additional anomalies with the DS200 and included the following:

• Ballot skew

• Ballot insertion problems

• Unlogged errors

• Logged system halts

• TDP Errors

Based upon the inconclusive information provided by the manufacturer regarding the freeze/shutdown anomaly, the EAC has concerns with the results of the root cause analysis that was performed by ES&S regarding the anomaly. At this time, the EAC does not have confidence that the actual root cause of the anomaly was discovered nor remedied.

Attachments

EAC Informal Inquiry Report Cleveland Plain Dealer Article ES&S Notification to the EAC EAC/Cuyahoga County Teleconference Minutes ES&S DS200 System Lockup Analysis EAC DS200 Freeze/Shutdown and XWindows Correlation

Appendix G - Addendum Recommendation to Refer forFormal Inquiry

U. S. ELECTION ASSISTANCE COMMISSION VOTING SYSTEM TESTING AND CERTIFICATION PROGRAM 1201 New York Avenue, NW, Suite 300 Washington, DC. 20005

To: Tom Wilkey, Executive Director/Decision Authority

From: Brian Hancock, Director of Voting System Testing and Certification

Cc: Matthew V. Masterson, Deputy Director, Voting System Testing and

Certification

Date: December 20, 2010

Subject: Addendum to Recommendation to Refer for Formal Inquiry

On October 15, 2010, I forwarded to your office a memorandum constituting a recommendation that you refer the ES&S DS200 Informal Inquiry for Formal Investigation under § 7.4 as required under §7.3.5 of the EAC’s Voting System Testing and Certification Program Manual (Manual).

The purpose of this addendum is to provide you with additional information regarding the ES&S DS200 Precinct Count Optical Scanner (Firmware Version 1.3.10.0) contained in the ES&S Unity 3.2.0.0 voting system. Since the October memorandum, the following additional information has come to light:

As a results of the analysis and changes submitted by ES&S (See section 2.1.5.2 DS200 Field Issue –Freeze and Shutdowns in Appendix H Amended Test Plan) reliability testing of the DS200 was required at iBeta Quality Assurance (iBeta).

The test at iBeta was schedule to run eight days (64 hours) on three units. Testing required that the DS200‟s operate for the full period of time without a loss of one or more functions or degradation of performance such that the device was unable to perform its intended function for longer than ten seconds. On the third day execution of the DS200 Reliability test was halted due to an issue encountered during test script iteration #67. The following observation report was provided to the EAC:

1) “After the first ballot was cast a second ballot was inserted in the DS200. 2) The ballot was an open primary with a vote in two parties (Cross Vote). This ballot issue was identified to the tester with the option to “Accept” or “Reject” the ballot. The tester selected “Accept” and the tester heard the ballot drop. (At this point the system has performed as “intended”.) 3) The tester then observed the screen flash two messages. The first contained the word “issue”. The second contained the word “return”. The motor did not engage or attempt to return the ballot. 4) The voting system continued operation by resetting to the “Welcome” page. It was in a state to accept a new ballot. 5) The tester observed that the ballot counter did not increment (1 vote was displayed).

6) The tester, recorder, ESS and EAC representatives observed there were two ballots

(voter 1 & voter 2) in the ballot box and the counter indicated a single vote.

7) The polls were closed. The reports were printed.

8) It was confirmed on the reports that only a single ballot was recorded. The Cross Vote audit log entry was not recorded, but a returned ballot entry was recorded in the audit log.

9) The system was shut down via the touch screen selection. 10) The system was restarted, polls were re-opened, and additional ballots were scanned

and reported, without error.

Based upon the previous inconclusive information provided by the manufacturer regarding the freeze/shutdown anomaly, as well as the potentially more serious issue of the DS200 accepting a voted ballot but not recording that ballot on its internal counter, the EAC now has concerns not only with the results of the root cause analysis that was performed by ES&S regarding the Freeze/Shutdown anomaly, but has additional concerns regarding this latest anomaly which, as of the date of this memo, has not been addressed by ES&S. Because of the reasons stated above and in the previous memorandum, I reiterate my recommendation that you refer the ES&S DS200 Informal Inquiry for Formal Investigation under § 7.4 of the EAC’s Voting System Testing and Certification Program Manual (Manual).

Attachments Appendix H of Amended Test Plan

Attachment

Excerpt from iBeta VSTL Test Plan: Unity 3.2.1.0 v. 6.0 Datred 12/13/10

2.1.5.2 DS200 Field Issue- Freeze and Shutdowns

DS200 system lockup condition was observed during pre-election logic and accuracy testing for the Cuyahoga County primary election held in May 2010. Systems locked up and had to be restarted. Systems restarted immediately. Lock ups were random and could not be repeated. Following internal diagnosis and testing, ES&S submitted their root cause analysis, testing and system changes to iBeta and the EAC. ES&S analysis identified that the problem occurred more frequently when workers were conducting the Administrator functions. These functions tend to occur before or after the polls are closed at times of greater touch screen interaction. Touch screen interaction during voting is very limited. Following internal diagnosis and testing, ES&S submitted their root cause analysis, testing and system changes to iBeta and the EAC. ES&S‟ analysis resulted in changes to address: Various memory de-allocation improvements (setting pointers to NULL) Various new audit log entries and user interface messages The replacement/upgrade of the flawed Linux X-windows function library used with DS200 v1.3.10.0 and certified with ESSUnity3200 on July 21, 2009.

The list of changes submitted by ES&S are identified in section 1.4.1.2 DS200 Field Issue - System Freezes and Shut Downs.

Scope of testing was expanded to add test scenarios (8 and 9) to the DS200 Functional Test Case (see Table 18) to test the functional enhancements and confirm the continued sufficiency of the battery backup. This testing addresses VVSG Volume 1 sections 2.1.1.b, 2.1.3, 2.1.4.g, 2.1.4.i, 2.1.5.1, 2.1.5.1.b.i, and 2.1.5.1.b.ii. In order to identify the parameters to test the reliability of the DS200, iBeta analyzed 1,713 DS200 diagnostic tests run by ES&S. The analysis found:

There was an averageof 10% freeze failures using the ES&S Touch Test script The six machines tested fai led between 8% and 13%

Of the 1,713 tests, ES&S ran 74 tests on the ESSUNITY3200 certified DS200 firmware v.1.3.10.0: The number of iterations per DS200 ranged between 9 and 14 The average fai lure rate on the certified version was 15% with the individual machine rates ranging from 7% to 25%

Using this analysis iBeta established the baseline average of 10% with a range of 8% to 13% as indicative of a comparable result. A pre-requisite of the DS200 Reliability Test Case will be to determine if iBeta can demonstrate that there is a comparable result on the ESSUNITY3200 certified DS200 firmware v.1.3.10.0 using a sample of 100 tests on five DS200s, with each running the ES&S Diagnostic Touch Test script 20 times. The three DS200s that demonstrate the highest incidence of failure shall be incorporated into the Reliability testing (see Table 19). In order to assess the impact of ENH18551, a change to the COTS X-Windows (see section 1.4.1.2), iBeta identified the modules with calls to X-Windows. These calls simulate keyboard button presses, generate keyboard events sent to X server, convert X-server character values and printable character values; create the calibration window, Recalibrate and Exit buttons and functions for X server to load/unload the driver. These calls will be tested in the DS200 Functional and Reliability Test Cases.

2.1.6 Assessment of DS200 System Halts

2

http:2.1.5.1.b.ii

Attachment

The EAC has instructed iBeta that the “Unity 3.2.1.0 test campaign is a test campaign that is testing the Unity 3.2.1.0 suite end-to-end. It is not a modification of an already certified system. There are no items within the Unity 3.2.1.0 system that are "out of scope" for testing as the entire system is being tested end-to-end. However, the EAC also recognizes that a large portion of the Unity 3.2.1.0 system has been tested and certified by the EAC as part of the 3.2.0.0 certification. Because of this the EAC has already recognized a large portion of the Unity 3.2.0.0 campaign as being applicable to Unity 3.2.1.0. Despite this allowance it is still incumbent on the EAC to fully evaluate the Unity 3.2.1.0 system especially given the already known field issues experienced by the Unity 3.2.0.0 system. Therefore, EAC instructs iBeta to examine the 27 error conditions that cause system halts per ES&S's system documentation and test to make sure each of these halts is properly handled per the standard. If iBeta feels this has been tested already iBeta may provide evidence of this for the EAC to review and accept or reject.” In a preliminary assessment iBeta took a two direction approach to identify the system halts. The first was a search of the code. iBeta identified 30 types of system halts. These systems halts were generated by approximately 140 instances of errors.

The second approach was a search of the documentation. In reviewing the documentation iBeta identified less than 27 errors noted by the EAC Reviewer as system halts.

A review of thecode determined that in some instances these were system halts and in other instances they were not Two documented instances of system halts were not found in the code

In order to ensure all items identified by the EAC were examined, the EAC was requested to provide a list. This will be used as a cross check. (See Table 18)

iBeta analyzed the code associated with the system halts assessment following the ES&S modifications to mitigate the failures. The mitigation consisted of modifying an error handler so that all errors were logged. In the previous versions, a log entry was determined by a flag set by the caller to the error handler and thus some errors were not logged. iBeta determined that of the 140 instances of errors, these errors followed three separate paths through the code. Therefore iBeta chose three instances of errors that were reproducible to test the three paths.

3

Appendix H - ES&S Technical Bulletin PRBDS2000010

Public Counter Does Not Increment When Ballot Is Dropped Into Ballot Box: PRBDS2000010

Technical Bulletin PRBDS2000010 Date February 11, 2011 Product Name DS200 Version See Note* Distribution All

*NOTE: This affects all current versions of the DS200. The updated firmware is part of the Unity 3.2.1.0 product suite currently pending Federal certification.

Problem:

During Federal certification testing of the DS200 by an independent test lab, an anomaly occurred under which the DS200 allowed a ballot to fall into the ballot box and the counter did not increment. After extensive follow‐up testing, ES&S has determined both the root cause of the issue and that this issue occurs only rarely; approximately one instance per every nine thousand ballots cast.

When this rare situation occurs, you will likely see one of the following error messages:

• Ballot Removed During Scan (137)

• Unable to Read Timing Band (123) • Missed Orientation Marks (100)

Confidential and Proprietary Information Election Systems & Software Inc.

This document, as well as the product described in it, is furnished under license and may be used or copied only in accordance with the terms of such license. The content of this document is furnished for informational use only, is subject to change without notice, and should not be con‐strued as a commitment by Election Systems & Software, Inc. Election Systems & Software, Inc., assumes no responsibility or liability for any errors or inaccuracies that may appear in this documentation. Except as permitted by such license, no part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, recording, or otherwise, without the prior written per‐mission of Election Systems and Software, Inc.

2 Bulletin PRBDS2000010

Resolution:

If you see any of the aforementioned error messages, poll workers should complete any necessary actions required by the error messages. In addition, the lead poll official should be notified that a potential ballot count issue has occurred and the public counter for the affected DS200 should be monitored for reoccurrence of the problem. You may also consider removing the unit from service if the situation were to re‐occur.

Any suspected ballot count issue can be validated after poll closing by comparison of the voter check‐in log against the total public count from the DS200 results tapes. State procedures should be followed in the event of a discrepancy.

The root cause of this issue is due to a sensor misread in the ballot transport system. Improvements have been made to the DS200’s firmware to fix this issue. The updated firmware is part of the Unity 3.2.1.0 product suite currently pending Federal certification.

Confidential and Proprietary Information Election Systems and Software, Inc.

Appendix I - ES&S Technical Bulletin FYIDS2000019

DS200 Threshold Settings: FYIDS2000019

Technical Bulletin FYIDS2000019 Date February 17, 2011 Product Name DS200 Version All Di

election systems and sotware unity 3.2.0...dec 20, 2011 · developmental testing and quality...

Documents