On Friday, January 26, 2007 the Election Assistance Commission (EAC) released the Interim Accreditation Assessment report submitted to the EAC back in July of 2006. The report “Election Assistance Commission (EAC) Interim Accreditation Independent Test Authorities (ITA) Assessment Report CIBER & Wyle” is an assessment of an on-site review conducted July 17 to July 22, 2006. It is a damning indictment of the ITA team of the CIBER lab and the Wyle lab, which tested the voting equipment used by at least 70% of the voters in the November 7, 2006 election.

The failures documented in the report exceed the direst fears of those who had come to question the independence, authority and testing competence of the National Association of State Election Directors (NASED) ITA labs. The cadre of doubters came to include many computer security experts, election integrity activists, several state examiners and eventually candidates themselves. Doubt grew with each NASED qualified system which later proved to have significant defects. Doubts grew with revelations of significant defects in systems qualified through NASED ITA program. The defects and illegal system configurations had gone undiscovered by these labs through many rounds of “testing” and remained undiscovered the two labs over the course of years. 

The recurring question of the doubters has always been: “How could such a system pass the qualification testing?”

And make no mistake – this was not an idle question. The NASED/ITA qualification process using ITA labs was vital to the sales and acceptance of many voting systems in many states.  For years manufacturers of voting equipment, state election officials, and current EAC personnel have repeatedly stated the testing done by the ITA labs was thorough and rigorous and thus ensured strict conformance of the qualified systems to the tough standards documented in the 2002 Voting Systems Standards (2002 VSS). The testing and conformance to standards has often been the first line of defense against reports of security vulnerabilities in voting systems.

However, this assessment report released on January 26, 2007, makes clear that for systems tested by the CIBER/Wyle team these frequent statements were unjustified. And even with the publication of this assessment report, most of the details such as what was and was not tested in those systems, remain cloaked behind a veil of secrecy. After reading this assessment, one also has to wonder why Wyle was granted interim accreditation.

The following discussion reveals that:

• CIBER’s test plan was incomplete, even after 10 years of testing voting systems. 

• CIBER did not even have a copy of the latest, incomplete test plan developed in conjunction with Wyle.

 

• CIBER and Wyle reapportioned testing tasks between them without authorization from NASED.

 

• The lack of communication between CIBER and Wyle allowed some testing to fall through the cracks.

 

• More than 30 items on the EAC assessment checklist indicate failures of either CIBER or Wyle, or both.

 

• The people who have been accepting the validity of CIBER/Wyle’s work for years are the same people who are, only now, finding significant failures in the ITA’s testing procedures and documentation.

Software Testing 101

This assessment report has so much information in its 31 pages that one can quickly become lost in the minutiae, technical shorthand and undefined acronyms, so, let’s step back and consider the art and practice of software testing for a moment. Software testing at its simplest can be described by the simple phrase: “Plan your Work and Work your Plan”. 

The first step for a software testing strategy is to define which things will be tested and which things will not be tested. In the case of voting systems, the 2002 Voting System Standards (VSS) is the definition of what is to be tested. After the requirements of the 2002 VSS are enumerated, the plan should then list all the proposed tests and ensure there is at least one scheduled test to verify that each particular requirement of the 2002 VSS is met. Plan your work. 

After the plan has been reviewed for correctness and completeness, the next steps are to schedule and execute the proposed tests, record the individual test results, record and track the repair of discovered defects, and summarize the individual test results into an overall recommendation. Work your plan.

But two paragraphs from the report’s summary of findings suggest that not only was the CIBER/Wyle’s test matrix inadequate, CIBER doesn’t even have a copy of the matrix.

... during the review, ITA Practice Director indicated that the testing for a product tends to either use vendor developed tests or new tests defined specifically for the product – they [CIBER] have no standard test methods defined.  This makes their testing dependent on the vendor input and vulnerable to unique vendor interpretations …

All the ITAs need to complete a review of the VSS 2002 and the new VVGS 2005 and update the requirements cross-reference matrix to be used to identify which requirements [of the voting system standards] have been tested and where or when. The former matrix developed jointly between the ITAs is missing significant requirements and variations on [those] requirements. (Note: Shawn Southworth, the ITA Practice Director, reports that CIBER does not have that version of the checklist.)

 
One is hard-pressed to decide which portions of the simple maxim: plan your work and work your plan, were followed by the CIBER, Inc.

Some Damning Details

1. The “jointly developed requirements cross-reference matrix” needs to be examined.  The finding of the assessment report is that this checklist is incomplete. This means the CIBER/Wyle team has a systemic blind spot in their test methodology. The requirements of the 2002 VSS which are missing from the cross-reference matrix are routinely untested regardless of the system under test or the vendor paying for the testing. Unfortunately, this matrix is likely considered a trade secret by CIBER and is not available for public inspection or production under the Freedom of Information Act. At this time it is unknown what testing has been routinely skipped, but the assessment report indicates the omitted, and thus untested, requirements are significant requirements.

 

2. On or before July of 2005, the CIBER/Wyle team redefined their ITA roles¹ and apportioned the work differently than the accredited apportionment defined by the NASED Voting Systems Board. Prior to July 2005, Wyle labs did all hardware testing and reviewed the source code all firmware and all application software that ran on a device that recorded ballots or vote totals. Prior to July 2005, CIBER did the source code review and functional testing of any application software running on the central Election Management Server. Then came the labs’ unauthorized reapportionment of work. From July of 2005 the NASED accredited apportionment of work was redefined. From July 2005 to the present CIBER personnel performed all source code reviews and all functional testing, regardless of the hardware platform. Apparently, Tom Wilkey, head of the NASED Voting Systems Board from 1998 to May 2005, Sandy Steinbach Chair of the NASED Voting Systems Board from May 2005 to the present, nor anyone on the technical subcommittee of the NASED Voting Systems Board (Steven V. Freeman, Paul Craft, and Brit Williams) noticed this reapportionment of work between the two halves of the CIBER/Wyle team.  This clear change of responsibilities affects and the accreditation of both the CIBER lab and the Wyle lab.² The change may even invalidate one or both accreditations.

 

3. From the summary of findings of the assessment report: "[The] ITA Practice [of] CIBER is unable to follow their own defined processes and procedures to ensure the quality of their work."

 

4. From the Summary of findings: "Cross-checking between the CIBER and Wyle reports has revealed at times that neither ITA has performed certain tests, expecting the test was done by the other." This cross-checking of reports is precisely what the technical subcommittee of the NASED Voting Systems Board (Craft, Freeman and Williams) claimed it was doing for more than five years. How it is that these gaps in testing are being reported for the first time by a member of this same technical subcommittee?  Moreover, which tests were not performed on which systems?

 

5. Physical Configuration Audits are used to precisely identify the system under test.  This is clearly not being done. (See items 1b, 1f, 2a, 2b, and 3a of the assessment report) Therefore, it is not possible to determine the Entity under Test (EUT) or to assure the state election officials that the system presented to a State for certification by a vendor is or is not the same system presented to the ITA lab for NASED qualification testing.

 

6. The scope of security testing is sharply limited and does not incorporate discoveries from the field.  See Item 2e of the assessment report.

 

7. No independent determination if an item designated by the vendor as a Commercial of the Shelf (COTS) item is actually a COTS item. Item 3j of the assessment report: "Part of Witness Build Documentation. [CIBER] need[s] to develop process. Have form and procedures." The significance of this omission is that under the weak voting system standards (both 2002 VSS and the 2005 VVSG), items designated as COTS are exempt from any testing. Finding 3j in this assessment report seems to indicate items were designated as commercial off the self (and thus untested) based on nothing more than the assertion by the vendor that the item was a commercial off the shelf product.

 

8. "Need to develop. Need to add. Not identified." These and similar phrases appear no less than 30 times in eight pages of findings from the assessment report. How is possible that a test lab authorized to test voting systems for more than 10 years has ANY undeveloped processes or undeveloped procedures, let alone more than 30?

Each of the thirty items that needed to be added, developed or identified are all points of concern, each of which deserves a paragraph unto itself. The list above has been limited to those this author considers most significant. Other software test personnel could, and likely will, come to differing judgments as to which testing flaws contained in report are the most significant. The assessment report is thick with jargon, often uses undefined three-letter acronyms and includes terse references to the requirements of the two standards documents: 2002 VSS and the 2005 VVSG.  Because of this, each of the items on the 8 pages of findings could be expanded into an article of a length similar to this one in order explain the jargon, define the acronyms, include excerpts of requirements, and then explain the relevance and significance of the assessment finding to the voting machine upon which you voted in November 7, 2006.

_______________