Post by Sapphire Capital on Mar 4, 2009 9:24:20 GMT 4
Report: Diebold Voting System Has 'Delete' Button for Erasing Audit Logs
Published on 03-03-2009
Source: Wired
Diebold
Following three months of investigation, California's secretary of state has released a report examining why a voting system made by Premier Election Solutions (formerly known as Diebold Election Systems) lost about 200 ballots in Humboldt County during the November presidential election.
But the most startling information in the state's 13-page report(http://www.sos.ca.gov/elections/voting_systems/sos-humboldt-report-to-eac-03-02-09.pdf) (.pdf) is not about why the system lost votes, which Threat Level previously covered in detail (http://blog.wired.com/27bstroke6/2008/12/unique-election.html), but that some versions of Diebold's vote tabulation system, known as the Global Election Management System (GEMS), include a button that allows someone to delete audit logs from the system.
Auditing logs are required under the federal voting system guidelines, which are used to test and qualify voting systems for use in elections. The logs record changes and other events that occur on voting systems to ensure the integrity of elections and help determine what occurred in a system when something goes wrong.
"Deleting a log is something that you would only do in de-commissioning a system you're no longer using or perhaps in a testing scenario," says Princeton University computer scientist Ed Felten, who has studied voting systems extensively. "But in normal operation, the log should always be kept."
Yet the Diebold system in Humboldt County, which uses version 1.18.19 of GEMS, has a button labeled "clear," that "permits deletion of certain audit logs that contain – or should contain – records that would be essential to reconstruct operator actions during the vote tallying process," according to the California report.
The button is positioned next to the "print" and "save as" buttons (see image above), making it easy for an election official to click on it by mistake and erase crucial logs.
In fact, the report says, this occurred recently in a California county when an official, while attempting to print out a copy of a so-called "poster log," inadvertently deleted it instead.
The system provides no warning to the operator that clicking on the button will result in permanent deletion of records in the log, nor does it require the operator to confirm the action before executing it.
Apparently Premier/Diebold was aware that having a "clear" button on its system was a bad idea. According to California's report, one of the system's developers wrote in an e-mail in 2001: “dding a 'clear' button is easy, but there are too many reasons why doing that is a bad idea.” Yet the company included the button in its system anyway.
The button was removed from version 1.18.20 of the software and following, but Premier/Diebold never went back to jurisdictions using previous versions to upgrade them, and version 1.18.19 is still used in three California counties as well as in other states. It's unclear how many previous versions of the software had the button, or why it was included in the first place.
According to the report:
The “Clear” buttons . . . allow inadvertent or malicious destruction of critical audit trail records in all GEMS version 1.18.19 jurisdictions, risking the accuracy and integrity of elections conducted using this voting system. Five years after the company recognized the need to remove the “Clear” buttons from the GEMS audit log screens, not only Humboldt, San Luis Obispo and Santa Barbara Counties in California but jurisdictions in other parts of the country, including several counties in Texas and Florida, continue to use GEMS version 1.18.19. . . .
The report states that the inclusion of the button violated the federal voting system standards under which the Premier/Diebold system was qualified to be used in elections. The standards require that voting system software automatically creates and permanently retains electronic audit logs of important system events that occur on the machine.
Premier/Diebold did not respond to a request for comment.
The "clear" button isn't the only problem with the auditing log in the Premier/Diebold system. I wrote previously about other issues with the logs -- for example, they don't record significant events that occur in the tabulation system, such as when someone deletes votes from the software.
The California report states that the "clear" button, along with other problems with the auditing logs as well as the software flaw that caused the system to lose votes in Humboldt County (see below for more information on that flaw), should have been red flags to the testing laboratories that certified the system and should have been sufficient to "fail" the system and prevent it from being used in any federal election.
As the report points out, under the voting system standards (VSS) "each of the errors and deficiencies in the GEMS version 1.18.19 software described in this report standing alone would warrant a finding by an Independent Testing Authority (ITA) of 'Total Failure' (indicated by a score of 1.0) had the flaw been detected. Under the 1990 VSS, a finding of 'Total Failure' required failure of the voting system."
"Presumably some organization, some lab, looked at this system and decided they thought it complies with the standard," says Felten. "And, obviously, they were wrong. Any state that uses GEMS should be looking at this seriously."
The findings raise questions about the auditing logs on voting systems made by other vendors and about what states that use the Premier/Diebold system will do now that they know their voting software does not create an adequate audit trail to ensure the integrity of an election.
California's secretary of state will be holding a public hearing on March 17 (.pdf) to discuss the report and whether version 1.18.19 of GEMS should be decertified in the state. The state can't order counties not to use the software, but decertifying this software version will force counties to upgrade to different versions.
As for addressing the fundamental problems with the auditing logs in all versions of the GEMS software, a spokeswoman for the secretary of state's office said only that the state sent the report to the federal Election Assistance Commission to post on its web site and communicate the issue to election officials in other jurisdictions.
A spokeswoman for the EAC told Threat Level that the commission has no authority to address problems with voting systems that were tested and qualified before it assumed responsibility for voting machines, even if those machines are in violation of the voting system standards. In 2002, Congress gave the EAC oversight responsibility for the testing and qualification of voting systems, but the commission has yet to shepherd a voting system through testing and certification. Prior to the EAC assuming responsibility for voting systems, the National Association of State Election Directors voluntarily assumed the oversight task and it was under NASED's watch that all of the voting systems currently in use in the country were tested and certified.
"There's no regulatory action that we could take," says EAC spokeswoman Jeannie Layson. "But certainly in the area of notification, when the voting sytem reports come out, the commissioners make sure that the test labs and independent reviewers who look at the test reports are aware of all that information."
The lab that was responsible for testing and qualifying GEMS version 1.18.19 with the "clear" button is Colorado-based Ciber, Inc. In 2007, the lab was barred from testing voting systems for not following quality-control procedures and for failing to document that it was conducting all tests. The EAC restored the lab's accreditation to test voting systems last year.
Ciber did not respond to a call for comment about its examination of the Premier/Diebold system and its approval of the "clear" button.
As mentioned above, the California report is the result of an investigation into what occurred in Humboldt County during the presidential election.
After county officials had already certified their election results to state officials, they discovered that the tabulation software they used to tally votes had dropped 197 paper vote-by-mail ballots from the totals at one precinct. The system did so without giving any warning or message to election officials that it was doing so. Humboldt uses a central-count optical-scan system made by Premier/Diebold.
The vendor acknowledged that a programming flaw in its software caused the system to randomly and silently delete votes and disclosed that it had known about the problem since October 2004 and provided some election officials with a workaround, though Humboldt County election director Carolyn Crnich had never been told of the problem.
The issue involved a programming error in version 1.18.19 of GEMS that caused ballots to be randomly dropped from the system. The GEMS software is used to tabulate votes on both touch-screen and optical-scan machines, but the problem only occurred when the software was used to tabulate votes scanned on a central-count optical-scan system -- a high-speed optical scanner that is used in a county's election headquarters, as opposed to precinct-based optical scanners that are used at polling places.
The California secretary of state investigated and confirmed that a flaw in the GEMS software can automatically delete the first batch of ballots scanned into the system if officials delete a subsequent batch -- something that occurs on a regular basis when officials make a mistake during scanning. The system provided no notice to officials whenever it deleted such ballots.
As the California report notes, the loss of votes in Humboldt County could have been much greater and was limited only because election officials had scanned only 197 ballots in their first batch.
Although Premier/Diebold knew about the problem for years, it didn't notify the federal Election Assistance Commission or the National Association of State Election Directors, which oversaw the testing and certification of voting systems at the time the Premier/Diebold system was certified, so that NASED could notify election officials around the country about the problem.
Instead, the report says, the company sent a "vague email" to election officials in the 11 California counties that, at the time, were using that version of the GEMS software with a central count optical-scan system. The e-mail, previously published here, provided officials with a workaround for the problem. But the e-mail never told officials why it was important for them to do the workaround -- that is, it never explained that there was a programming flaw in the software and that if they didn't do the workaround, the system would be at risk of silently dropping votes.
An employee in Humboldt County received the e-mail, but never wrote the workaround instructions into the office's election procedures and never told Crnich about the issue before he left to go work in another county. As a result, Humboldt was vulnerable to the flaw during the presidential election.
The flaw was fixed in version 1.18.24 of the software in May 2005. But until that occurred, Premier/Diebold continued to allow other jurisdictions across the country to use at least five flawed versions of the software and never explained the problem or the workaround in its user documentation. Diebold has said that no jurisdiction outside California used these versions of GEMS with a central count optical-scan system and therefore were not at risk from the flaw. California officials backed this statement in their report. But even when the flaw was fixed in version 1.18.24, the vendor allowed California counties to continue to use the flawed software rather than upgrade them to the fixed version. The company never informed state officials about the problem with its system.
Secretary of State Debra Bowen has sponsored legislation that would require voting machine vendors to notify the state in writing (.pdf) any time it discovers a problem with its voting system. The vendor would have to notify the state, and any California jurisdiction using the voting system, within five working days of discovering a flaw in software or hardware. The bill also requires vendors to disclose any flaws it already knows about systems that are currently in use in the state. These reports will then be submitted to the Election Assistance Commission so that officials in other states will know about them as well. The bill also provides for civil penalties of $10,000 per violation against vendors for undisclosed flaws or for making unauthorized changes to a voting system.
Kate Folmar, spokeswoman for the secretary of state's office, said Bowen hopes that the bill, if passed, "could become a model for other states for dealing with similar anomalies and problems that pop up with their voting systems."
Published on 03-03-2009
Source: Wired
Diebold
Following three months of investigation, California's secretary of state has released a report examining why a voting system made by Premier Election Solutions (formerly known as Diebold Election Systems) lost about 200 ballots in Humboldt County during the November presidential election.
But the most startling information in the state's 13-page report(http://www.sos.ca.gov/elections/voting_systems/sos-humboldt-report-to-eac-03-02-09.pdf) (.pdf) is not about why the system lost votes, which Threat Level previously covered in detail (http://blog.wired.com/27bstroke6/2008/12/unique-election.html), but that some versions of Diebold's vote tabulation system, known as the Global Election Management System (GEMS), include a button that allows someone to delete audit logs from the system.
Auditing logs are required under the federal voting system guidelines, which are used to test and qualify voting systems for use in elections. The logs record changes and other events that occur on voting systems to ensure the integrity of elections and help determine what occurred in a system when something goes wrong.
"Deleting a log is something that you would only do in de-commissioning a system you're no longer using or perhaps in a testing scenario," says Princeton University computer scientist Ed Felten, who has studied voting systems extensively. "But in normal operation, the log should always be kept."
Yet the Diebold system in Humboldt County, which uses version 1.18.19 of GEMS, has a button labeled "clear," that "permits deletion of certain audit logs that contain – or should contain – records that would be essential to reconstruct operator actions during the vote tallying process," according to the California report.
The button is positioned next to the "print" and "save as" buttons (see image above), making it easy for an election official to click on it by mistake and erase crucial logs.
In fact, the report says, this occurred recently in a California county when an official, while attempting to print out a copy of a so-called "poster log," inadvertently deleted it instead.
The system provides no warning to the operator that clicking on the button will result in permanent deletion of records in the log, nor does it require the operator to confirm the action before executing it.
Apparently Premier/Diebold was aware that having a "clear" button on its system was a bad idea. According to California's report, one of the system's developers wrote in an e-mail in 2001: “dding a 'clear' button is easy, but there are too many reasons why doing that is a bad idea.” Yet the company included the button in its system anyway.
The button was removed from version 1.18.20 of the software and following, but Premier/Diebold never went back to jurisdictions using previous versions to upgrade them, and version 1.18.19 is still used in three California counties as well as in other states. It's unclear how many previous versions of the software had the button, or why it was included in the first place.
According to the report:
The “Clear” buttons . . . allow inadvertent or malicious destruction of critical audit trail records in all GEMS version 1.18.19 jurisdictions, risking the accuracy and integrity of elections conducted using this voting system. Five years after the company recognized the need to remove the “Clear” buttons from the GEMS audit log screens, not only Humboldt, San Luis Obispo and Santa Barbara Counties in California but jurisdictions in other parts of the country, including several counties in Texas and Florida, continue to use GEMS version 1.18.19. . . .
The report states that the inclusion of the button violated the federal voting system standards under which the Premier/Diebold system was qualified to be used in elections. The standards require that voting system software automatically creates and permanently retains electronic audit logs of important system events that occur on the machine.
Premier/Diebold did not respond to a request for comment.
The "clear" button isn't the only problem with the auditing log in the Premier/Diebold system. I wrote previously about other issues with the logs -- for example, they don't record significant events that occur in the tabulation system, such as when someone deletes votes from the software.
The California report states that the "clear" button, along with other problems with the auditing logs as well as the software flaw that caused the system to lose votes in Humboldt County (see below for more information on that flaw), should have been red flags to the testing laboratories that certified the system and should have been sufficient to "fail" the system and prevent it from being used in any federal election.
As the report points out, under the voting system standards (VSS) "each of the errors and deficiencies in the GEMS version 1.18.19 software described in this report standing alone would warrant a finding by an Independent Testing Authority (ITA) of 'Total Failure' (indicated by a score of 1.0) had the flaw been detected. Under the 1990 VSS, a finding of 'Total Failure' required failure of the voting system."
"Presumably some organization, some lab, looked at this system and decided they thought it complies with the standard," says Felten. "And, obviously, they were wrong. Any state that uses GEMS should be looking at this seriously."
The findings raise questions about the auditing logs on voting systems made by other vendors and about what states that use the Premier/Diebold system will do now that they know their voting software does not create an adequate audit trail to ensure the integrity of an election.
California's secretary of state will be holding a public hearing on March 17 (.pdf) to discuss the report and whether version 1.18.19 of GEMS should be decertified in the state. The state can't order counties not to use the software, but decertifying this software version will force counties to upgrade to different versions.
As for addressing the fundamental problems with the auditing logs in all versions of the GEMS software, a spokeswoman for the secretary of state's office said only that the state sent the report to the federal Election Assistance Commission to post on its web site and communicate the issue to election officials in other jurisdictions.
A spokeswoman for the EAC told Threat Level that the commission has no authority to address problems with voting systems that were tested and qualified before it assumed responsibility for voting machines, even if those machines are in violation of the voting system standards. In 2002, Congress gave the EAC oversight responsibility for the testing and qualification of voting systems, but the commission has yet to shepherd a voting system through testing and certification. Prior to the EAC assuming responsibility for voting systems, the National Association of State Election Directors voluntarily assumed the oversight task and it was under NASED's watch that all of the voting systems currently in use in the country were tested and certified.
"There's no regulatory action that we could take," says EAC spokeswoman Jeannie Layson. "But certainly in the area of notification, when the voting sytem reports come out, the commissioners make sure that the test labs and independent reviewers who look at the test reports are aware of all that information."
The lab that was responsible for testing and qualifying GEMS version 1.18.19 with the "clear" button is Colorado-based Ciber, Inc. In 2007, the lab was barred from testing voting systems for not following quality-control procedures and for failing to document that it was conducting all tests. The EAC restored the lab's accreditation to test voting systems last year.
Ciber did not respond to a call for comment about its examination of the Premier/Diebold system and its approval of the "clear" button.
As mentioned above, the California report is the result of an investigation into what occurred in Humboldt County during the presidential election.
After county officials had already certified their election results to state officials, they discovered that the tabulation software they used to tally votes had dropped 197 paper vote-by-mail ballots from the totals at one precinct. The system did so without giving any warning or message to election officials that it was doing so. Humboldt uses a central-count optical-scan system made by Premier/Diebold.
The vendor acknowledged that a programming flaw in its software caused the system to randomly and silently delete votes and disclosed that it had known about the problem since October 2004 and provided some election officials with a workaround, though Humboldt County election director Carolyn Crnich had never been told of the problem.
The issue involved a programming error in version 1.18.19 of GEMS that caused ballots to be randomly dropped from the system. The GEMS software is used to tabulate votes on both touch-screen and optical-scan machines, but the problem only occurred when the software was used to tabulate votes scanned on a central-count optical-scan system -- a high-speed optical scanner that is used in a county's election headquarters, as opposed to precinct-based optical scanners that are used at polling places.
The California secretary of state investigated and confirmed that a flaw in the GEMS software can automatically delete the first batch of ballots scanned into the system if officials delete a subsequent batch -- something that occurs on a regular basis when officials make a mistake during scanning. The system provided no notice to officials whenever it deleted such ballots.
As the California report notes, the loss of votes in Humboldt County could have been much greater and was limited only because election officials had scanned only 197 ballots in their first batch.
Although Premier/Diebold knew about the problem for years, it didn't notify the federal Election Assistance Commission or the National Association of State Election Directors, which oversaw the testing and certification of voting systems at the time the Premier/Diebold system was certified, so that NASED could notify election officials around the country about the problem.
Instead, the report says, the company sent a "vague email" to election officials in the 11 California counties that, at the time, were using that version of the GEMS software with a central count optical-scan system. The e-mail, previously published here, provided officials with a workaround for the problem. But the e-mail never told officials why it was important for them to do the workaround -- that is, it never explained that there was a programming flaw in the software and that if they didn't do the workaround, the system would be at risk of silently dropping votes.
An employee in Humboldt County received the e-mail, but never wrote the workaround instructions into the office's election procedures and never told Crnich about the issue before he left to go work in another county. As a result, Humboldt was vulnerable to the flaw during the presidential election.
The flaw was fixed in version 1.18.24 of the software in May 2005. But until that occurred, Premier/Diebold continued to allow other jurisdictions across the country to use at least five flawed versions of the software and never explained the problem or the workaround in its user documentation. Diebold has said that no jurisdiction outside California used these versions of GEMS with a central count optical-scan system and therefore were not at risk from the flaw. California officials backed this statement in their report. But even when the flaw was fixed in version 1.18.24, the vendor allowed California counties to continue to use the flawed software rather than upgrade them to the fixed version. The company never informed state officials about the problem with its system.
Secretary of State Debra Bowen has sponsored legislation that would require voting machine vendors to notify the state in writing (.pdf) any time it discovers a problem with its voting system. The vendor would have to notify the state, and any California jurisdiction using the voting system, within five working days of discovering a flaw in software or hardware. The bill also requires vendors to disclose any flaws it already knows about systems that are currently in use in the state. These reports will then be submitted to the Election Assistance Commission so that officials in other states will know about them as well. The bill also provides for civil penalties of $10,000 per violation against vendors for undisclosed flaws or for making unauthorized changes to a voting system.
Kate Folmar, spokeswoman for the secretary of state's office, said Bowen hopes that the bill, if passed, "could become a model for other states for dealing with similar anomalies and problems that pop up with their voting systems."