Northrop Grumman today apologized for an outage that began last Wednesday and caused 26 Virginia state agencies to lose their Web services, some for more than a week.
"Northrop Grumman deeply regrets the disruption and inconvenience this has caused state agencies and Virginia citizens. Over the last week, we have deployed significant resources and our staff has worked tirelessly in unison with VITA and our partners to thoroughly verify and restore access to data and restore normal operations to the agencies," Linda Mills, president of Northrop Grumman Information Systems, said in a statement.
The Virginia Information Technologies Agency (VITA) outsources the management of its data centers to Northrop Grumman through a 10-year, $2.4 billion contract that it signed in 2005.
The outage affected 13% of the Commonwealth's file servers.
Northrop Grumman did not respond to a request for additional details surrounding the outage.
VITA's contract with Northrop Grumman has been criticized in the past for a number of project delays, cost overruns and performance problems that included other service outages.
After an audit by Virginia's Joint Legislative Audit and Review Commission last year, VITA's contract with Northrop Grumman was modified, resulting in more stringent performance requirements and greater accountability. The contract, however, also boosted payments to Northrop Grumman by $105 million over nine years.
Earlier this year, after a long history of problems within VITA, Virginia Gov. Bob McDonnell placed the agency's CIO under his direct authority, appointing former Virginia lawmaker Sam Nixon to the position. Nixon, who co-authored the legislation establishing VITA in 2003, was charged with the task of fixing the IT problems.
Gov. McDonnell has called for an outside investigation of the latest incident.
On Aug. 25, the failure of a storage area network (SAN) caused Web site outages at 26 of Virginia's 83 state agencies. VITA originally said 27 agencies had lost connectivity, but later admitted it had included Northrop Grumman's systems in that count.
As of Tuesday, all but three agency sites had been restored, leaving the commonwealth's Department of Motor Vehicles (DMV), the Department of Taxation and the state Board of Elections without services.
Yesterday, a VITA spokesman said the state's tax department was back online, but the DMV was still unable to process drivers' licenses at its customer centers. The State Board of Elections (SBE) has is now partially operational.
At the root of the outage was an EMC DMX-3 storage array -- the vendor's flagship product, according to Virginia Secretary of Technology Jim Duffey.
The EMC SAN malfunction has been blamed on a memory card failure. A backup SAN that was supposed to act as a fail-over system then also experienced problems, according to published reports.