Earlier this month, two public sector IT systems in the US state of Pennsylvania lost data – and human error was blamed.
One dataset lost was a police department application for managing and logging evidence in criminal proceedings. The other concerned user-generated pension estimates.
But if there was human error, is it fair to blame – and fire, as happened in this case – the administrator who carried out the maintenance and lost the data? Or do the fairly regular news stories of data lost during routine IT work point to failings at the level of policy and procedure, and especially the importance of backup?
How did Pennsylvania lose police data?
Of the two incidents, the loss of police evidence recording appears far more serious because of its impact on the workings of the justice system, but the police were keen to point out that no physical evidence was lost. Only, presumably, records of its submission to the police system.
The state administration office blamed “human error” for “limited data loss” during “routine server maintenance” on 1 and 3 January and in 77 servers. At the time of writing, not all the data had been recovered.
The state police declared one of two computer systems used to track evidence was back online by 15 January. Meanwhile, some county officials downplayed the data loss. One said they had a paper trail of evidence anyway; the other had their own forensics lab. Neither response really tackles the key point – which is that such data should not be irretrievably lost in the first place.
What is the importance of backup in a data protection strategy?
The fundamental takeaway from this example is that organisations should always have backups of their data – and those backups should be stored away from the primary systems on which data is processed.
In this case – according to the scant information available – it looks like some server maintenance took place. So far, so humdrum. But, server maintenance, upgrades, and so on, can go wrong. That is leaving aside that, in extreme cases, buildings with servers in them can flood, burn down, or be hit by other catastrophes.
For that reason, it is a fundamental of IT data protection that organisations should always have a second copy of the data, backed up and preferably copied off-site. It’s pretty safe to assume that in the case of Pennsylvania’s police evidence records, there was no backup. If there had been, data could surely have been restored in a matter of days at the most.
So, was there a human error element to this data loss? Almost certainly, the administrator responsible for server maintenance went ahead with something without having backups to roll back to if something went wrong.
But the fundamental error does not lie there.
In fact, the bigger error here is what appears to be a failure of the state’s IT operation to ensure its servers were protected, with backups that were safely stored on a different system altogether, preferably off-site and regularly tested.
Perhaps also, the mention only of “servers” indicates that there is no shared storage, like a SAN or NAS, that would have kept some separation between compute and storage. Such an architecture may have afforded some level of protection from error on servers while leaving data intact.
So, the error was human – but at the level of IT strategy and ensuring backups are regularly taken and stored in a safe place to allow rapid roll-back should systems lose data.