In the late 1990s, the outcomeof a battle between two factions within the Post Office triggered the IT problems that, combined with corporate malfeasance, led to what is today known as the Horizon scandal.
If a moment in the Post Office’s automation project can be identified as the trigger of the monumental failures of the past two decades, it was the decision to reject an established retail software package in favour of starting from scratch, according to former Post Office veteran Rupert Lloyd Thomas.
Thomas spent 27 years at the Post Office, with his final one involved in choosing a supplier to provide the electronic point-of-sale (Epos) system in the Horizon project.
The contract to automate thousands of branches was already agreed with British tech company ICL – which was later taken over by Fujitsu, the chosen supplier – but a key decision was still to be made about the Epos software: would the Post Office use an established tried and tested package already in the market, or would it build from scratch?
Make versus buy battle
There were two opposing groups within the Post Office. Lloyd Thomas, along with others, was pushing for the adoption of a commercially available retail system.
“The main issue was, were we going to use some packaged software that had been used elsewhere, or were we going to write one from scratch? I was firmly in the packaged software camp,” he told Computer Weekly.
Lloyd Thomas said the team had a successful track record in implementing commercial software after its decision to move to SAP financials was successful. “I fought the good fight when they wanted to replace the ledgers and won by getting SAP implemented. It had a lot of advantages,” he said.
Lloyd Thomas said he wanted the same approach for the Epos system. “I sat in meeting after meeting, saying, ‘no bespoke, no bespoke, just leave it alone, because once you start kludging it, you can’t rely on software upgrades coming from the supplier’.”
ICL owned a retail system known as GlobalSTORE, which was an option being pushed by the team Lloyd Thomas was on. “You want to rely on software upgrades from the supplier and install them cleanly. If you kludge the system you won’t be able to do it,” he said.
Lloyd Thomas said the “make versus buy” debate went on from March 1996 to September 1996, when his team lost the battle and the Post Office opted for a bespoke system that used middleware from Escher, known as Riposte. It was non-technical Post Office executives who rejected the buy-in approach, he added.
Terrible misgivings
Lloyd Thomas said he had terrible misgivings. “It was akin to gambling,” he told Computer Weekly.
In late 2022, the Post Office Horizon IT Inquiry heard that the teams tasked with writing the Epos software were also not up to the job.
“I wasn’t involved in the prosecution of anybody, but I worked there, and I feel terrible about it,” he said. “I’m not interested in fighting old battles, I just want to help the victims” Rupert Lloyd Thomas, former Post Office veteran
They were even considered “the joke of the building” at Fujitsu/ICL UK, according to software developer David McDonnell, who told the inquiry that, when he arrived, the Epos development team was “like the Wild West” with “no standards, a lack of rules and no design”.
A taskforce set up to investigate problems in Horizon’s Epos development reported that whoever wrote the code “had no understanding of elementary mathematics or the most basic rules of programming”.
Lloyd Thomas, who left the Post Office in 2001 after over a quarter of a century, is speaking out because of his involvement in the choice of supplier. “I wasn’t involved in the prosecution of anybody, but I worked there, and I feel terrible about it,” he said. “I’m not interested in fighting old battles, I just want to help the victims.”
Poor IT decision-making
Poor IT decision-making played a significant role in the Horizon disaster and Lloyd Thomas said a lot of this was down to how “backward” the Post Office was in terms of IT.
Armed with a degree in geography, history and politics from Durham University, he joined the Post Office in 1974, straight out of university.
Lloyd Thomas started working in the marketing department. At the time, he knew little about computers. “All I knew about computing when I was at university was what some of my chums were studying,” he said.
In the late 1970s, Lloyd Thomas transferred to the managing director’s office, where he came across word processing for the first time. “I became immersed in word processing and thought I should really get on this bandwagon. From there, I rapidly discovered that nobody at the Post Office knew much about computing,” he said.
On leaving the managing director’s office around 1980, it was arranged that Lloyd Thomas would spend time with ICL, a supplier of IT to the Post Office.
“One of the reasons they wanted to send me to ICL is that they didn’t have anyone who knew much about computing. The idea was they would get this youngster to go to ICL and come back with knowledge,” he said.
Nobody was listening
But he felt his efforts were to no avail and it was no more than a box-ticking exercise for Post Office managers.
“I wrote a long report on my six months at ICL and they patted me on the back, but it seemed to be just an academic exercise that didn’t have any practical application. It was almost as if I had been at college and didn’t dawn on them that I might be saying something useful,” he said.
“The Post Office loved hiring consultants – the money they spend on them would blow your mind. They would never listen to their own experts. They also loved trials and had them perpetually. They went on and on and on and reached no serious conclusion” Rupert Lloyd Thomas, former Post Office veteran
Lloyd Thomas, who spent time abroad working for the British Post Office Consultancy, which offered services to post offices in other countries, also described a culture of overspending on external consultants and running “perpetual pilot schemes”.
“The Post Office loved hiring consultants – the money they spend on them would blow your mind. They would never listen to their own experts. They also loved trials and had them perpetually. They went on and on and on and reached no serious conclusion.”
He described a trial of an electronic financial transfer point-of-sale system across branches in Southampton, which was “bewitching to people”.
There was also an automation project with four different pilots running with four suppliers: ICL, Burroughs, Philips and NCR.
He said part of the dependency on external consultants resulted from “getting their fingers burnt” in the past. There were some disasters, including what was known as the Thames Valley Pilot in the 1990s.
“They spent a fantastic amount of money, many millions, in the late 1980s on the pilot of an online system connected to the IT centre in Farnborough with hardware on counters,” said Lloyd Thomas. “But it was a turkey. It was so expensive. The cost per transaction was utterly unsustainable and the whole thing got cancelled.”
Early spark for Horizon project
By the 1990s, Horizon was an idea and, according to Lloyd Thomas, there was an intermediate stage which, beginning in 1992, saw the Capture software system used by over 1,000 sub Post Office branches, which computerised accounting, and Ecco+ software in Crown branches for the same purpose.
He said both systems had serious deficiencies.
“Ecco+ was chronically unreliable, it was a cheap and nasty system. I had a lot to do with work trying to get that system fixed.”
He said the “saving grace” with Ecco+ was when there was a fault, it was obvious, and users would know, rather than being shocked down the line when numbers didn’t add up and being blamed for the shortfalls.
The same cannot be said for the Capture system, or the Horizon system after its roll-out in 1999.
Since the airing of the ITV drama and documentary about the Post Office Horizon scandal, former subpostmasters who suffered unexplained losses using the pre-Horizon Capture system have come forward. Some were prosecuted and others were forced to repay unexplained losses.
The Post Office said it is looking into the cases. “We are particularly concerned about allegations of prosecutions, and we are looking into this along with all available facts about Capture, including whether shortfalls could have been caused by faults in this software, and the potential impacts if so,” it said in a statement.
In the 1990s, during a role in charge of post offices, Lloyd Thomas visited a Crown Post Office branch in West London, where he found a filing cabinet with hundreds of floppy disks stored away. “When I asked what they were for, I was told they had crashed so they don’t use them anymore,” he said.
“I came back to London and I couldn’t get anyone interested. They would say, ‘We are getting this new system called Horizon. Why would we spend money mending our old system?’”
Computer Weekly first exposed the scandal in 2009, revealing the stories of seven subpostmasters and the problems they suffered as a result of the Horizon system. Subpostmasters were prosecuted, jailed, sacked and expected to make up for phantom shortfalls, which were later proved to be caused by computer errors (see timeline of Computer weekly articles below).