The British Library has published extensive details of its devastating experience at the hands of the Rhysida ransomware gang, revealing how the cyber criminals likely accessed its systems in the first place, the effects of the cyber attack, its response and the lessons it has learned.
The British Library’s systems were attacked by an affiliate of the Rhysida ransomware-as-a-service (RaaS) gang in the autumn of 2023, resulting in significant disruption to the organisation’s services, which has still not been fully resolved. The gang also stole 600GB of data, including details of service users, which was leaked when the British Library refused to engage.
Roly Keating, chief executive of the British Library, said the organisation hoped that opening up and opting for full transparency over the incident would help other organisations plan and protect themselves against similar cyber attacks.
“The threat of aggressive and disruptive cyber attacks is higher than it has ever been, and the organisations behind these attacks are increasingly advanced in their techniques and ruthless in their willingness to destroy whole technical systems,” he said.
“This is of especial importance for libraries and all those institutions who share our mission to collect and make accessible knowledge and culture in digital form, and preserve it for posterity. Though the motive of the attack on the British Library appears to have been purely monetary, it functioned as, effectively, an attack on access to knowledge.
“Wherever possible … we have tried to err on the side of openness, and not everything here makes comfortable reading for ourselves as an organisation,” said Keating. “We have significant lessons to learn.
“We are also conscious of our duty as data controllers and deeply regret the loss of control of some personal data, for which we apologise wholeheartedly to everyone affected,” he said. “If the outcome is increased resilience and protection against attack for the UK collections sector and others, then at least one good thing will have emerged from this deeply damaging criminal attack.”
Timeline of an attack
Such was the scale of the destruction they wrought, it may never be known precisely when the Rhysida gang gained access to its systems, but the British Library said that according to forensic analysis, it may have been on 25 October 2023, six days before it confirmed a cyber attack.
It revealed that its security manager received an alert about possible suspicious activity in the early hours of 26 October, but that this activity was blocked. The security manager escalated this for investigation, but no further malicious activity was found, and the account was then unblocked following a password reset. With the benefit of hindsight, this appears to have been Rhysida performing recon.
Rhysida’s exact entry point onto the network has also not been identified thanks to the damage they caused and the obfuscation they employed, but the first detected access was at the Terminal Services server, put in place in 2020 to enable external partners and IT support suppliers to access the network, which replaced an insecure remote access system in the early days of the Covid-19 pandemic. The investigators therefore believe Rhysida probably compromised a privileged account belonging to someone outside the British Library via a phishing or spear-phishing attack.
The British Library said it had been aware of the risk of something like that happening and had been in the process of reviewing and tightening its security provisions related to third-party access, but that this work had not been completed as of October 2023. Additionally, it had failed to apply multi-factor authentication (MFA) to the Terminal Services server – even though it had introduced MFA in 2020 across its wider estate, for reasons of cost and practicality, connectivity to its domain was out-of-scope of that project.
The British Library first learned it had been affected by a ransomware attack on the morning of Saturday 28 October, when a member of the IT team found they were unable to access the network. Over the subsequent hours, the incident was swiftly escalated and crisis management plans swung into action.
By that afternoon, the National Cyber Security Centre (NCSC) had been involved, and was assisting with incident handling and communications. It also learned that Jisc had identified unusual data traffic volumes leaving the Library’s estate at 1:30am on 28 October, likely the data exfiltration in progress.
A day later, on the afternoon of 29 October, it confirmed via X it was experiencing an outage, and two days later, on 31 October, it revealed this was the result of a cyber incident, at which point the incident began to pick up mainstream media coverage.
As to its engagement with Rhysida, the British Library confirmed in its report widespread speculation that it had not cooperated with its attackers.
“The Library has not made any payment to the criminal actors responsible for the attack, nor engaged with them in any way,” the report reads. “Ransomware gangs contemplating future attacks such as this on publicly funded institutions should be aware that the UK’s national policy, articulated by NCSC, is unambiguously clear that no such payments should be made.”
Effective crisis management
On the whole, the British Library said, its crisis-management plans performed well, with a practiced Gold/Silver command structure sliding into place, convening senior technical staff, external advisors, and the Library’s data protection officer and senior management, all of whom came together to coordinate the technical response, temporary workarounds where possible, and crisis communications.
Throughout the process, extensive support was provided both through the Department for Culture, Media and Sport (DCMS), and the NCSC, which helped the British Library keep readers, staff and stakeholders, including journalists, informed without sharing any detail that could help Rhysida. For internal comms, this meant resorting to cascading information through email or WhatsApp, while external updates came largely in the form of social media updates.
Once it was determined safe to do so, the British Library’s teams started contacting readers, supporters and others on its mailing lists, signposting NCSC guidance and incorporating user feedback to build more effective FAQs and keep its interim website updated. It was also able to keep a tight lid on what was told to whom when, and made sure all staff had sight of external comms prior to making them public.
It said proactive engagement with management and the Library’s trade unions also helped address staff concerns and effectively disseminate grassroots-level information and advice externally.
Rebuilding the British Library
With a diverse and complex technology estate and, as we have seen, a high number of legacy products, the British Library was always going to be faced with a complex reconstruction task in the case of a major event, and candidly, this appears to have been something the organisation was aware of before the attack, but it often lacked the funding or the impetus to do much about it.
It now believes the quirky nature of its IT estate contributed significantly to the severity of the attack, gifting Rhysida more access than they should have been able to have in a more modern design, among other things.
Making matters worse, besides the exfiltration of data and encryption of servers, Rhysida also destroyed servers to inhibit system recovery, and it was this stage of the attack that caused the most damage to the British Library, which now believes that although it will be possible to restore all of the data, it has no viable infrastructure to be able to do so – this system rebuild is expected to be completed in April 2024.
It admitted its vulnerability to such an attack had been exacerbated by reliance on old legacy applications that can’t now be fixed, either because they are completely obsolete, have been end-of-lifed, or cannot be run securely. Many systems need to be rebuilt from scratch.
But looking on the bright side, the British Library said it had a golden opportunity to transform how it uses and manages technology, adopting and embedding security best practice, and implementing policies and processes fit for a public organisation in the 2020s.
Indeed, it could go on to become a beacon of good practice for its peers. Among many other things, the British Library wants its new IT estate to incorporate best-practice network design, including segmentation and defence-in-depth approaches; a hybrid compute landscape; role-based access controls and least privilege policies; a more robust and resilient backup service with immutable, air-gapped and off-site copies; a holistic and integrated security suite covering the whole organisation, with managed security services for incident detection and response; MFA; improvements in incident, event and vulnerability management; and better IT lifecycle and software delivery governance.
As to things that readers will see, it also proposes to consolidate a number of key systems with more user-centric applications, centralising and replacing an old platform and legacy catalogues, reader registration, digital preservation and enquiries management. Multiple customer data systems will also be consolidated into a new data management and reporting architecture.
Lessons learned
Looking ahead, the British Library said there was still much work to be done, and new risks to be accounted for. Its change programme and new focus on cyber security will increase the need to foster an improved security culture internally, with management buy-in and ongoing support, for example.
Elsewhere, its already-stretched IT teams will need more capacity, and there are incumbent risks in moving more systems to the cloud, as it proposes to do.
Appropriate change management will need to be the watchword throughout the coming months, and this is set against a backdrop of increased risk from gangs such as Rhysida – having been a target once, many organisations frequently find other criminal groups take an interest.
The British Library said many of the other institutions overseen by DCMS and the wider cultural sector would likely have similar risks in terms of investment in security, legacy systems and overworked IT staff
“Investment, boldness and relentless focus are all needed to ensure that we are as secure as we can be against this threat, as the cost of investing in prevention is outweighed by the risk of failing to prevent,” the report reads. “Although the security measures we had in place on 28 October 2023 were extensive and had been accredited and stress-tested, with the benefit of hindsight, there is much we wish we had understood better or had prioritised differently.”
As such, the British Library has shared a list of early lessons that others may wish to incorporate into their thinking:
- Enhance network monitoring on old networks. The British Library had a modern system in place but it couldn’t monitor or protect properly because the legacy network topology hindered its effectiveness;
- Retain external expertise to improve resilience, speed of response and incident analysis capabilities early on;
- Implement and enforce MFA across all systems, especially those used by suppliers;
- Enhance intrusion response processes, conducting in-depth reviews after even the smallest signs of an intrusion;
- Implement proper network segmentation. Had the British Library done this, Rhysida would likely have caused far less damage;
- Implement and practice business continuity plans;
- Try to think more holistically about risk, flagging any and all IT security risk to the appropriate levels. The British Library said it had been doing this well for out-of-appetite security risks, but had been missing a lot of low-level signals;
- Keep on top of legacy systems and lifecycle management, and prioritise fixing issues that arise from legacy kit;
- Enthusiastically invest in backups and recovery capabilities;
- Clue the board in on risk to enable them to make better buying decisions, and ensure there is cyber-specific representation on the board;
- Train staff properly, and regularly top up their knowledge;
- Manage staff and user wellbeing;
- Review acceptable personal use of IT. During the investigation, the British Library found Rhysida had been scanning the network specifically for keywords such as ‘passport’ or ‘personal’ to target personal items stored by staff, which was permitted at the time.
- Collaborate and share information with others in your sector;
- And finally, implement government standards and policies. The British Library in fact became Cyber Essentials Plus certified in 2019, but changes to the scheme in 2022 meant it dropped out of compliance because it needed to replace some legacy systems.