David Fuhr, CTO at intcube, talks about a security cohort for public administration, where 27 municipalities hardened Windows Server and Active Directory together over 12 weeks. This Help Net Security interview covers how a cohort works, with weekly sessions, homework and measurements at the start and end.
He explains how the group handled settings that can break systems, such as SMB signing, LDAP signing and Kerberos hardening, how changes were tested in a lab, and how rollback planning worked. He also covers who owns the settings afterward and why the group was split in two.
You planned for 10 to 15 municipalities and got 27. What does that number tell you about the state of Active Directory security in local government that no survey would have revealed?
The first thing we learned is that the problem is not awareness. The municipalities that signed up knew they needed to work on their server security, on their Active Directory. Another survey would only show how big the problem is. It wouldn’t help them start in any meaningful way.
So when we came up with the offer, here’s how 10 to 15 of you can work together, simultaneously, efficiently, we didn’t need to convince them much that this is a sensible approach. It was unfamiliar for them, so they needed a moment to get used to the structure and the way of working. In the end we had to split them into two groups, morning and afternoon, because 27 municipal IT organizations wanted to participate.
If the root cause isn’t technical, what is it? Why do known AD weaknesses stay open for years in organizations with one or two admins?
Technically, it’s all solvable. It’s hard, but solvable. Active Directory is an old technology with a lot of baggage: technical debt, organizational debt, process debt, especially in organizations that have been on Microsoft technologies for decades.
You could think resources are the constraint. In a way that’s true, but it’s not money. Even with more money for tools or licenses, you’d still have one or two admins with so many topics and tasks on their plate. What really lacks is motivation. Not that they don’t want to do it. They’re all trying hard to plug the different holes. What’s missing is the motivation of: here’s how we can really get this one thing solved. That’s what makes it compelling. That’s what got everybody on board, and on fire in the end.
The starting point was a statewide security audit that left many municipalities with reports of up to 150 pages. How do you get from that to a plan a two-person team can execute?
Even if the big reports are good, nobody really reads 150 pages, let alone acts on them. It’s a game with bad incentives. You could boil a report down to the essential five to eight pages. But you still need the time and the energy to tackle it. Energy is even better than motivation. That’s why it helps to have a structure and a group, like in sports, or like a roped party on a mountain. You do it together.
How does a cohort work in practice? Why does this get more hardening done than a classic one-on-one engagement?
The magic of a cohort is that you have a group of organizations, in our case municipalities, that share a problem. They have the same problem in a similar way, or at similar stages. For them, we design a program that runs for a predefined number of weeks. Depending on the problem, that could be 12, 16 or 24 weeks.
It has a rhythm. For example, two hours every Wednesday morning or afternoon, plus asynchronous work on a platform, homework as it were. Over those weeks, we work through the different stages of problem solving together. There’s a lot of knowledge transfer. Like in consulting, we as the experts bring the knowledge, tools and templates needed to make it work. But the work is done by everybody solving their own problem in parallel, learning from each other, from the team, and from prior cohorts.
We measure at the beginning and at the end, so the group always sees where it stands and how much is left to reach the common goal. That creates momentum: it’s week nine, three more weeks to go, the last cohort was at this point, we’re almost there. If we really power through, we can get 100 or 120 percent of what we set out to achieve. That makes it motivational.
SMB signing, LDAP signing and Kerberos hardening are known for breaking things. How did the group handle that risk, and how did you decide when an exception was acceptable?
We had different phases in the cohort. In the first phase, apart from understanding things, we set up basic logging and auditing, so everybody could see in their own infrastructure what was going on: which SMB versions were being used, whether LDAP signing was on or not. Once that was set up and working for everyone, we moved on to other topics, and came back to our audit logs a few weeks later to check what we’d found. That’s how we knew we could proceed.
For some organizations, we had to handle exceptions individually. That’s always to be expected. In a cohort of 15, there will be two or three where you sit down one-on-one and work through whatever they need to stay on track with the rest of the group. We talked openly about what could go wrong. We didn’t make the call to accept the residual risk; that was up to each organization. But we had this open space for exchange, so everybody felt confident in the end. The vast majority who took the respective steps had positive experiences.
How did you test changes before they touched 27 production environments, and what did rollback planning look like?
We tested all changes in our own testing environment. We had a little lab modeled after a very simple municipal infrastructure, where we showed everything: not just the standard case, but all the things that could go wrong. Rollback planning happened individually for everybody, because it depends very much on the local infrastructure. And that’s something all the participants were very well versed in. So basically what we did was risk management. Everybody did it for themselves. We provided the methodology and the forum, and talked it through in the group.
Configuration drift is where hardening projects quietly die. Who owns these settings six months from now in a municipality with two IT staff?
There are two answers to that. The first is that the cohort builds a community as a byproduct. Even when the cohort is done, in this case after 12 weeks, people stay in contact. We still have the platform and the exchange channels, and they keep using them to work on things. Not as intensively as before, not weekly, but the community is alive. I won’t claim it lives forever without support, but it’s a basis to work from.
The second is that a single cohort can’t fix all the IT security, or even all the AD security challenges, of one municipality. On the bigger scale, what it needs is a program, a set of cohorts that organizations run through rhythmically. Smaller rhythms inside bigger rhythms, like a pyramid. Each one helps them reach the next level, then move on to the next topic where they have gaps. That way they advance at their own tempo and reach new levels of maturity.
You split the 27 into two groups. Where does the cohort model hit its limits?
I wouldn’t call it a limit. The safe space of a cohort needs a group where everybody can see where the others are. That doesn’t mean everybody talks to everybody; it doesn’t have to be a full mesh. But you want people to listen to each other. Think of school classes: 25 to 35 students is obviously too many. So we aim for 10 to 15 per cohort.
When we got 27 applications for the first ever municipal IT security cohort, we split them into two groups. That turned out to be fertile. The two groups can still learn from each other and share a common space. We can try something in the morning and adapt it a little in the afternoon. So it enhances the single-cohort model.
Download: The IT and security field guide to AI adoption
