Wiring a Winning Security Organization

Gene Kim and Dr. Steven J. Spear recently published Wiring the Winning Organization. You may recognize Gene’s name from his prior work on The Phoenix Project and Accelerate.

The book’s premise, in a sentence:

I found a variety of lessons for security teams buried in this simple statement.

These three ideas form the book’s core thesis.

Slowification is about slowing operational tempo in order to improve problem solving and allow for learning.

Simplification is about incrementalizing, modularizing, and linearizing problems so that they are well-bounded, with limited factors, and can be tackled by smaller teams.

• Incrementalization: Split novel units of work from known ones, into self-contained and coherent elements. Attack novelty in increments, versus all at once.

• Modularization: Partition integrated systems into simpler coherent pieces. Decrease coupling via clear, stable, and well-defined interfaces. Allow for independence of action and parallelization, while decreasing coordination costs.

• Linearization: Partition complex operations into independent, decoupled workflows, based on their objectives. Prevent resource sharing across workflows, and define handoffs between steps within workflows.

Amplification is all about feedback, identifying and calling out problems early, loudly, and consistently. This allows problems to be swarmed and contained before they become systemic.

Slowification makes problem-solving easier, by moving as much of it as possible to the safety and pace of the planning and practice stages.

In Security, one of the best applications of Slowification is in the incident management process.

Poorly run incidents often bring out the worst of an organization’s inherent entropy towards chaos. Slowification can:

1. ensure preparation and planning are done safely and well in advance

2. aid effective response in the moment through judicious introduction of “pauses.”

Tabletop scenarios (like the amazing @badthingsdaily), incident response exercises, and red team engagements all allow you to move out of the fast-paced and unforgiving world of an incident, into a controllable and repeatable practice environment.

You can also control pace and introduce pauses to your actual incidents, another form of Slowification. This is the role of a trained Incident Commander:

• asking questions, clarify jargon, and make sure responders understand each other

• keep the bigger picture in mind, advocating for mitigation first

• shutting down freelancers

• reducing noise

Simplification reduces the complexity of the problems themselves through incrementalization, modularization, and linearization.

This project contained a mix of novel and well understood elements. We incrementalized the project by defining coherent elements, and splitting novelty for known work within each:

• Signing commits was a well understood approach, but adopting a signing tool would be novel, as was integrating with our custom X.509 Okta Device Trust certificates.

• Deploying a system to verify commits using Lambda was well paved, but the exact verification criteria was still getting clarified.

• Improving the representation of signed commits in the Github UI was well understood.

• Handling the most common git client flows was well understood, but there was novelty in heterogeneous git clients (such as those supporting commit stacks, GUI clients, etc.)

Modularization was accomplished through the early establishment of stable interfaces between, local clients, Github, and our verification service. Additionally, the verification service itself was built with a clear interface for adding additional verification methods, making it extensible independently and in parallel.

This allowed us to parallelize and decentralize work on supporting various clients, as well as developing the heuristics for different bot identities. It also allowed for adoption of the verification infrastructure for new use cases with low coordination. For example, we ended up enhancing the commit verification with additional rigor around stale approvals to protect configuration files for our Terraform Automation.

Linearization was accomplished primarily across projects, versus within this project specifically. Projects were broken down into milestones with clear handoffs. Clear DRIs ran independent, decoupled workstreams. Time was allocated to limit cognitive overhead, by having every individual work on projects in series versus in parallel.

Amplification combats the normalization of deviance by highlighting problems explicitly, early, and often.

The book shares the concept of Toyota’s andon cord: if someone on a production line identifies a problem (defect, malfunction, safety issue), they can pull to cord to request help from a supervisor or team lead.

• it provides the practice (see Slowification) necessary to run more significant incidents well

• it ensures you formalize the process of collecting long term insights

• it can clarify priority using explicit, and well understood incident severity ratings

• it brings the right help into the room, based on the severity

There are many other opportunities and tactics for amplification:

• During onboarding, your security training can emphasize how to get in touch with security and highlight that security will never be upset about a false positive if someone has a concern.

• In Slack, consider a reacji to allow anyone to flag something for security attention. This lowers the bar and normalizes getting security eyes on questions or threads!

• When you establish invariants, monitor them. If they are only enforced via policy or practice, make sure you do not normalize deviance. If you are asserting an invariant that isn’t being implemented in practice, it’s better to reconcile with reality.

• Implement PR-time feedback with SAST, and have security periodically review those findings to synthesize generic controls and lessons from those local findings

• “Socializing the impact of a bug across the company is a powerful and underused tool to improve security. Its like a seasoning, don’t overdo it but I believe most orgs are not doing it enough. I was surprised that one of the most powerful things I’ve done for security is compiling and sharing a pseudo-monthly newsletter of interesting, exciting or representative security bugs we have found in our travels through the codebase. It gets the company talking, and thinking, about security.” – Collin Greene

Beyond the core triad of Slowification, Simplification and Amplification, the book contains a variety of other lessons that are applicable to security leaders.

One important message: expediting work often adds to chaos – I mean, have you ever seen an episode of Kitchen Nightmares?

The book points to the core truth that control systems have to be faster than the systems being controlled. In practice, this is a warning to security leaders to be cautious and judicious about how they introspect and engage in the work.

Standards feed amplification. They make success easier, by clarifying expectations to highlight when something is difficult or even wrong. In security, this could be paved roads, or detective guardrails for unexpected actions.

I highly recommend that you check out the book if you found this interesting! It’s full of really engaging case studies to further illuminate each of these tactics.