How threat hunting evolves at scale

At Red Canary, our deep focus on mechanized detection engineering has always been complemented by an underlying need to understand emerging threats, patterns, and vulnerabilities before they can be automated. Threat hunting, which yields raw intelligence and behavioral insight needed to stay ahead of adversaries, is the bridge that makes this happen.

While often beginning informally, threat hunting requires deliberate strategies to grow effectively. It’s a journey every organization needs to take: Moving from individual ad hoc explorations by curious analysts to a more critical, structured discipline that demands continuous evolution.

In this blog, we’ll walk through what a journey to a scaled, mature threat hunting program looks like, something that can provide valuable insights for any organization looking to enhance its defensive capabilities and build on informal investigations.

Flexible and exploratory beginnings

Threat hunting often starts in an organic, almost intuitive manner. In its nascent stages, an organization may not have a dedicated threat hunting team or a formal program. Instead, it’s driven by the curiosity of SOC analysts or detection engineers who may leverage existing tools, parse log data, or follow leads inspired by indicators of compromise (IOCs) sprinkled throughout security blogs.

This early approach is characterized by:

• Inquisitive analysts: Individuals with diverse experiences playing with available tools and signatures
• Ad hoc queries: Investigations driven by hunches or emerging intelligence
• Informal knowledge sharing: Learnings passed through word-of-mouth or internal wikis rather than structured documentation
• Unconstrained methodologies: Generally threat hunts are fluid but follow a “theory -> query -> explore -> pivot” pattern

These beginning phases can be surprisingly effective early on. They boast several advantages: low overhead, creativity, rapid feedback loops, and low cost. They’re also iterative and not constrained by heavy process flow, making it easier for analysts to get more “reps” in; Those involved begin to better understand the environment, building foundational knowledge and demonstrating that hunting can deliver tangible benefits, even if it’s from a single data source like EDR telemetry or Okta logs.

The tipping point: Why scaling becomes necessary

While effective, this makeshift model eventually hits a ceiling. As your organization grows and the security landscape becomes more complex, the limits of an informal process become glaring. Initial wins generate demand, leading to a need to scale, incorporate more environments, and support a broader range of stakeholders. This signals the need for a more structured program.

Key drivers for scaling include:

• Inability to communicate full value: Without metrics, coverage maps, or a clear return on investment (ROI), it becomes challenging to justify resources or demonstrate value beyond anecdotal wins
• Expanding environments: Threat hunting usually starts in a corporate environment but needs to extend to developer environments, cloud infrastructure, endpoint telemetry, identity systems, and SaaS apps; this expansion can quickly outpace the bandwidth of a small team
• Too much data: The sheer volume and diversity of data sources, including the environments discussed above (EDR, cloud logs, identity data) can overwhelm hunters, making it hard to maintain visibility and context
• Consistent output: The need to consistently capture value, iterate on findings, and improve detection coverage or identify issues across the organization drives the need for repeatable processes

The demand for answers around coverage, value from each data source, and impact on your organization’s overall security posture requires a strategic shift.

Growing pains: Friction emerges at scale

A lot of the time, you may find that scaling threat hunting without the commensurate structural changes can result in nothing breaking but everything slowing down. This increased complexity brings friction.

Inconsistencies

Different analysts with varying skill levels and approaches solve similar problems in different ways. This leads to inconsistent hunt outcomes and can make it hard to compare or aggregate findings. Data schemas across multiple sources also vary wildly, something that complicates correlation.

Inefficiencies

A lack of a unified view across tools and environments forces analysts to spend more effort during hunts often losing context. The lack of a centralized hunt library or query storage means analysts repeatedly recreate content, leading to wasted time.

Lack of repeatability

Decentralized knowledge can result in knowledge gaps. Without structured hunts and documented methodologies, it becomes difficult to reproduce investigations or ensure consistent quality.

Data volume

Ever-increasing volumes of data, from diverse sources like cloud and identity telemetry, create challenges around storage, processing, and the need for security information and event management (SIEM solutions) etc.

It’s around this point that it could take you 20 minutes to run a query because your data source has doubled or you need to reproduce the same logic flow that a different analyst or hunter did in order to re-execute a hunt or query. Dialing in that system tooling stack so it’s procedural and you have a robust framework to do it efficiently and consistently is key.

Shifting focus: Strategies for scalable hunting

To overcome these challenges, threat hunting programs must evolve their focus beyond just running queries to emphasize structure, reuse, and streamlined workflows

Data shaping: From raw to analyst-ready

Raw data is rarely analyst-ready. Each vendor has its own format, nuances, and quirks. Cloud and identity telemetry can often come as large JSON blobs, while EDR provides device and file events. Correlating these disparate data sources—like linking a cloud identity to an EDR identity—requires significant effort. The goal is to avoid custom logic for every recurring problem.

This necessitates a robust data engineering approach to process, model, and normalize data.

Reuse logic and aligned workflows

Efficiency at scale hinges on reusability. Instead of analysts constantly reinventing the wheel, organizations need to develop shared logic and standardized processes. Having a structured approach, or even just the right level of automation, makes a tangible difference. If you can capture the learnings from a single hunt—which data sources proved valuable, which approaches worked, what context was worth pulling in, etc.—and centralize that, even something as simple as a shared query library, you immediately reduce redundant discovery work. Over time, that compounding effect is what drives real efficiency. You’re iterating on what you’ve learned rather than starting from scratch every time.

Enabling laptop-scale analytics

While advanced tooling and infrastructure are vital for large-scale operations, it’s equally important to empower individual analysts. Tools that bridge the gap between simple spreadsheets and complex enterprise solutions are crucial. DuckDB, an in-memory analytical database, paired with Parquet files, is a good way to solve the data volume problem here.

This allows analysts to:

• Query large datasets locally: Without needing heavy infrastructure, effectively creating a data lake on your laptop
• Combine and transform data: Perform SQL queries, join different datasets, and execute high-powered analytics efficiently, even on gigabytes of telemetry
• Improve performance: Enjoy fast and predictable data access, reducing the time spent optimizing code

Analysts often start with limited resources but need the power to perform complex analysis. While spreadsheets are great for simple, single-source hunts, tools like DuckDB, which leverages SQL for data manipulation and analysis, become invaluable for joining multiple data sources, performing transformations, or conducting more complex statistical analysis. Often when choosing between tools, the best option will come down to the complexity of the hypothesis and the analyst’s skillset.

A continuous feedback loop: From hunt to detection

The goal of scaling a threat hunting program is not just finding threats but making those findings actionable. This brings us back to the interplay between detection engineering and threat hunting. When a hunt successfully uncovers a threat, consider:

Automate as a detection

If the identified pattern is high-fidelity, consistently indicates a threat, and produces a low amount of noise, it should be integrated as an automated detection rule within the SIEM or other security tool.

Automate the workflow

For more complex hunts that can’t be translated to a simple detection, the workflow itself can be automated. Standardizing initial queries, analysis steps, and pivot points can effectively filter noise and make subsequent investigations easier and more repeatable.

Baseline drift

Beyond immediate threats, hunting also helps establish baselines. Observing drift—changes in expected behavior over time—can be a significant indicator of evolving risks, even if no explicit threat is found.

This continuous feedback loop ensures that insights gained from proactive hunting directly strengthens the organization’s defensive posture, making it more resilient in the long run.

Scale your team

To scale threat hunting, organizations need to enable analysts by streamlining workflows. This means making data access and structure feel natural, fostering reusability through modular queries and playbooks, and standardizing processes so that good tooling supports rather than constrains investigations.

By being flexible, anticipating the challenges of growth and investing in data shaping, workflow standardization, and accessible analytical tools, organizations can build threat hunting programs that not only keep pace with the evolving threat landscape but actively shape their security posture for the better.