SigninLogs
| where TimeGenerated > ago(1h)
| where ResultType in ("50034", "50126", "700016") or isempty(AppDisplayName)
| summarize
DistinctClientIDs = dcount(AppId),
ResultCodes = make_set(ResultType),
Usernames = make_set(UserPrincipalName)
by SourceIPAddress, UserAgent, bin(TimeGenerated, 15m)
| where DistinctClientIDs > 5
| where ResultCodes has "700016"The two variables that matter most are DistinctClientIDs, because a single source cycling through many unregistered app IDs is the tell that per-application thresholds miss, and the presence of AADSTS700016 in that same window, which elevates the event from configuration noise to possible credential validation in progress. Layer in username-pattern detection, alphabetic or dictionary progression across attempts from the same source, to catch the OutFlareAZ-style wordlist pattern specifically.
Tune the DistinctClientIDs threshold against your own tenant’s baseline before trusting it in production. A dev team running CI against a handful of test app registrations can produce a smaller version of the same shape, and Conditional Access policies scoped only to named applications will not catch a fabricated client ID that never matches an intended application scope in the first place.
Wire the rule into existing SOAR or ticketing workflows rather than a standalone dashboard nobody checks on a Friday afternoon. A detection that fires into the same queue as password-spray and impossible-travel alerts gets triaged with the same urgency; one that lands in an isolated identity-hygiene report gets read weeks later, if at all.
