Intro

There was a question I saw the other day asking if detection engineers (DE), the folks who author alerts, should also be primarily responsible for their triage. There was large support for having the detection authors also play a large role in initial triage, often as part of the oncall. I largely disagree with this stance and am going to use some time Friday to quickly argue why you’re likely compensating for technology and process failures if you do that.

The Argument in Favor

Those who supported having DE triage alerts did have valid arguments that clearly came from a place of shared pain. One argument was that by having DE triage alerts, they’d be able to directly feel the pain that poor quality alerts or an alert flood causes. The idea being that tuning is slow when there’s no direct pain felt and therefore the initial triage has to sit on their hands while their queue floods with garbage.

Another often cited argument is that DE is best equipped to triage as they wrote the detection and therefore will more efficiently respond to the alert.

While these are both solutions to problems that secops has, I believe they mask symptoms rather than treat root causes.

The Metrics Problem i.e. Shared Destiny

It is not untrue that detection engineers can be slow to tune alerts. The question then, is why? In an application of Goodhart’s Law, detection engineers will strive to be successful at what they’re measured against. Often, that’s detection coverage. Yes, false positives/true positives/etc are oft incorporated. But most SOCs have so poor metrics in this anyways that a spike is absorbed into the noise. Therefore, my solution is that the DE and triage team must have a shared destiny here. That is, when an alert is causing a flood or has quality issues, it must prevent new work from being pushed into the system.

One way to implement this is to have alerts disabled after a certain volume per time period or a certain amount of false positives in a row. And then, either through policy or other linting, you can prevent further merges until this issue is resolved. These outages must be tracked and reported to leadership with as much importance as any other metric.

The ultimate goal of a system like this is to encourage only high quality detections to enter the system to discourage floods. And if something does flood the system, the system (not humans) should gracefully handle it. Having DE share the pain implies that pain is an acceptable part of the system as opposed to something to remove.

Expert Knowledge

As referenced earlier, another aspect that’s referenced is DE knowledge improving alert triage accuracy and speed. While this is true on its surface, if that’s your approach, you’re building a system based on human knowledge which is inherently not resilient. If your expert is sick, on PTO, or wins the lottery, what is your backup plan? In this case, I’ll refer to my previous blog “how to write an actionable alert”. In addition to the recommendations there, one could also have their IR/triage team approve detections before they hit production. This would be their “yes, we’re ready to receive this signal and do IR on this system if necessary” check off.

Exceptions

In a smaller team where your secops function can be counted on one hand, the math for separate teams doesn’t quite work out. The exact number of when to split is going to depend on if you need 24/7, complexity of your systems, etc.

Additionally, I can see my argument completely dissolving in the face of the growing AI wave. I would not be surprised if the “detection engineer” role disappears entirely and companies have 1-3 “security operations” engineers again who do it all (pipelines, DE, IR, etc) with AI-enabled tooling.

Conclusion

In short, I think having your DEs triage alerts is compensation for poor alerts, poor metrics, and poor technology. Work to eliminate pain from your SOC rather than having it suck for everyone.