{"id":1197,"date":"2026-11-02T04:05:00","date_gmt":"2026-11-02T04:05:00","guid":{"rendered":"https:\/\/pilotsdeal.com\/blog\/?p=1197"},"modified":"2026-11-02T04:05:00","modified_gmt":"2026-11-02T04:05:00","slug":"aiops-for-security-two-analyst-story","status":"publish","type":"post","link":"https:\/\/pilotsdeal.com\/blog\/aiops-for-security-two-analyst-story\/","title":{"rendered":"Two Analysts and an Agent: AIOps for Security in Practice"},"content":{"rendered":"

Northline Air is a fictional regional operator \u2014 forty aircraft, three hundred staff, and a security team of exactly two analysts plus one IT generalist who also runs the help desk. Which makes Northline a useful stand-in for most aviation businesses. When the queue of suspicious emails and odd access alerts began doubling every quarter, the team piloted AIOps for Security<\/strong> tooling: machine-learned scoring, anomaly detection, and an investigation assistant. What follows is their trial reconstructed precisely, because its outcome \u2014 useful tool, denied autonomy \u2014 is the one most mid-size operators should expect and plan for. Names invented, numbers and gates real; the procedure is the reproducible part.<\/p>\n

The Setup: Two Analysts and a Rising Queue<\/h2>\n

By spring, the shared inbox processed six hundred suspicious-message reports a month, and each analyst spent the first two hours of every shift clearing it. Genuine incidents were rare \u2014 a compromised crew account the previous autumn, a wire-fraud attempt caught by luck \u2014 which is precisely the arithmetic that makes triage miserable: high volume, low hit rate, and the knowledge that somewhere in the pile might be the one that matters. The generalist proposed what every vendor demo promised: let the machine read everything, close the obvious, and escalate the rest. The analysts agreed to the pilot on one condition \u2014 nothing gets closed automatically, by anyone, for thirty days. They had read enough incident reports to know the failure mode they feared most: not the machine missing a threat, but everyone assuming someone else had checked the machine.<\/p>\n

The Demo, and What It Left Out<\/h2>\n

The vendor’s session was polished: accuracy percentages, cheerful dashboards, a slide announcing “fully autonomous phishing triage”. Missing from the deck: which traffic the numbers came from, whether any observation-only mode existed, and what happened to false closes. Pressed on the last point, the sales engineer admitted the product begins closing mail from its first production minute \u2014 a documented design choice as of October 2026, with no observe-only switch. Northline’s workaround became contractual: the deployment would run with its closing function disabled by configuration, decisions logged but never executed, until evidence said otherwise. The vendor agreed. Procurement noted the precedent, which mattered: the next three suppliers were asked the same observe-mode question before pricing, and the question alone changed how seriously the decks took evidence.<\/p>\n

Thirty Days of AIOps for Security in Shadow<\/h2>\n

The trial month ran like a lab notebook. Every message the machine would have closed was independently judged by the analysts first, without seeing the machine’s verdict; labels went into one column, machine decisions into the other, and only at month-end did the two meet. The workload felt heavy for exactly four days, then settled into rhythm. One discovery mattered as much as any accuracy figure: relabelling history exposed that the team’s own past decisions disagreed with each other \u2014 the same scam family closed differently in different months. Before the machine could be graded, the humans had to reconcile themselves, which quietly upgraded the whole operation’s documentation. Two analysts also began keeping a one-page lab notebook \u2014 dates, counts, decisions, doubts \u2014 on the theory that a trial nobody can reconstruct afterwards is a trial that never happened.<\/p>\n

The Verdict the Numbers Delivered<\/h2>\n

Month-end brought a verdict worth reading slowly. Across 1,240 blind-judged decisions, the machine had correctly handled 71 genuine threats and wrongly dismissed 2. The raw miss figure \u2014 2.8% \u2014 looked like a pass against the 2% gate most frameworks require for autonomous closing. The bound said otherwise: with two failures sitting among seventy-odd genuine cases, the statistics placed a defensible ceiling near 8.6% \u2014 a clear failure \u2014 and the team’s written summary explained why in one line: small samples forgive generously, so claims must be stated as ceilings, not point values. The same evaluation showed close decisions 99.8% precise, threat verdicts 86% precise, and 93.5% of triage workload genuinely removable. Every number true. One conclusion drawn: recommend level, yes; closing level, not yet. The presentation to leadership took eleven minutes \u2014 five on the numbers, five on the ladder, one on the ask \u2014 and the ask was granted without debate, because every objection had already been answered on paper.<\/p>\n

The analysts also logged the forward arithmetic: spotless runs cap the true rate only near three-over-n, so defending a 2% ceiling costs 236 labelled cases once a single miss appears \u2014 and 313 once a second does.<\/p>\n

What Changed on Monday: AIOps for Security at Work<\/h2>\n

Northline adopted the assistant at the recommend tier and never looked back. The ranked queue arrived each morning with the likely-junk at the bottom, drafted summaries attached to the serious items, and the two analysts working a list a third of the old length. Three safeguards came with it. Planted canaries \u2014 seeded threats tucked into the flow quarterly, with the privilege reviewed downward if several went uncaught \u2014 became the team’s early-warning system. One-click reopen stayed available for every machine-suggested close, and the register of allowed actions lived in a document the whole team could read. When MITRE’s guidance says a person must evaluate the machinery’s output, this is what compliance looks like at two-analyst scale: small, written, and boring. The queue dropped from two analyst-hours each morning to roughly forty minutes, and the recovered time went to the access-review backlog that had been waiting since winter. Autonomy was not granted that quarter. It was not missed, either \u2014 nobody outside the team noticed any slowdown, which is precisely the point of a human-first rollout.<\/p>\n

Where the Northline Trial Goes Next<\/h2>\n

Quarter two has a written plan. The second category \u2014 odd access patterns around crew scheduling \u2014 enters the same shadow process, because the phishing numbers say nothing about how a different data stream behaves. Reviews move to a monthly per-category rhythm, with drift monitors watching for the slow slide that retraining always risks, and a cost baseline now records what each minute of triage actually costs the operation. The closing gate stays shut until either the labelled history grows past the 236-case threshold or a fresh blind trial clears the bound on its own \u2014 whichever happens first, and only the data will say. Leadership holds, in writing, the exact conditions under which autonomy gets revisited. That single page has already outlasted two vendor account managers, which may be the quietest argument for writing such things down.<\/p>\n

What Northline Would Tell Peers<\/h2>\n

The team’s hindsight compressed into five lines any operator can reuse:<\/p>\n

    \n
  1. Insist on the silent phase even if the product lacks one \u2014 configuration plus contract can build it.<\/li>\n
  2. Label blind, reconcile your own history first, and expect the reconciliation to be the most valuable part.<\/li>\n
  3. Treat every accuracy claim as a point value needing a bound; ask for the sample size behind it.<\/li>\n
  4. Adopt the lowest useful rung and let evidence, not enthusiasm, request the climb.<\/li>\n
  5. Plant canaries before autonomy, not after the first disaster.<\/li>\n<\/ol>\n

    For operators studying the broader toolkit, our pieces on AIOps tools, techniques, and use cases<\/a> and how AI-assisted operations reshape infrastructure and automation<\/a> map the surrounding territory, and the career-side read is the certified AIOps engineer path<\/a>. The shared reference behind Northline’s trial \u2014 autonomy rungs, bounds, canary practice \u2014 is the Ops Atlas page covering AIOps for security<\/a>. Fictional team, real lesson: the machinery earned its keep exactly because nobody let it skip the exam.<\/p>\n","protected":false},"excerpt":{"rendered":"

    A fictional airline’s two-analyst security team trials AIOps for Security in shadow mode \u2014 and finds impressive numbers that still fail the autonomy gate.<\/p>\n","protected":false},"author":6,"featured_media":1198,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[748],"tags":[],"class_list":["post-1197","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-pilot-tech"],"_links":{"self":[{"href":"https:\/\/pilotsdeal.com\/blog\/wp-json\/wp\/v2\/posts\/1197","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pilotsdeal.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pilotsdeal.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pilotsdeal.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/pilotsdeal.com\/blog\/wp-json\/wp\/v2\/comments?post=1197"}],"version-history":[{"count":1,"href":"https:\/\/pilotsdeal.com\/blog\/wp-json\/wp\/v2\/posts\/1197\/revisions"}],"predecessor-version":[{"id":1213,"href":"https:\/\/pilotsdeal.com\/blog\/wp-json\/wp\/v2\/posts\/1197\/revisions\/1213"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/pilotsdeal.com\/blog\/wp-json\/wp\/v2\/media\/1198"}],"wp:attachment":[{"href":"https:\/\/pilotsdeal.com\/blog\/wp-json\/wp\/v2\/media?parent=1197"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pilotsdeal.com\/blog\/wp-json\/wp\/v2\/categories?post=1197"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pilotsdeal.com\/blog\/wp-json\/wp\/v2\/tags?post=1197"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}