Spandan Pattanayak

Drafts worth signing

Salesforce's incident root cause analyses took 10 to 15 days and covered 40% of mandatory incidents. I had GPT-4 write the first draft and kept people on the sign-off.

Follow an incident Step through the prompts
RCA bot#inc-auth-gateway · 14:09
Confidence HIGH Draft for review · P1 · resolved 14:02
A config change cut the auth service's database connection limit. The gateway's 503s were downstream of it.
Root causeauth-service connection pool exhausted Triggerconfig deploy, 13:41 Contributingno alert on pool saturation; gateway retries added load Related3 past incidents in Jira
13:41Max connections lowered on auth-service13:44Pool errors begin on auth-service13:47Gateway starts returning 503s14:02Limit restored, incident resolved
Edit & approveApprove
Role
Product lead: PRD, architecture spec, prompts and rollout
Team
Reliability Engineering, two backend engineers built the pipeline
Time
12 weeks
Stack
GPT-4 on Azure OpenAI, PagerDuty, Splunk, Datadog, Slack, Jira
100%
of P0 and P1 incidentsget a formal RCA, up from 40%
−35%
median delivery timeabout 12 business days down to 8
86%
root-cause matchagainst human-written RCAs, up from 60%
<8%
multi-service misreadsdown from 25% within a week of the fix

Delivery time runs from PagerDuty resolution to Jira closure at VP sign-off, against the trailing six-month median. Optional P2+ incidents also rose from about 15% to 60% coverage because teams chose to use it.

The problem

Where the days went

The brief from my engineering manager was short: coverage is 40% and an RCA takes two weeks when we do write one. Find out why.

I ran 30-plus interviews in three weeks with RCA owners, engineering managers, compliance and the customer teams, and shadowed two on-call engineers through live incidents, one of them a P1 at 2am.

The question that told me most was "when you skip an RCA, why?" The answer kept coming back: it takes too long and nobody reads it.

The analysis itself took two or three days. Most of the time went to finding data. By the time a rotating owner was assigned, logs had rotated and the war-room threads were buried. One owner called it archaeology. So the fix was a timing change: capture everything while the incident is live, and write the draft only once it is resolved.

Median RCA delivery time: about 12 business days before, about 8 after. Ranges 10 to 15 days before, 7 to 10 after. Before manual median 12, range 10–15 After LLM draft median 8, range 7–10 3–4 days of log hunting removed, 1–2 fewer days of review back-and-forth 051015business days Median RCA delivery time: about 12 business days before, about 8 after. Before: median 12, range 10–15 After: median 8, range 7–10 3–4 days of log hunting removed, 1–2 fewer days of review back-and-forth 051015business days
The system

Three stages

Capture early, write late, sign by hand. Each stage runs on its own trigger, so data is never stale and nothing is written before the cause is known.

When the incident fires in PagerDuty

Assemble the context

Within seconds the pipeline pulls application logs from Splunk, metrics for the service and its neighbours from Datadog, the alert timeline and the last 24 hours of deploys. I designed a six-field schema to normalise them. About 15% of logs were free text, so instead of writing regex parsers I proposed a small LLM call to structure them, about $10 a month. My manager worried about latency; it runs asynchronously, so nobody waits on it.

Context package · P1 auth gatewayCaptured 13:47
Splunk application logs, 30 min windowauth-service, gateway
Datadog metrics, service + 1 hoppool usage, 5xx rate
PagerDuty alert timelineescalations
Deploys and config changes, 24h1 config change
PII scrubbed before any model callemails, IPs, tokens
When the incident is resolved

Draft the RCA

The package goes to GPT-4 inside Salesforce's Azure tenant, so data stays in region. The prompt returns structured JSON: root cause, a confidence level, contributing factors, a timeline, remediation and related past incidents from Jira.

When the RCA owner is assigned

Review and sign

The draft posts in the incident's Slack channel. The owner edits and approves, then it routes to the engineering manager and the VP. Every step records who, when and the diff from the model's text, so Compliance got a better audit trail than the manual process ever had.

Sign-off record · exampleSigned
  1. Draft generated · RCA bot14:09
  2. RCA owner · approved with 2 editsDay 1
  3. Engineering manager · approvedDay 3
  4. VP, Reliability · signedDay 5
The map

Follow one incident through

Every component the three stages touch, from the PagerDuty alert to the signed RCA. Pick an incident and step through it. The dashed layer is the extension I'd build next. I designed it after the project ended, and it never ran at Salesforce.

Shipped Proposed
  • Shipped
  • Proposed
  • Designed for v2, not shipped
  • Active in this step

Stage 1 · when the incident opens

PagerDutyIncident opened
SplunkLogs, alert window
DatadogMetrics, +1 hop
Deploy trackerLast 24h of changes
Alert timelineAlerts, escalations
PII scrubBefore any model call
Translators~85% into 6 fields
LLM data janitor~15% free text
Context packageJSON, per incident

Stage 2 · when it resolves

PagerDutyIncident resolved
Load packageFrom the store
Dependency graphAffected subgraph
Recent changesDeploys and config
Prompt v6Five sections, one call+ human context
GPT-4Azure, in tenant
RCA draftStructured JSON
Related incidentsJira keywords, top 3
Vector searchv2, not shipped

Stage 3 · when the owner is assigned

ConfidenceHigh, medium or low
Senior reviewLow only
Slack postIncident channel
RCA ownerEdit and approve
Eng managerApprove and route
VPFinal sign-off
Jira recordStructured, queryable
Customer RCARequested via MyTrust
Audit trailWho, when and the diff, recorded at every approval

OverviewA rate limiter config change, caught while it was fresh. Press play or step through.

Decisions

Calls I made

  1. Capture at incident time, draft at resolution

    over doing both days later

    Data goes stale within a day, and a root cause can't be written while it is still being found. Splitting the two triggers removed the reconstruction step entirely.

  2. A confidence score that routes review

    over a single pass-through draft

    LOW confidence sends the draft to a senior engineer before anyone else sees it. This is what brought the VP and Compliance on board.

  3. One fixed schema for every draft

    over the free-text RCAs teams already wrote

    Owners wrote RCAs in Confluence or Jira in whatever shape they liked, and the VP was signing drafts of wildly different quality. Fixed JSON fields gave her the same document every time. They also made root causes queryable, so leadership could ask which causes kept recurring, and gave shadow mode a field to score against the human RCA.

  4. Jira keyword search for related incidents

    over a vector index in v1

    Matching on service and root-cause tags was good enough for common failures and shipped in days. I designed the embedding search for v2.

  5. Ship v1 fast, fix in the pilot

    over waiting for the dependency graph

    It cost me: multi-service incidents were misread 25% of the time until I added the graph. Two teams nearly left the pilot, and I would include it from day one next time.

Try it

Six prompt versions

The same multi-service incident, drafted by each version. A config change starved the auth service of database connections, and the gateway, which calls it on every request, filled the logs with 503s. Reconstructed from the pilot's documented failure case, with service names generalised.

v6 · Recent changes in context

Added the last 24 hours of deploys and config changes to the prompt, so the draft names the change that started it.

86% root-cause match across 50 historical incidents.

Root-cause match by prompt version: v3 60%, v4 72%, v6 86%. v1 and v2 were not scored; v5 was measured on multi-service incidents only. 100%50%0% v1v2v3v4v5v6 60%72%86% see note
Trust

Trusting the score

In shadow mode I checked the model's own confidence against whether its root cause matched the human RCA. The labels held up well enough to route review on them.

ConfidenceMatched the human RCAWhat happens next
HIGH91%Standard review chain
MEDIUM68%Senior review recommended
LOW40%A senior engineer must review before the chain starts
Adoption

Who had to say yes

"If I sign this, I'm putting my name on something an AI wrote."
A VP, early on. I showed her that 70% of pilot drafts were approved with no or minor edits by the engineer who owned them, and the diff of every change. By week 10 she said she had better visibility than in the manual process.
"I used to dread the RCA assignment. Now I edit a draft instead of writing from scratch."
An RCA owner after rollout. What took a week now took about a day.
Weeks 7–8

Shadow mode

Ran the pipeline on 50 historical incidents and compared every draft with the human RCA. This is where the multi-service misreads showed up.

Weeks 8–10

Pilot, five volunteer teams

Moved the confidence score to the top of the Slack message and added a one-line summary. When a P2 misread nearly cost me two teams, I reviewed every one of their drafts for three days while the fix shipped. Both stayed.

Weeks 10–12

Full rollout and handoff

Every P0 and P1 got a draft by week 11. I left an architecture doc, a prompt guide covering all six versions, a runbook and the v2 roadmap, and trained two engineers to keep iterating.

Honest scope

What I'd change

Designed after the project. Not built at Salesforce.

Next: keep what the responders knew

Stage 1 captures telemetry for one service, one time window and one hop. The people who fixed the incident often know more. A responder might search logs two services away, or type the real mechanism into Slack, and when the incident closes that trail is gone. The model is left to guess. Its most common remaining miss was the right service with the wrong mechanism.

The cost is a small human step in a system sold on removing toil. I think one line at resolve time is worth it against the days of digging it replaces.

  1. Resolution note

    One structured line when the incident is resolved: what stopped the impact and which service the responder suspects. The prompt treats it as a hypothesis. If the draft disagrees with it, confidence drops and a senior engineer reviews the draft first.

  2. Query trail

    Splunk's audit log already records every search a responder ran. The ones that reached outside the captured scope are re-run with a read-only, capped account and come in as evidence.

  3. Slack observations

    The incident channel is read once at resolution, with bot posts and acks filtered out. Each message left is tagged as a hypothesis, a confirmation or an observation, so a guess never carries the weight of a metric. Screenshots are dropped, since a pattern scrub can't read an image.

Known limits

  • Capture covers one service, one window and one hop. A cause outside that still needs a manual pull.
  • In one cache incident the draft named the right service and guessed eviction policy. The cause was lock contention.
  • The 86% match is scored against VP-signed RCAs, and those can be wrong too.
  • PII patterns catch the formats someone listed. A new format can slip past.
  • On a much larger service graph the whole subgraph won't fit in one prompt. That needs graph-aware retrieval.

Designed but not shipped

  • Status updates to customers in the trust portal at each review step. Faster RCAs shortened the silence; they didn't end it.
  • Embedding search over past RCAs, replacing keyword matching.
  • Blast-radius estimates and suggested remediation playbooks.

Next time

  • Put the service dependency graph in v1.
  • Treat VPs as users in discovery, not as a step in the process.
  • Start the pilot with two teams, not five, so one person can watch every draft.

Let's talk

Building something where a model drafts and a person signs? I'd like to hear about it.

pattanayakspandan@gmail.com
LinkedIn RaaS case Deal engine case Other work