Before Adding an LLM, Define What a Good Answer Looks Like

An on-call engineer gets a message: “Customers are getting timeouts when they try to check out.”

The report has no alert link. The team has playbooks, but the engineer needs to find the entry that applies.

A search tool might help. Before deciding whether it needs an LLM, though, I would want to settle something simpler: what should it return for that message, and what should it do when none of the playbooks fits?

This is a fictional design example.

Give the finder tool a specific job

If an alert already links to the relevant playbook, the engineer can follow it. The case worth exploring is an investigation that starts without that reference: a user report, an unfamiliar symptom, or a problem that has moved beyond the original alert.

Suppose a team maintains three playbook entries for its checkout service:

ID	Playbook entry	Scope
pb-001	Investigating checkout timeouts	Requests timing out during checkout
pb-002	Investigating order-processing backlogs	Orders waiting longer than expected for processing
pb-003	Investigating certificate expiry	Connection failures involving an expired certificate

For the first version, I would keep the behaviour small:

Return one playbook entry relevant to the reported symptom. Return no match if the description does not support a clear selection.

That restriction has a cost. An incident involving both timeouts and a processing backlog may need two entries. For this exercise, the finder would decline to select one. If that happened frequently in real use, it would be a reason to change the requirement.

A result would include the entry’s title, identifier, canonical link, service scope, owner, and last-reviewed date. Those details give the engineer something to check before following the guidance.

The result means “this entry may help you investigate.” It does not mean “we have diagnosed the incident.” The timeout entry might lead the engineer to a slow dependency, exhausted capacity, or something else entirely.

Try a baseline whose mistakes are easy to explain

The simplest implementation could look for phrases:

"checkout timeout"    → pb-001
"queue backlog"       → pb-002
"certificate expired" → pb-003

If there is no matching phrase, it returns no match. If phrases for two different entries appear, it also returns no match.

This is easy to inspect. It is also easy to break.

Request	Expected result
“There is a queue backlog in order processing.”	pb-002
“Customers say checkout keeps timing out.”	pb-001
“There is no queue backlog. We are investigating an unrelated database issue.”	No match
“The incident notes mention checkout timeout, but this report is about incorrect prices.”	No match

The first request contains the expected phrase.

The second describes checkout timeouts without using “checkout timeout.” A literal matcher would miss it.

The third and fourth contain recognised phrases, but neither describes a current problem covered by those entries. A literal matcher would still return a result.

No model is needed to see the problem in these rules: they treat the presence of words as evidence that the corresponding symptom is present. Running the implementation would let us check that behaviour and record the results.

Count the two kinds of mistake

A single accuracy score would hide something useful here.

One mistake is failing to find an entry that would help. The other is returning an entry that does not apply. During an incident, both can waste time, but in different ways.

I would measure them separately:

• Correct matches: Of the requests with an applicable playbook, how many receive the right entry?
• Correct no-match results: Of the requests that should receive no entry, how many actually receive no match?

The second behaviour is often called abstention. Here, it simply means the application declines to select a playbook.

A finder that always abstains would do perfectly on the second measure and fail every answerable request. A finder that always returns its closest match could look useful while repeatedly sending engineers to irrelevant guidance.

The acceptable balance needs discussion with the people who will use it. In this design, an engineer checks the reference before acting. Automatically executing a suggested mitigation would require a much stricter set of controls and a different evaluation.

For the experiment, I would choose the acceptance criteria before comparing implementations. Otherwise, it would be too easy to favour whichever measure makes the preferred approach look better.

Ask a question only when the answer would help

“Requests are timing out” might be enough within a collection dedicated to one service. Across a whole platform, it probably needs more context.

In that case, “Which service is affected?” is a useful question because the answer could change the selection.

Asking for logs, timestamps, error codes, and endpoints every time would add work even when the description was already sufficient.

For the first version, I would keep the output as either a candidate or no match, with a short suggestion when nothing is selected:

No clear match in this collection. Add the affected service or browse the available playbook entries.

The wording matters. The finder has searched a particular collection. It has not established that no relevant guidance exists or that the incident has no solution.

Keep document problems separate from search problems

Before testing retrieval, I would assume the three fictional entries are current and accessible to the engineer. A real application cannot leave those assumptions implicit.

It should search only content the engineer is allowed to access. Each entry needs an owner, and the team needs a way to identify guidance that no longer reflects the service.

Showing a review date helps the engineer assess the source, but it does not prove the instructions are still correct. Better search will not repair an obsolete playbook.

These requirements belong to the application regardless of whether it uses keywords, embeddings, or an LLM.

Use the failures to choose the next experiment

If the baseline misses useful entries because engineers describe symptoms differently, I would test semantic retrieval: searching by similarity in meaning.

The comparison would still need the negation and background-context cases. Finding more paraphrases would be an improvement, but returning more irrelevant entries could outweigh it.

I would use some examples to develop the approach and keep a separate set for evaluation. The decision would depend on whether useful matches improved while incorrect selections stayed within the agreed limit.

Generated explanations would come later, if engineers needed help interpreting the entries. That would introduce another question: does the explanation preserve the source’s instructions, conditions, and limitations? Linking to the right document would not be enough to establish that.

For now, the smallest useful experiment is a phrase matcher, a labelled set of requests, and a record of what it gets wrong.

That gives the next decision something concrete to rest on: does a more capable search method help engineers find the right playbook, and what new mistakes does it introduce?