Semantic Search Found the Playbook but Missed the “Not”

“The certificate has not expired; the error is a missing DNS record.”

A playbook finder returned the certificate expiry entry for that request. Its similarity score was 0.688, higher than the scores for several requests that actually described an expired certificate.

The result also passed the application’s rejection rules. The closest match was comfortably ahead of the other candidates.

This was one of the failures in a small, assistant-assisted experiment comparing phrase matching with semantic search. It captures the problem explored here: a document can be similar to a request without being an appropriate answer.

From defining success to testing retrieval

The first article described a fictional playbook finder for an on-call engineer investigating an incident without a clear documentation link.

The proposed behaviour was deliberately narrow: return one relevant playbook entry, or return no match when the description does not support a clear selection.

This experiment tested that behaviour against three short entries:

Playbook	Content
Checkout timeouts	Inspect service health and dependency latency; escalate to the checkout owner
Worker queue backlogs	Inspect worker availability and queue age; escalate to the worker owner
Certificate expiry	Verify certificate validity and service ownership; escalate to the certificate owner

These were brief teaching documents, not production playbooks. The experiment ran in a CPU-only Docker container, with networking disabled during evaluation. It generated no operational advice and executed no mitigation actions.

What semantic search changes

The original baseline looked for literal phrases such as checkout timeout, queue backlog, and certificate expired.

That makes its decisions easy to inspect, but wording matters. “Checkout keeps timing out” does not contain “checkout timeout.”

Semantic search uses an embedding model to turn text into numerical vectors. The experiment used all-MiniLM-L6-v2, which produces a 384-dimensional representation of each text. Each playbook’s title and body were embedded, then compared with the embedded request using cosine similarity. Model documentation.

The practical difference is that a request can match an entry without repeating its exact words.

However, the search still produces a nearest candidate when none is appropriate. The application needs a separate decision about whether to return it.

Three behaviours, one test set

The comparison included:

1. Phrase matching: return the entry associated with a recognised phrase. Return no match if none or multiple entries match.
2. Semantic search: return the nearest entry for every nonblank request.
3. Semantic search with rejection rules: return the nearest entry only if its score is high enough and sufficiently ahead of the second candidate.

All three returned no match for blank input.

The new dataset contained 24 development cases and 24 test cases. Each group had twelve answerable requests and twelve requests labelled no match. Cases covered paraphrases, unsupported problems, negation, background mentions, missing context, and multiple symptoms.

The rejection thresholds were selected using only the development cases. The selection objective gave equal weight to correct answers and correct abstentions. This was an illustrative choice, not an agreed operational risk policy.

The selected rule required:

• A cosine similarity score of at least 0.575.
• A gap of at least 0.275 between the first and second candidates.

Those thresholds were fixed before evaluating the test cases.

Better retrieval did not automatically mean better answers

Here are the results on the 24 test cases:

Method	Correct answers on 12 answerable requests	Correct no-match on 12 other requests	Incorrect selections
Phrase matching	6/12	9/12	3
Semantic search	12/12	1/12	11
Semantic search with rejection rules	9/12	10/12	2

Semantic search ranked the correct entry first for all twelve answerable requests.

Its eleven incorrect selections came from returning an entry for every nonblank request that should have received no match. That result reflects the return policy; it does not mean the ranking failed on the answerable cases.

Adding rejection rules produced a more useful balance. Compared with phrase matching, it returned three additional correct answers and made one fewer incorrect selection.

It also rejected three requests whose nearest entry was correct.

A paraphrase that worked

Consider this request:

Shoppers reach checkout but the request times out before completing.

Phrase matching returned no match. Semantic search selected the checkout-timeout entry with a score of 0.581 and a lead of 0.311 over the second candidate.

Both values passed the rejection rules, so the application returned the relevant playbook.

This is the improvement we hoped to see: the engineer could describe the symptom naturally without knowing the wording used in the matching rules.

A negation that still passed

Now return to the opening example:

The certificate has not expired; the error is a missing DNS record.

The expected result was no match. The collection contained no DNS playbook, and the request explicitly ruled out certificate expiry.

Semantic search nevertheless selected the certificate entry:

Measure	Observed value	Required minimum
Similarity score	0.688	0.575
Lead over second candidate	0.608	0.275

Neither rule rejected it.

The important observation is the returned result. This experiment does not establish what the model internally represented about negation. It establishes that the similarity-based decision failed to respect the distinction in this request.

A wider lead over other candidates did not help: the other two entries were even less similar.

Simply raising the score threshold would also have a cost. A cutoff above this incorrect match would reject every valid certificate request in this test set, whose scores ranged from approximately 0.590 to 0.646.

A high similarity score was not reliable evidence that the playbook applied.

Rejection also removed a useful answer

The request below had a supported answer:

Customers wait at checkout until the operation hits its time limit.

The checkout entry ranked first with a score of 0.622, above the minimum. But its lead over the next candidate was 0.268, just below the required 0.275.

The application returned no match.

The decision can be expressed in a few lines:

if not query.strip():
    return None

if top_score < minimum_score or top_score - second_score < minimum_gap:
    return None

return top_playbook

The rule is easy to implement. Choosing values that produce acceptable behaviour is the harder part.

These particular numbers belong to this model, document representation, collection, and development set. They should not be copied into another application as universal defaults.

What the experiment supports

For these cases, semantic retrieval with rejection rules improved on the exact phrase matcher. That is enough to justify investigating it further.

It does not establish superiority over keyword search generally. The comparison did not include BM25, expanded matching rules, hybrid retrieval, or other embedding models. The phrase matcher used authored phrases; semantic search used document titles and bodies.

The dataset is also small and synthetic. Both splits were authored with assistant help and share similar patterns. Keeping test cases out of threshold selection helps, but does not make them representative of actual incident reports.

Document permissions, freshness, service filtering, response latency, and time saved by engineers were not evaluated.

The next useful step would be to compare a stronger lexical baseline with semantic retrieval using more realistic playbooks and independently labelled requests. The failures already inspected can become regression cases; a fresh test set would be needed to assess subsequent improvements.

For now, the finding is concrete: semantic search recovered useful paraphrases, while the application still returned an entry for a symptom the request had ruled out. Evaluating only the successful matches would have missed that failure.

References

• all-MiniLM-L6-v2 model card: model purpose and embedding representation.
• Sentence Transformers — Semantic Textual Similarity: embedding comparison and similarity calculations.