Cosine Similarity for Semantic Search: Geometry, Scores, and Evaluation

Checked on October 3, 2026. The arithmetic and evaluation examples were executed with Python 3.12.14. Vectors and relevance labels are synthetic. No embedding model, external API, or retrieval benchmark was run for this article.

A semantic search system returns a document with a score of 0.8. A developer displays “80% confidence,” then accepts the document as evidence for an answer. The number looks reassuring, but cosine similarity does not provide that probability or establish the document's factual correctness.

Cosine similarity is a geometric comparison between vectors. Its usefulness depends on how the vectors were constructed and how well the resulting ranking serves the search task. Understanding that separation makes it easier to diagnose poor retrieval without treating every high score as trustworthy evidence.

Short answer: cosine compares vector direction, removing magnitude from the comparison. Use it with embeddings from a compatible representation space and the scoring convention expected by the model. Reject invalid vectors, evaluate retrieval against labeled queries, and verify the retrieved content separately. A score threshold is a policy to test, not a universal truth boundary.

What cosine actually measures

For two nonzero real vectors q and d, cosine similarity is their dot product divided by the product of their Euclidean lengths:

cos(q, d) = dot(q, d) / (length(q) × length(d))

The result lies between -1 and 1 in exact arithmetic. A value of 1 means the vectors point in the same direction; 0 means they are orthogonal; -1 means they point in opposite directions. These geometric statements do not themselves assign an English meaning to each direction.

An embedding model supplies that representation. The original Sentence-BERT paper describes sentence embeddings that can be compared using cosine similarity. It supports the use of learned sentence representations; it does not imply that all embedding models have identical score distributions or that similarity establishes a factual claim.

Vectors from arbitrary incompatible models should not be mixed just because they have the same number of dimensions. Corresponding coordinates must belong to a compatible representation space. Some systems deliberately use different query and document encoders trained to align their outputs; compatibility is about that design, not merely using identical function names.

An exact example: direction and magnitude

Take query q = (1, 0) and five candidate vectors. These are hand-created coordinates, not embeddings of real documents. Larger length here does not represent document length, popularity, or quality.

A and B point in the same direction, so cosine ties them. Raw dot product gives B a much larger score because its magnitude is larger. Raw Euclidean distance places B far from the query. Three scoring choices can therefore produce different orderings on unnormalized vectors.

When dot product and Euclidean ranking agree

Normalize a vector by dividing it by its nonzero length. Both A and B then become (1, 0). For unit-length vectors, dot product equals cosine similarity because the denominator is one.

There is another useful identity for unit vectors:

squared_distance(q_unit, d_unit) = 2 - 2 × cos(q, d)

Consequently, descending cosine and ascending Euclidean distance produce the same ranking when both sides are unit normalized, subject to ties and numerical precision. This follows by expanding the squared difference: the two squared lengths contribute one each, while the cross term contributes twice the dot product.

The Sentence Transformers similarity documentation notes the equivalence of dot product and cosine on normalized embeddings. This equivalence is conditional. Switching a vector index from cosine to raw dot product without considering normalization can change rankings.

Index implementations also expose different score conventions. A result might be cosine similarity, cosine distance such as 1 - cosine, or a negative distance. Check whether larger or smaller is better before sorting, applying a threshold, or labeling an interface. The word “score” alone does not identify the convention.

A small implementation with input checks

This standalone function is an educational reference for finite real vectors. Scaling before computing the norm avoids overflowing simply because several components are very large. Production search should generally use the vector library or index appropriate to its workload.

import math

def cosine(left, right):
    left = tuple(float(value) for value in left)
    right = tuple(float(value) for value in right)
    if not left or len(left) != len(right):
        raise ValueError("Vectors need equal, nonzero dimensions")
    if not all(math.isfinite(value) for value in left + right):
        raise ValueError("Vector components must be finite")

    def unit(values):
        scale = max(abs(value) for value in values)
        if scale == 0:
            raise ValueError("Cosine is undefined for a zero vector")
        scaled = tuple(value / scale for value in values)
        norm = math.hypot(*scaled)
        return tuple(value / norm for value in scaled)

    score = math.fsum(a * b for a, b in zip(unit(left), unit(right)))
    return max(-1.0, min(1.0, score))  # Bound floating-point roundoff.

query = (1, 0)
candidates = {"A": (1, 0), "B": (10, 0), "C": (0.8, 0.6),
              "D": (-1, 0), "E": (0, 1)}
expected = {"A": 1.0, "B": 1.0, "C": 0.8, "D": -1.0, "E": 0.0}
for name, vector in candidates.items():
    assert math.isclose(cosine(query, vector), expected[name], abs_tol=1e-12)
assert math.isclose(cosine((1e308, 1e308), (1, 1)), 1.0, abs_tol=1e-12)

for invalid in [(0, 0), (float("nan"), 1), (float("inf"), 1), (1, 2, 3)]:
    try:
        cosine(query, invalid)
    except ValueError:
        pass
    else:
        raise AssertionError("Invalid vector should be rejected")
print({name: round(cosine(query, vector), 3) for name, vector in candidates.items()})

A zero vector has no direction, so the denominator is zero. Assigning it an arbitrary score can hide a failed embedding pipeline. Record and investigate zero vectors before indexing them. NaN and infinity also require an explicit policy; they should not be allowed to contaminate ranking calculations silently.

The function converts inputs to floats and can raise conversion errors for nonnumeric values. It is not a schema validator or a fast batch implementation. Its role is to make the arithmetic and failure cases inspectable without downloading a model.

Query and document preparation affect the score

Search may be symmetric, such as finding questions similar to another question, or asymmetric, such as matching a short question to a longer explanatory passage. The Sentence Transformers semantic search guide distinguishes those tasks and recommends using the model's query and document encoding interfaces where appropriate.

Follow the selected model's instructions for prompts, pooling, truncation, normalization, and scoring. A missing query prefix or changed pooling step can alter the representation even when the text is unchanged. Do not insert generic prefixes into every model without checking its contract.

Record the model revision, embedding dimensions, preprocessing policy, and corpus version. Re-embedding only queries with a different representation while keeping an incompatible old document index can make scores meaningless. A dimension check catches some mistakes, but not all incompatible changes.

Chunk boundaries matter too. A long page might discuss a refund exception in one paragraph and a general policy elsewhere. A single embedding can make it hard to retrieve the exact supporting passage. Test chunking using actual search questions and maintain source links and section context for the returned chunks.

Evaluate rankings against the task

An attractive score distribution is not an evaluation. Build a set of representative queries and label which corpus items satisfy each information need. Include easy queries, near-miss documents, exact identifiers, ambiguous wording, and queries for which the corpus has no adequate answer.

Precision at k measures how many of the first k results are relevant. Recall at k measures how many of the labeled relevant items are retrieved within those results. The Stanford information retrieval text defines precision and recall; using them at a cutoff makes the cutoff part of the claim.

For a tiny example, suppose “refund-policy” and “invoice-correction” are both relevant, while “general-faq” is not. Two rankings retrieve the same relevant items in their first three positions, but one presents useful material earlier.

def metrics(ranking, relevant, k):
    if k < 1 or len(ranking) < k or len(set(ranking)) != len(ranking):
        raise ValueError("Need a unique ranking with at least k results")
    if not relevant:
        raise ValueError("This exercise requires at least one relevant item")
    top = ranking[:k]
    hits = sum(item in relevant for item in top)
    first = next((position for position, item in enumerate(top, 1)
                  if item in relevant), None)
    return hits / k, hits / len(relevant), 0.0 if first is None else 1 / first

relevant = {"refund-policy", "invoice-correction"}
a = metrics(["refund-policy", "general-faq", "invoice-correction"], relevant, 3)
b = metrics(["general-faq", "invoice-correction", "refund-policy"], relevant, 3)
assert a == (2 / 3, 1.0, 1.0)
assert b == (2 / 3, 1.0, 0.5)
print("A:", a, "B:", b)

This function reports reciprocal rank at the chosen cutoff for one query. Averaging that quantity across queries produces a corresponding mean reciprocal rank measure. The Sentence Transformers retrieval evaluator supports retrieval metrics at configurable cutoffs.

The fixture is intentionally small and cannot estimate a system's general quality. Its purpose is to show why identical recall can coexist with different result ordering. For a real evaluation, disclose query count, corpus snapshot, label procedure, cutoff, and treatment of queries without relevant items.

A score is not confidence or factual support

Cosine 0.8 is not an 80% probability that a document answers the question. The score comes from vector geometry. A probabilistic interpretation would require a separately defined target, calibration method, and evaluation on suitable data.

The highest-scoring document can still be irrelevant when every candidate is poor. It can also be outdated, refer to a different product, or state the opposite of the requested claim while using similar vocabulary. Inspect dates, entities, negation, and the passage that actually supports the answer.

Evaluate thresholds on held-out labeled queries and review false positives and false negatives. Raising a threshold may reduce unsuitable results while also losing useful evidence. Scores from different models or preprocessing versions should not be assumed comparable under one permanent cutoff.

If exact codes or names are important, compare a lexical or hybrid retrieval baseline. If early ordering remains weak, test a reranker on the retrieved candidates. Evaluate each change against the same documented task rather than assuming a more elaborate pipeline must improve it.

For a generated answer, retrieval and answer verification remain separate stages. Read retrieval-augmented generation for the broader workflow and structured output versus factual validation for why a well-formed answer can still contain unsupported claims.

Sources and reproducibility

• Reimers and Gurevych, Sentence-BERT, 2019: learned sentence representations compared by cosine.
• Sentence Transformers similarity guide: scoring conventions and normalized dot product.
• Sentence Transformers semantic search guide: query/document preparation and search task types.
• Stanford information retrieval text: precision and recall.
• Sentence Transformers evaluators: metrics and cutoffs.

Both Python blocks run independently with the standard library. Every numerical result in the tables follows from the published coordinates or relevance labels. No universal similarity threshold or empirical accuracy claim is made.