lenatriestounderstand

Lab · runnable experiments

Chunking Strategies

Created Jul 4, 2026 Updated Jul 4, 2026

Read the parent note

Everyone agrees chunking matters, then splits documents every thousand characters and wonders why the answers drift. The note argues chunking is architectural, and that it fails in two opposite ways. Amputation: a chunk small enough to retrieve precisely, but cut off from the sentence that governs it — “vision is covered,” minus “only after twelve months.” Blur: a chunk big enough to hold the answer and five neighbouring rules it could be confused with — the whole refund policy returned for one question about annual plans. Shrink chunks to beat blur and you amputate; grow them to beat amputation and you blur.

This lab puts that trade-off on a scoreboard. Ten policy documents, sixty-one questions tagged by type, seven chunking strategies through one retriever. We score the obvious thing — recall@3, did we find the answer — and the things that usually go unmeasured: answer-supported (did we retrieve every span the answer depends on), distractor rate (did we drag in a competing rule that could mislead), and the context cost of doing so. The recurring lesson is that a healthy recall number can sit on top of answers that are quietly unsupported, quietly contaminated, or three times more expensive than they need to be — and that no single strategy wins every column.

Requirements

pip install sentence-transformers numpy pandas matplotlib torch

Setup

import re, datetime
import numpy as np, pandas as pd, torch
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle
from collections import defaultdict
from sentence_transformers import SentenceTransformer

np.random.seed(0)
# retrieval is CPU-cheap; a GPU, if present, only speeds the E5 embedding — the
# conclusions do not depend on it, though tiny numerical differences can occur across platforms
device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
e5 = SentenceTransformer("intfloat/multilingual-e5-base", device=device)
def embed(texts, kind):                                    # E5 wants query:/passage: prefixes
    return e5.encode([f"{kind}: {t}" for t in texts], normalize_embeddings=True, show_progress_bar=False)

INK, LINE, CREAM = "#2a2a2a", "#d9d3c7", "#f6f2e9"
BLUE, EMBER, FOREST, GOLD, GRAY = "#3b6ea5", "#c4521e", "#2e7d5b", "#c69214", "#9a9384"
plt.rcParams.update({"font.size": 9, "axes.edgecolor": LINE, "axes.linewidth": 0.8,
                     "axes.spines.top": False, "axes.spines.right": False,
                     "xtick.color": INK, "ytick.color": INK, "text.color": INK, "axes.labelcolor": INK})
print("executed:", datetime.date.today().isoformat())
executed: 2026-07-04

The corpus and the questions

Ten short policy documents — a health plan, a SaaS billing policy, a device warranty, travel insurance, a school attendance code, an expense policy, a bank’s chargeback rules, a cloud SLA, a rental agreement, a privacy policy. The corpus is synthetic on purpose: hand-writing the documents is what lets us plant known amputation and blur traps and score them against golden spans exactly, which a scraped real-world corpus would not. Two traps are wired in. Amputation traps: an answer sentence near the top of a section, its governing condition a few lines below, phrased with a back-reference (“the benefit,” “it”) so the condition carries none of the question’s words and won’t retrieve on its own. Blur traps: sections that stack several similar-but-distinct rules — the refund section holds monthly, annual, add-on, promo, enterprise and chargeback rules at once — so any chunk large enough to be safe from amputation drags competing rules along with it.

The ten documents (click to expand)
DOCS = {
"Northwind Health Plan — Benefits Summary": """## Vision
Vision correction is a covered benefit under every plan tier, including glasses and contact lenses.
Members may choose any in-network optometrist without a referral, and a routine eye exam is included once a year.
The plan reimburses up to two hundred dollars toward frames, which resets each January.
The benefit begins only after twelve months of continuous enrollment, so a new member is not eligible in the first year.

## Dental
Routine dental cleanings are covered twice per calendar year at no cost to the member.
Basic procedures such as fillings are covered at eighty percent after the deductible is met.
Major dental work such as crowns, bridges, and root canals requires pre-authorization from the plan before treatment.
Cosmetic procedures like whitening are never covered under any tier.

## Mental Health
Therapy sessions with a licensed in-network provider are fully covered with no copay.
Members can start care through the online portal or a primary-care referral.
This coverage is limited to twenty sessions per calendar year, after which the member pays out of pocket.
Inpatient psychiatric care follows the separate hospitalization schedule.""",

"Cloudly — Billing, Refunds and Cancellation Policy": """## Refunds
Monthly plans qualify for a full refund within thirty days of the original purchase.
Annual plans are non-refundable once the term has begun, and annual customers may only cancel future renewals.
Add-on purchases and one-time credits are non-refundable in all cases.
Promotional credits expire after ninety days and are never refundable for cash.
Enterprise contracts follow custom refund terms negotiated with the account manager.
A chargeback filed with the bank immediately suspends the workspace until the balance is settled.

## Cancellation
You may cancel your subscription at any time from the account dashboard, effective at the end of the current cycle.
Cancelling stops all future charges but does not refund the current billing period, which stays active until it ends.
There is no cancellation fee on any plan.

## Data retention
Your workspace data is retained for ninety days after cancellation so you can reactivate without loss.
During that window you can export everything as a single archive from the settings page.
After the ninety-day window the data is permanently deleted and cannot be recovered by support.""",

"Northwind Devices — Warranty and Returns": """## Warranty coverage
The standard warranty covers manufacturing defects for two years from the delivery date.
It includes the battery, the display, and all internal components under normal use.
It requires registering the device within thirty days of purchase, otherwise claims are declined.

## Accidental damage
Accidental damage such as cracked screens and liquid spills is not covered by the standard warranty.
Accidental damage protection can be added for an extra monthly fee within the first month of ownership.

## Returns
Unopened devices may be returned within fourteen days of delivery for a full refund.
Opened devices in good condition can still be returned, but are subject to a restocking fee of fifteen percent.
Engraved or custom-configured devices are final sale and cannot be returned.
Clearance items marked as final sale are not eligible for any return.
Bulk orders of ten units or more follow a separate return agreement signed at purchase.""",

"Globetrek Travel Insurance — Policy Summary": """## Medical
Emergency medical treatment abroad is covered up to one million dollars per trip.
The policy also covers emergency dental for the immediate relief of pain.
Coverage requires notifying the assistance line within twenty-four hours of admission.

## Trip cancellation
Cancellation due to a documented illness or injury is fully refundable.
Cancellation because you simply changed your mind is not covered under any tier.
Cancellation caused by severe weather is reimbursed at fifty percent of the trip cost.
Cancellation caused by an airline fault must be claimed from the airline, not the policy.
Losses arising from a declared pandemic are excluded from all cancellation cover.

## Baggage
Lost or stolen baggage is reimbursed up to one thousand dollars per person.
A claim requires filing a report with the carrier within twenty-one days of the incident.
Valuables such as jewelry and electronics are capped at two hundred dollars each.""",

"Riverside School — Attendance Policy": """## Absences
An absence for a documented illness is excused with a note from a parent or doctor.
An absence with no note within two school days is recorded as unexcused.
An absence for a family emergency is excused at the principal's discretion.
An absence for a recognized religious observance is excused when arranged in advance.
An absence for a college visit is excused for juniors and seniors only.

## Tardiness
Students are allowed three late arrivals per quarter without penalty.
The fourth late arrival in a quarter results in an after-school detention.

## Truancy
Truancy is reported to the district after ten unexcused absences in a school year.
A truancy referral triggers a mandatory meeting with the family.""",

"Acme Corp — Employee Expense Policy": """## Travel
Airfare is reimbursed in economy class only, unless a flight exceeds eight hours.
Hotel stays are reimbursed up to two hundred dollars per night, excluding taxes.
Meals are reimbursed up to seventy-five dollars per day on travel days.
Rideshare and taxi fares are reimbursed with an itemized receipt.
A rental car requires manager approval before the trip.

## Reimbursement
Expenses are reimbursed only when receipts are submitted within thirty days of the expense.
Reimbursements are paid to the employee with the next payroll cycle.

## Non-reimbursable
Alcohol is never reimbursed, even during a client dinner.
Personal entertainment such as movies and minibar charges is not reimbursed.
Traffic fines and parking tickets are the employee's own responsibility.""",

"First Meridian Bank — Card Chargeback Policy": """## Disputes
An unauthorized charge can be disputed for a provisional refund.
A dispute must be filed within sixty days of the statement showing the charge.
The cardholder must attempt to resolve the issue with the merchant first.

## Merchant errors
A duplicate charge for a single purchase is refunded once verified.
A charge for the wrong amount is corrected to the agreed price.
A charge for goods that never arrived is refunded after an investigation.
A subscription billed after cancellation is reversed with proof of the cancellation.

## Provisional credit
A provisional credit is issued while the dispute is investigated.
The credit is applied within ten business days of the filed dispute.
The credit is reversed if the dispute is later found to be invalid.""",

"StratusCloud — SLA and Support Tiers": """## Uptime
The platform guarantees ninety-nine point nine percent monthly uptime.
Scheduled maintenance windows are announced in advance and excluded from the calculation.
A service credit requires opening a ticket within thirty days of the incident.

## Support tiers
Basic support answers email within twenty-four hours on business days.
Standard support answers within eight hours including weekends.
Premium support offers one-hour phone response around the clock.
Enterprise support assigns a dedicated technical account manager.

## Service credits
An uptime between ninety-nine and ninety-nine point nine percent earns a ten percent credit.
An uptime between ninety-five and ninety-nine percent earns a twenty-five percent credit.
An uptime below ninety-five percent earns a fifty percent credit.""",

"Seaside Rentals — Booking and Cancellation": """## Cancellation
A cancellation thirty days or more before arrival receives a full refund.
A cancellation between fourteen and thirty days before arrival receives fifty percent.
A cancellation within fourteen days of arrival receives no refund.
A booking made during peak season is non-refundable regardless of notice.
If the owner cancels the booking, the guest receives a full refund plus a comparable rebooking.

## Check-in
Standard check-in begins at three in the afternoon.
An early check-in requires arrangement with the owner in advance.

## Damage deposit
The damage deposit is refunded after the property passes inspection.
The refund is issued within fourteen days of departure.""",

"Nimbus — Privacy and Data Retention": """## Retention
Account data is retained until the user requests deletion of the account.
Server logs are retained for ninety days and then rotated out.
Encrypted backups are kept for thirty days before they are overwritten.
Records under an active legal hold are retained indefinitely until the hold is lifted.
Marketing contact data is retained until the user opts out of communications.

## Deletion
A deletion request is honored for all personal data on file.
Deletion is completed within thirty days of the verified request.

## Sharing
Personal data is never sold to third parties.
Data is shared with processors only under a signed data processing agreement.
Data may be disclosed when required by a lawful legal request.""",
}

Each question carries its type, its answer span plus any governing conditions, and — for the competing-rule questions — the distractor spans it must not be answered from. Those distractors are what let us measure blur.

The sixty-one questions with golden spans (click to expand)
# (question, type, [answer, *conditions], [distractors])
QUESTIONS = [
 ("Is vision correction covered by the health plan?", "condition",
   ["Vision correction is a covered benefit", "after twelve months of continuous enrollment"], []),
 ("When do vision benefits start for a new member?", "condition",
   ["after twelve months of continuous enrollment"], []),
 ("Is a routine eye exam included?", "direct", ["a routine eye exam is included once a year"], []),
 ("Are routine dental cleanings covered?", "direct", ["Routine dental cleanings are covered twice"], []),
 ("Do I need approval before getting a crown?", "condition",
   ["Major dental work such as crowns", "requires pre-authorization from the plan"], []),
 ("Is teeth whitening covered?", "negative", ["Cosmetic procedures like whitening are never covered"], []),
 ("Is therapy covered by the plan?", "condition",
   ["Therapy sessions with a licensed", "limited to twenty sessions per calendar year"], []),
 ("How many therapy sessions do I get per year?", "condition",
   ["limited to twenty sessions per calendar year"], []),
 ("Can I get a refund on a monthly plan?", "competing",
   ["Monthly plans qualify for a full refund within thirty days"],
   ["Annual plans are non-refundable", "Add-on purchases and one-time credits are non-refundable", "Promotional credits expire after ninety days"]),
 ("Can I get a refund on an annual plan?", "competing",
   ["Annual plans are non-refundable once the term has begun"],
   ["Monthly plans qualify for a full refund within thirty days", "Add-on purchases and one-time credits are non-refundable"]),
 ("Are add-on purchases refundable?", "competing",
   ["Add-on purchases and one-time credits are non-refundable"],
   ["Monthly plans qualify for a full refund within thirty days", "Annual plans are non-refundable"]),
 ("Do promotional credits ever get refunded?", "competing",
   ["Promotional credits expire after ninety days and are never refundable"],
   ["Monthly plans qualify for a full refund within thirty days"]),
 ("What happens if I file a chargeback?", "direct",
   ["A chargeback filed with the bank immediately suspends the workspace"], []),
 ("If I cancel, do I get money back for the current period?", "condition",
   ["does not refund the current billing period"], []),
 ("Is there a fee to cancel?", "direct", ["There is no cancellation fee on any plan"], []),
 ("How long is my data kept after I cancel?", "direct",
   ["retained for ninety days after cancellation"], []),
 ("Can support recover my data long after cancelling?", "negative",
   ["permanently deleted and cannot be recovered"], []),
 ("How long is the device warranty?", "condition",
   ["covers manufacturing defects for two years", "registering the device within thirty days"], []),
 ("Do I need to register my device for the warranty?", "condition",
   ["registering the device within thirty days"], []),
 ("Is a cracked screen covered under warranty?", "negative",
   ["cracked screens and liquid spills is not covered"], []),
 ("Can I add accidental damage protection?", "condition",
   ["Accidental damage protection can be added", "within the first month of ownership"], []),
 ("Can I return an opened device?", "competing",
   ["subject to a restocking fee of fifteen percent"],
   ["Unopened devices may be returned within fourteen days", "Engraved or custom-configured devices are final sale"]),
 ("Can I return an engraved device?", "competing",
   ["Engraved or custom-configured devices are final sale"],
   ["Unopened devices may be returned within fourteen days", "subject to a restocking fee of fifteen percent"]),
 ("How long do I have to return an unopened device?", "competing",
   ["Unopened devices may be returned within fourteen days"],
   ["subject to a restocking fee of fifteen percent", "Engraved or custom-configured devices are final sale"]),
 ("Is emergency medical treatment abroad covered?", "condition",
   ["Emergency medical treatment abroad is covered", "notifying the assistance line within twenty-four hours"], []),
 ("Do I have to report a medical emergency within a time limit?", "condition",
   ["notifying the assistance line within twenty-four hours"], []),
 ("Can I cancel my trip because I got sick?", "competing",
   ["a documented illness or injury is fully refundable"],
   ["Cancellation because you simply changed your mind is not covered", "Cancellation caused by severe weather is reimbursed at fifty percent"]),
 ("Can I cancel because I changed my mind?", "competing",
   ["Cancellation because you simply changed your mind is not covered"],
   ["a documented illness or injury is fully refundable", "Cancellation caused by severe weather is reimbursed at fifty percent"]),
 ("What if bad weather forces me to cancel?", "competing",
   ["severe weather is reimbursed at fifty percent"],
   ["a documented illness or injury is fully refundable", "changed your mind is not covered"]),
 ("Is a pandemic-related cancellation covered?", "negative",
   ["a declared pandemic are excluded"], []),
 ("Is lost baggage covered, and any conditions?", "condition",
   ["Lost or stolen baggage is reimbursed up to one thousand dollars", "filing a report with the carrier within twenty-one days"], []),
 ("Is an absence for illness excused?", "competing",
   ["absence for a documented illness is excused"],
   ["no note within two school days is recorded as unexcused", "college visit is excused for juniors and seniors only"]),
 ("Is a college visit an excused absence?", "competing",
   ["college visit is excused for juniors and seniors only"],
   ["absence for a documented illness is excused", "family emergency is excused at the principal's discretion"]),
 ("What happens on my fourth late arrival?", "condition",
   ["fourth late arrival in a quarter results in an after-school detention"], []),
 ("How many late arrivals are allowed?", "direct",
   ["allowed three late arrivals per quarter"], []),
 ("When is truancy reported to the district?", "condition",
   ["reported to the district after ten unexcused absences"], []),
 ("Can I fly business class?", "condition",
   ["reimbursed in economy class only", "unless a flight exceeds eight hours"], []),
 ("How much can I spend on a hotel per night?", "competing",
   ["reimbursed up to two hundred dollars per night"],
   ["Meals are reimbursed up to seventy-five dollars per day", "Airfare is reimbursed in economy class only"]),
 ("What is the daily meal limit?", "competing",
   ["reimbursed up to seventy-five dollars per day"],
   ["reimbursed up to two hundred dollars per night", "Airfare is reimbursed in economy class only"]),
 ("Do I need approval for a rental car?", "direct",
   ["rental car requires manager approval"], []),
 ("By when must I submit expense receipts?", "condition",
   ["receipts are submitted within thirty days"], []),
 ("Can I expense alcohol at a client dinner?", "negative",
   ["Alcohol is never reimbursed"], []),
 ("How long do I have to dispute a charge?", "condition",
   ["dispute must be filed within sixty days"], []),
 ("Can I dispute an unauthorized charge right away?", "condition",
   ["unauthorized charge can be disputed", "must attempt to resolve the issue with the merchant first"], []),
 ("What if I was charged twice for one purchase?", "competing",
   ["duplicate charge for a single purchase is refunded"],
   ["charge for the wrong amount is corrected", "goods that never arrived is refunded after an investigation"]),
 ("What if I was billed after cancelling a subscription?", "competing",
   ["subscription billed after cancellation is reversed"],
   ["duplicate charge for a single purchase is refunded", "charge for the wrong amount is corrected"]),
 ("When do I get the provisional credit?", "condition",
   ["applied within ten business days"], []),
 ("What uptime is guaranteed?", "direct",
   ["ninety-nine point nine percent monthly uptime"], []),
 ("How do I claim a service credit?", "condition",
   ["service credit requires opening a ticket within thirty days"], []),
 ("What response time does Premium support give?", "competing",
   ["Premium support offers one-hour phone response"],
   ["Basic support answers email within twenty-four hours", "Standard support answers within eight hours"]),
 ("What does Basic support offer?", "competing",
   ["Basic support answers email within twenty-four hours"],
   ["Premium support offers one-hour phone response", "Enterprise support assigns a dedicated technical account manager"]),
 ("What credit do I get for uptime below ninety-five percent?", "competing",
   ["below ninety-five percent earns a fifty percent credit"],
   ["ninety-nine and ninety-nine point nine percent earns a ten percent credit", "ninety-five and ninety-nine percent earns a twenty-five percent credit"]),
 ("Do I get a refund if I cancel three weeks before arrival?", "competing",
   ["between fourteen and thirty days before arrival receives fifty percent"],
   ["thirty days or more before arrival receives a full refund", "within fourteen days of arrival receives no refund"]),
 ("Is a peak-season booking refundable?", "competing",
   ["peak season is non-refundable regardless of notice"],
   ["thirty days or more before arrival receives a full refund", "between fourteen and thirty days before arrival receives fifty percent"]),
 ("What time is check-in?", "direct",
   ["check-in begins at three in the afternoon"], []),
 ("When do I get my damage deposit back?", "condition",
   ["refunded after the property passes inspection", "issued within fourteen days of departure"], []),
 ("How long are server logs kept?", "competing",
   ["Server logs are retained for ninety days"],
   ["Account data is retained until the user requests deletion", "Encrypted backups are kept for thirty days"]),
 ("How long are backups kept?", "competing",
   ["Encrypted backups are kept for thirty days"],
   ["Server logs are retained for ninety days", "Account data is retained until the user requests deletion"]),
 ("How long does account deletion take?", "condition",
   ["completed within thirty days of the verified request"], []),
 ("Is my personal data sold to third parties?", "negative",
   ["Personal data is never sold"], []),
 ("Can my data be disclosed for legal reasons?", "direct",
   ["disclosed when required by a lawful legal request"], []),
]
from collections import Counter
counts = Counter(t for _, t, _, _ in QUESTIONS)
print(f"{len(DOCS)} documents · {len(QUESTIONS)} questions")
print("by type:", dict(counts))
print(f"multi-span questions needing answer + condition: {sum(len(a) > 1 for _, t, a, _ in QUESTIONS if t != 'competing')}"
      f"  ·  competing questions (with distractors): {sum(bool(d) for *_, d in QUESTIONS)}")
10 documents · 61 questions
by type: {'condition': 22, 'direct': 10, 'negative': 6, 'competing': 23}
multi-span questions needing answer + condition: 10  ·  competing questions (with distractors): 23

Seven ways to cut the same text

Each chunker returns (embed_text, context_text) pairs. For most they match; parent-child is the exception — it retrieves on a small child but returns the whole parent section, which is the entire point of it.

def clean(t): return " ".join(t.split())
def sents(t): return [s.strip() for s in re.split(r"(?<=[.])\s+", t) if s.strip()]
def secs(doc):
    out, cur = [], None
    for line in doc.splitlines():
        if line.startswith("## "):
            if cur: out.append(cur)
            cur = [line[3:].strip(), ""]
        elif cur is not None and line.strip(): cur[1] += " " + line.strip()
    if cur: out.append(cur)
    return out
def by_size(text, size):                                   # greedy sentence packing under a char budget
    out, buf = [], ""
    for s in sents(text):
        if len(buf) + len(s) + 1 <= size: buf = (buf + " " + s).strip()
        else:
            if buf: out.append(buf)
            buf = s
    if buf: out.append(buf)
    return out

SIZE = 220
def ck_fixed(t, d, size=SIZE):                             # raw character windows — respects nothing
    b = clean(re.sub("## ", "", d)); return [(b[i:i+size], b[i:i+size]) for i in range(0, len(b), size)]
def ck_overlap(t, d, size=SIZE, ov=80):
    b = clean(re.sub("## ", "", d)); step = size - ov
    return [(b[i:i+size], b[i:i+size]) for i in range(0, len(b), step)]
def ck_recursive(t, d, size=SIZE):                         # split on sentence boundaries, pack under size
    return [(c, c) for h, b in secs(d) for c in by_size(clean(b), size)]
def ck_semantic(t, d, drop=0.12):                          # break where consecutive sentences diverge
    out = []
    for h, b in secs(d):
        ss = sents(clean(b))
        if not ss: continue
        E = embed(ss, "passage"); buf = [ss[0]]
        for i in range(1, len(ss)):
            if float(E[i] @ E[i-1]) < 1 - drop and len(" ".join(buf)) > 60:
                out.append(" ".join(buf)); buf = [ss[i]]
            else: buf.append(ss[i])
        out.append(" ".join(buf))
    return [(c, c) for c in out]
def ck_section(t, d):                                      # one chunk per section
    return [(f"{h}. {clean(b)}", f"{h}. {clean(b)}") for h, b in secs(d)]
def ck_parent_child(t, d, size=SIZE):                      # retrieve small children, return the parent section
    out = []
    for h, b in secs(d):
        parent = f"{h}. {clean(b)}"
        for child in by_size(clean(b), size): out.append((child, parent))
    return out
def ck_contextual(t, d, size=SIZE):                        # prepend document + section context before embedding
    out = []
    for h, b in secs(d):
        for child in by_size(clean(b), size): out.append((f"{t}{h}: {child}",) * 2)
    return out

STRATEGIES = [("fixed-size", ck_fixed), ("fixed + overlap", ck_overlap), ("recursive", ck_recursive),
              ("semantic", ck_semantic), ("section-aware", ck_section),
              ("parent-child", ck_parent_child), ("contextual", ck_contextual)]

A note on honesty: contextual here prepends the document and section heading as a cheap, deterministic stand-in for the note’s version, which spends an LLM call per chunk to write that context. Same idea — give the chunk knowledge of its surroundings — without an API in the render loop.

Retrieve, then score what actually matters

Embed every chunk, embed every question, take the top-3 by cosine similarity (with parent-child, the child ranks but its parent is what we count). Then six numbers. Two are the usual retrieval metrics — recall@3 (is the answer there?) and answer-supported (are all the needed spans there — answer and every condition?). One synthesizes them — clean grounding, the share of questions where we retrieved everything the answer needs and nothing that could mislead it. The last three are the ones pipelines skip: distractor rate, the share of competing-rule questions whose top-3 also contains a rival rule that could mislead the model; answer density, the fraction of retrieved characters that are actually part of the answer rather than filler; and context chars, the raw size of what we hand the LLM. Since there is no LLM in this loop, distractor rate and answer density are proxies for blur — they measure the contamination and bloat that make a model pick the wrong rule, without pretending to measure the model itself. Note that answer density is not classical retrieval precision: it is how much of the text handed to the model is actually needed for the answer, which penalises a big blurry chunk even when that chunk technically contains the right span.

qemb = embed([q for q, *_ in QUESTIONS], "query")

def retrieved(chunks, cemb, qi, k=3):
    order = np.argsort(-(cemb @ qemb[qi])); seen, cov = set(), []
    for idx in order:
        c = chunks[idx][1]
        if c in seen: continue
        seen.add(c); cov.append(c)
        if len(cov) >= k: break
    return cov

def answer_density(cov, spans):                            # fraction of retrieved chars that are answer, not filler
    tot = sum(len(c) for c in cov)
    return sum(len(s) for c in cov for s in spans if s in c) / tot if tot else 0.0

HEAT, CLEAN_AGG = {}, {}                                    # per-type rates; question-weighted clean grounding
def run(name, chunker):
    chunks = [(ct, cx) for t, d in DOCS.items() for ct, cx in chunker(t, d)]
    cemb = embed([c for c, _ in chunks], "passage")
    rec = sup = fneigh = precip = ctx = clean_g = 0.0; ncomp = 0
    bytype = defaultdict(lambda: [0, 0])
    for qi, (q, typ, ans, dist) in enumerate(QUESTIONS):
        cov = retrieved(chunks, cemb, qi); j = " || ".join(cov)
        supported = all(a in j for a in ans)
        contaminated = any(d in j for d in dist)
        clean_ground = supported and not contaminated                  # got everything, misled by nothing
        rec += ans[0] in j; sup += supported; clean_g += clean_ground
        precip += answer_density(cov, ans); ctx += sum(len(c) for c in cov)
        if dist: ncomp += 1; fneigh += contaminated
        bytype[typ][0] += clean_ground; bytype[typ][1] += 1
    n = len(QUESTIONS)
    HEAT[name] = {t: v[0] / v[1] for t, v in bytype.items()}
    CLEAN_AGG[name] = clean_g / n                                       # share of ALL questions, not mean-of-types
    return {"strategy": name, "chunks": len(chunks), "recall@3": round(rec/n, 2),
            "supported": round(sup/n, 2), "clean grounding": round(clean_g/n, 2),
            "distractor rate*": round(fneigh/ncomp, 2),
            "answer density": round(precip/n, 2), "ctx chars": round(ctx/n)}

table = pd.DataFrame([run(name, fn) for name, fn in STRATEGIES])
table
strategy chunks recall@3 supported clean grounding distractor rate* answer density ctx chars
0 fixed-size 46 0.77 0.69 0.43 0.78 0.06 631
1 fixed + overlap 67 1.00 0.97 0.64 0.87 0.10 627
2 recursive 54 1.00 0.98 0.66 0.87 0.11 486
3 semantic 87 0.95 0.85 0.61 0.65 0.13 388
4 section-aware 30 0.98 0.98 0.61 1.00 0.06 941
5 parent-child 54 1.00 1.00 0.62 1.00 0.06 978
6 contextual 54 1.00 1.00 0.69 0.83 0.09 639

distractor rate* is measured over the 23 competing-rule questions only; every other column is over all 61.

Read across a single row and the trap is obvious. section-aware and parent-child post a perfect supported of 1.00 — and a distractor rate of 1.00 with the largest context on the board, which is why their clean grounding lands mid-pack despite that perfect support. They never amputate because they always return the whole section; they always contaminate for exactly the same reason. fixed-size is the opposite kind of bad: character windows so crude they miss the answer outright a quarter of the time (recall@3 0.77, clean grounding 0.43). The strategies that actually balance the two — recursive, contextual, overlap — keep supported high while pulling in less filler, and top the clean grounding column. There is no row that wins every column; that is the whole point of the chapter, made numeric.

The scoreboard

The one number a retrieval dashboard usually shows is recall@3. Put it next to answer-supported and next to clean grounding — the share of questions where we retrieved everything the answer needs and nothing that could mislead it — and the strategies re-sort.

order = sorted(CLEAN_AGG, key=CLEAN_AGG.get)                # share of all 61 questions, matches the table
tbl = table.set_index("strategy")
rec = [tbl.loc[s, "recall@3"] for s in order]
sup = [tbl.loc[s, "supported"] for s in order]
cln = [CLEAN_AGG[s] for s in order]
x = np.arange(len(order)); w = 0.26
fig, ax = plt.subplots(figsize=(7.4, 3.9))
ax.set_axisbelow(True)                                      # grid behind the bars, so heights are readable
ax.grid(axis="y", which="major", color=LINE, linewidth=0.9)
ax.grid(axis="y", which="minor", color=LINE, linewidth=0.6, alpha=0.6)
ax.bar(x - w, rec, w, label="recall@3", color=GRAY)
ax.bar(x,     sup, w, label="answer-supported", color=BLUE)
ax.bar(x + w, cln, w, label="clean grounding", color=EMBER)
ax.set_xticks(x); ax.set_xticklabels(order, rotation=28, ha="right")
ax.set_yticks(np.arange(0, 1.01, 0.2)); ax.set_yticks(np.arange(0, 1.01, 0.1), minor=True)
ax.set_ylim(0, 1.05); ax.set_ylabel("rate")
ax.legend(loc="lower center", bbox_to_anchor=(0.5, 1.01), ncol=3, frameon=False, fontsize=8)
plt.tight_layout(); plt.show()

recall@3 flatters everyone; answer-supported separates the amputators from the rest; clean grounding — complete AND uncontaminated — is where the whole-section strategies give back what they won. Sorted by clean grounding.

The grey bars are almost level — by recall alone you would call these strategies interchangeable. The orange bars are not: contextual and recursive lead on clean grounding while section-aware and parent-child, the two that looked perfect on supported, fall back, because their guaranteed contamination costs them every competing-rule question. A recall-only scoreboard would have shipped exactly the wrong choice.

Where each strategy breaks

Averages hide which questions break. Score clean grounding per strategy per question type and the two failure modes separate cleanly.

types = ["direct", "condition", "negative", "competing"]
labels = ["direct\nfact", "answer +\ncondition", "negative", "competing\nrule"]
names = [n for n, _ in STRATEGIES]
M = np.array([[HEAT[n].get(t, np.nan) for t in types] for n in names])
fig, ax = plt.subplots(figsize=(6.4, 4.2))
im = ax.imshow(M, cmap="RdYlGn", vmin=0, vmax=1, aspect="auto")
ax.set_xticks(range(len(types))); ax.set_xticklabels(labels, fontsize=8)
ax.set_yticks(range(len(names))); ax.set_yticklabels(names, fontsize=8)
for i in range(len(names)):
    for j in range(len(types)):
        ax.text(j, i, f"{M[i, j]:.2f}", ha="center", va="center", fontsize=8,
                color="#222" if 0.25 < M[i, j] < 0.85 else "#fff")
ax.set_title("clean grounding · strategy × question type", fontsize=9, pad=8)
plt.tight_layout(); plt.show()

Clean-grounding rate by strategy and question type. The condition column is the amputation axis — small/crude chunks drop the governing clause, whole-section chunks fix it. The competing column is the blur axis — and every strategy struggles there, because at k=3 a section of near-identical rules always leaks a distractor. Chunking fixes amputation; it does not fix blur.

The left three columns are the amputation story: fixed-size is red across the board, and the condition column climbs from crude to structured chunking as the governing clause stops getting cut. The right column is the blur story, and it is red down the whole column — a structural 0.00 for the whole-section strategies, and only partially recovered even by the best of the rest. The spread is real but small; the honest reading is that no chunking strategy fixes competing-rule contamination at k=3. That failure lives one layer up, in metadata filters, rerankers, and query routing — not in where you draw the chunk boundary.

Three autopsies

Aggregates convince the head; one query at a time convinces the gut. Here are the two failure modes and the distractor between them, caught in the act.

Amputation. Take the first condition question that the sentence-respecting recursive splitter — not a strawman — retrieves the answer for yet fails to support, and map which sentences of the answer’s section each strategy actually delivers.

rc = [(a, b) for t, d in DOCS.items() for a, b in ck_recursive(t, d)]
pc = [(a, b) for t, d in DOCS.items() for a, b in ck_parent_child(t, d)]
rce, pce = embed([c for c, _ in rc], "passage"), embed([c for c, _ in pc], "passage")
def joined(chunks, cemb, qi): return " || ".join(retrieved(chunks, cemb, qi))
qi = next(i for i, (q, t, a, d) in enumerate(QUESTIONS)
          if t == "condition" and len(a) > 1 and a[0] in joined(rc, rce, i) and a[1] not in joined(rc, rce, i))
q, typ, ans, _ = QUESTIONS[qi]
# locate the section holding the answer and split it into sentences
sec_text = next(clean(b) for _, d in DOCS.items() for h, b in secs(d) if ans[0] in clean(b))
sec_sents = sents(sec_text)
def role(s):
    if ans[0] in s: return FOREST
    if any(a in s for a in ans[1:]): return EMBER
    return "#e7e2d6"
rows = [("recursive", joined(rc, rce, qi)), ("parent-child", joined(pc, pce, qi))]
fig, ax = plt.subplots(figsize=(7.4, 2.5))
for r, (nm, ctx) in enumerate(rows):
    y = len(rows) - 1 - r
    for c, s in enumerate(sec_sents):
        covered = s in ctx
        ax.add_patch(Rectangle((c, y), 0.92, 0.8, facecolor=role(s), alpha=0.95 if covered else 0.28,
                     edgecolor=INK if covered else "none", linewidth=1.8 if covered else 0))
    ax.text(-0.15, y + 0.4, nm, ha="right", va="center", fontsize=9)
ax.set_xlim(-2.2, len(sec_sents)); ax.set_ylim(-0.2, len(rows))
ax.set_xticks([]); ax.set_yticks([]); [ax.spines[s].set_visible(False) for s in ax.spines]
ax.text(len(sec_sents)/2, len(rows)-0.02, f"Q: {q}", ha="center", va="bottom", fontsize=8.5, style="italic")
handles = [Rectangle((0,0),1,1,facecolor=FOREST), Rectangle((0,0),1,1,facecolor=EMBER),
           Rectangle((0,0),1,1,facecolor="#e7e2d6"), Rectangle((0,0),1,1,facecolor="#fff",edgecolor=INK,linewidth=1.6)]
ax.legend(handles, ["answer", "condition", "filler", "retrieved"], ncol=4, loc="lower center",
          bbox_to_anchor=(0.5, -0.18), fontsize=8, frameon=False)
plt.tight_layout(); plt.show()

One section, sentence by sentence. Green is the answer, orange is the governing condition, grey is filler; a solid outline means the sentence made it into the top-3 context. Recursive retrieves the answer and leaves the condition on the cutting-room floor; parent-child returns the whole section, so the condition rides along.

Distractor on top. The same recursive splitter, on a competing-rule question, ranks a different rule above the right one — the answer is retrieved, but a rival sits on top of it, and a system that reads rank-1 first answers from the wrong clause.

fi = next(i for i, (q, t, a, d) in enumerate(QUESTIONS) if t == "competing"
          and (lambda cov: any(x in c for c in cov for x in d) and
               (a[0] not in " || ".join(cov[:1])))(retrieved(rc, rce, i)))
q, typ, ans, dist = QUESTIONS[fi]; cov = retrieved(rc, rce, fi)
print(f"Q: {q}\n  right answer: «{ans[0]}»\n")
for r, c in enumerate(cov, 1):
    tag = "  <- ANSWER" if ans[0] in c else ("  <- a competing rule" if any(x in c for x in dist) else "")
    print(f"  rank {r}: {c[:88]}{'…' if len(c) > 88 else ''}{tag}")
Q: Is a college visit an excused absence?
  right answer: «college visit is excused for juniors and seniors only»

  rank 1: An absence for a family emergency is excused at the principal's discretion. An absence f…  <- a competing rule
  rank 2: An absence for a documented illness is excused with a note from a parent or doctor. An a…  <- a competing rule
  rank 3: An absence for a college visit is excused for juniors and seniors only.  <- ANSWER

Blur. And the whole-section strategy on that same family of question hands back one dense blob — the answer and every rival rule fused into a single vector, so retrieval can’t tell them apart and the model has to.

sec = [(a, b) for t, d in DOCS.items() for a, b in ck_section(t, d)]
sece = embed([c for c, _ in sec], "passage")
bi = next(i for i, (q, t, a, d) in enumerate(QUESTIONS) if "annual plan" in q)
q, typ, ans, dist = QUESTIONS[bi]; top = retrieved(sec, sece, bi)[0]
present = sum(d in top for d in dist)
print(f"Q: {q}\n  right answer: «{ans[0]}»\n")
print(f"  section-aware returns one chunk of {len(top)} characters:")
print(f"  “{top[:340]}…”\n")
print(f"  answer present: {ans[0] in top}   ·   competing rules also in that chunk: {present} of {len(dist)}")
Q: Can I get a refund on an annual plan?
  right answer: «Annual plans are non-refundable once the term has begun»

  section-aware returns one chunk of 538 characters:
  “Refunds. Monthly plans qualify for a full refund within thirty days of the original purchase. Annual plans are non-refundable once the term has begun, and annual customers may only cancel future renewals. Add-on purchases and one-time credits are non-refundable in all cases. Promotional credits expire after ninety days and are never refun…”

  answer present: True   ·   competing rules also in that chunk: 2 of 2

Chunk size trades one failure for another

If amputation comes from chunks too small and blur from chunks too big, it is tempting to believe there is a size that beats both. Sweep the recursive splitter from one sentence to whole-section and watch two metrics move in opposite directions.

sizes = [60, 110, 170, 240, 340, 500, 750, 1100]
sup_s, prec_s, ctx_s = [], [], []
for size in sizes:
    ch = [(c, c) for t, d in DOCS.items() for h, b in secs(d) for c in by_size(clean(b), size)]
    ce = embed([c for c, _ in ch], "passage")
    s = p = cx = 0.0
    for qi, (q, t, ans, d) in enumerate(QUESTIONS):
        cov = retrieved(ch, ce, qi)
        s += all(a in " || ".join(cov) for a in ans); p += answer_density(cov, ans); cx += sum(len(c) for c in cov)
    n = len(QUESTIONS); sup_s.append(s/n); prec_s.append(p/n); ctx_s.append(cx/n)

fig, ax1 = plt.subplots(figsize=(7.2, 3.7))
ax1.set_axisbelow(True); ax1.grid(axis="y", color=LINE, linewidth=0.8, alpha=0.8)
ax1.plot(sizes, sup_s, "o-", color=BLUE, label="answer-supported ↑")
ax1.plot(sizes, prec_s, "o-", color=EMBER, label="answer density ↓")
ax1.set_xlabel("chunk size (characters)"); ax1.set_ylabel("rate"); ax1.set_ylim(0, 1.05)
ax2 = ax1.twinx(); ax2.plot(sizes, ctx_s, "o--", color=GRAY, label="context chars (cost)")
ax2.set_ylabel("avg context chars", color=GRAY); ax2.tick_params(axis="y", colors=GRAY); ax2.spines["right"].set_visible(True)
l1, la1 = ax1.get_legend_handles_labels(); l2, la2 = ax2.get_legend_handles_labels()
ax1.legend(l1 + l2, la1 + la2, loc="center right", fontsize=8, framealpha=0.9)
plt.tight_layout(); plt.show()

As chunks grow, answer-supported rises toward 1.0 (amputation heals) while answer density — the share of retrieved text that is actually the answer — falls away (blur sets in), and the context you pay for more than triples. The metric a dashboard watches goes green; the one it doesn’t goes red. There is no size that wins both.

answer-supported climbs monotonically — read it alone and you would keep enlarging chunks forever. answer density falls just as steadily, and the context bill triples along the way. The two curves are a scissors, not a hill: growing the chunk does not find a sweet spot, it moves the failure from amputation to blur while charging you more tokens for the privilege. Which side of the scissors you want depends on which failure your application can least afford — and that is a design decision, not a default.

What we just did

We put seven chunking strategies through one corpus and sixty-one typed questions, and scored them past the usual recall number: did we retrieve everything the answer depends on, and nothing that could mislead it. Two failure modes fell out of real retrieval, not assertion. Amputation — small and crude chunks dropping the governing condition — is fixed by giving the splitter structure; the whole-section strategies never amputate. Blur — big chunks fusing an answer with its rivals — is the price they pay for it, and it turned out to be the failure no chunking strategy solved at k=3: competing near-identical rules leak into the top-3 regardless of where you draw the boundary. The size sweep made the trade-off literal: supported and answer density move in opposite directions, so chunk size relocates the failure rather than removing it. The honest conclusions are the two the note keeps insisting on. First, a healthy recall@3 is an upper bound on grounding, not a measure of it — the strategies that tied on recall ranged from 0.43 to 0.69 on clean grounding. Second, chunking is one layer of a retrieval system, not the whole of it: it can buy back amputation, but blur on competing rules is a job for metadata, reranking, and query routing — the tools the note reaches for next.

Read the parent note