feat(evals): cache haystack document embeddings across benchmark runs (#1973)

Local-embeddings ingest is almost all embedding work on a haystack that is
identical every run. The harness now points each local-embeddings server at
evals/.cache/embeddings/ through the eval-only AI_MEMORY_EVAL_EMBEDDING_CACHE_DIR
variable (read once at startup, like AI_MEMORY_EVAL_SKIP_WIKI_CHECKPOINTS).
The server wraps its embedder so embed_document vectors are stored under
{provider, model identity, dim, sha256(text)}; queries stay live and the
wrapper reports the inner identity, so stored rows keep their triple.

The text hash masks RFC 3339 UTC timestamps: replayed session pages list
the server's wall-clock event times, so an exact-text key missed on every
page. --no-embedding-cache runs cold; reports record embedding_cache.

Measured on --stratify 40 (230 questions): cold 1,015 s, cached 132 s,
with every question's hit/recall/retrieved sessions identical to the run
that filled the cache. Cold runs differ from each other on 10-12 questions
(#1854). Release numbers stay cold (AGENTS.md).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.gitignore

@@ -51,3 +51,4 @@ evals/runs/
 *.bak-*
 evals/datasets/
 evals/models/
+evals/.cache/

AGENTS.md

@@ -746,6 +746,13 @@ Additional boundary rules:
     throughput, and say for each material change whether it is a deliberate
     trade the CHANGELOG states or a possible regression. Run-to-run noise is
     about ±0.005 overall and ±0.02–0.03 on small slices.
+  - *Published local numbers come from a cold run.* Run the release
+    candidate's local leg with `--no-embedding-cache`. A cached run reuses
+    one earlier cold run's document vectors, its ingest throughput and query
+    latency measure the cache, and its key names the model, not the
+    embedding code. Cached
+    runs are for iteration: re-running the questions that moved, or comparing
+    two builds without the cold-run variance (`docs/benchmarks/README.md`).
   - *A drop does not block the release.* Publish the honest numbers in
     `docs/benchmarks/`, keep `docs/comparison.md` and
     `docs/competitive-parity.md` in step, and open an issue for the next

CHANGELOG.md

@@ -95,6 +95,13 @@ version in [`docs/changelog/`](docs/changelog/):
   a WAL-consistent read-only snapshot, with explicit project mapping,
   client-side sanitization, sensitive-record exclusion, exact-body resume, and
   dry-run by default (#1840).
+- Added a document-embedding cache to the `ai-memory-eval retrieval`
+  benchmark. Local-embeddings runs store each haystack page's vector under
+  `evals/.cache/embeddings/`, keyed by provider, model, dimension and a hash of
+  the page text with the replay's wall-clock timestamps masked, and later runs
+  reuse it. Query embeddings and ranking stay live. `--no-embedding-cache`
+  runs cold. The server side is an eval-only variable the harness sets; normal
+  configuration has no such setting. (#1973)
 - Added `ai-memory-eval retrieval-compare`, which pairs two saved retrieval
   reports by question and lists the questions whose first hit moved, the
   cutoffs each crossed, and how many moved per question type. `--fts` marks

Cargo.lock

@@ -47,6 +47,7 @@ dependencies = [
  "ai-memory-workstream",
  "anyhow",
  "askama",
+ "async-trait",
  "axum",
  "base64 0.22.1",
  "clap",

crates/ai-memory-cli/Cargo.toml

@@ -43,6 +43,7 @@ secrecy.workspace = true
 sha2.workspace = true
 base64.workspace = true
 anyhow.workspace = true
+async-trait.workspace = true
 axum.workspace = true
 # Sets SO_KEEPALIVE on accepted `serve` sockets (#792); not a workspace dep,
 # `ai-memory-cli` is the only crate that needs it.

crates/ai-memory-cli/src/commands/eval_embedding_cache.rs

@@ -0,0 +1,347 @@
+//! Benchmark-only document-embedding cache (#1973).
+//!
+//! The LongMemEval harness in `evals/` replays the same haystack sessions on
+//! every run, and in local-embeddings mode embedding them is almost all of the
+//! ingest time. When the harness starts a server with
+//! `AI_MEMORY_EVAL_EMBEDDING_CACHE_DIR` set, the server wraps its embedder in
+//! [`EvalEmbeddingCache`], which stores each document vector on disk under
+//! `{provider, model identity, dim, sha256(text)}` and serves it back on the
+//! next run instead of calling the model.
+//!
+//! The text hash skips RFC 3339 UTC timestamps. The server stamps replayed
+//! hook events with its own clock, and the session pages built from them list
+//! those times (`started_at`, each observation's `@` time), so the same
+//! haystack session never produces the same page text twice. With the times
+//! masked, a later run reuses the vector the first run computed for that page;
+//! the page text it ranks differs only in the replay's wall-clock times.
+//!
+//! Only [`Embedder::embed_document`] is cached. Query embeddings, ranking and
+//! fusion stay live, so a cached run still measures the code under test. The
+//! wrapper reports the inner embedder's identity unchanged, so every stored
+//! row keeps its `{provider, model, dim}` (invariant #8), and a model or
+//! dimension change lands in a different cache directory.
+//!
+//! The variable is read once at startup, only the eval harness sets it, and
+//! it is deliberately absent from `config.toml` and the documented settings.
+
+use std::path::{Path, PathBuf};
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, Ordering};
+
+use ai_memory_llm::{Embedder, LlmResult};
+use async_trait::async_trait;
+use sha2::{Digest, Sha256};
+
+/// Environment variable naming the cache root. Set only by the eval harness.
+pub(crate) const EVAL_EMBEDDING_CACHE_DIR_ENV: &str = "AI_MEMORY_EVAL_EMBEDDING_CACHE_DIR";
+
+/// Read the benchmark cache root once at startup. An empty value is off.
+pub(crate) fn eval_embedding_cache_dir() -> Option<PathBuf> {
+    std::env::var_os(EVAL_EMBEDDING_CACHE_DIR_ENV)
+        .filter(|value| !value.is_empty())
+        .map(PathBuf::from)
+}
+
+/// Wraps an embedder and caches its document vectors on disk.
+pub(crate) struct EvalEmbeddingCache {
+    inner: Arc<dyn Embedder>,
+    /// `<root>/<provider>-<dim>-<model digest>`: one directory per identity.
+    dir: PathBuf,
+    dim: u32,
+    tmp_seq: AtomicU64,
+}
+
+impl EvalEmbeddingCache {
+    pub(crate) fn new(inner: Arc<dyn Embedder>, root: &Path) -> Self {
+        let dim = inner.dim();
+        let model_digest = hex_digest(inner.model_identity().as_bytes());
+        let dir = root.join(format!(
+            "{}-{dim}-{}",
+            inner.provider(),
+            &model_digest[..16]
+        ));
+        Self {
+            inner,
+            dir,
+            dim,
+            tmp_seq: AtomicU64::new(0),
+        }
+    }
+
+    fn entry_path(&self, text: &str) -> PathBuf {
+        let key = text_key(text);
+        self.dir.join(&key[..2]).join(format!("{key}.f32"))
+    }
+
+    /// A stored vector, or `None` for a miss or an entry of the wrong size.
+    async fn load(&self, path: &Path) -> Option<Vec<f32>> {
+        let bytes = tokio::fs::read(path).await.ok()?;
+        if bytes.len() != self.dim as usize * 4 {
+            return None;
+        }
+        Some(
+            bytes
+                .chunks_exact(4)
+                .map(|c| f32::from_le_bytes([c[0], c[1], c[2], c[3]]))
+                .collect(),
+        )
+    }
+
+    /// Best effort: a failed write only costs a recompute on the next run.
+    /// Writes go through a unique temp file and a rename, so a concurrent
+    /// reader never sees a partial vector.
+    async fn store(&self, path: &Path, vector: &[f32]) {
+        if vector.len() != self.dim as usize {
+            return;
+        }
+        let Some(parent) = path.parent() else {
+            return;
+        };
+        let bytes: Vec<u8> = vector.iter().flat_map(|v| v.to_le_bytes()).collect();
+        let tmp = parent.join(format!(
+            ".tmp-{}-{}",
+            std::process::id(),
+            self.tmp_seq.fetch_add(1, Ordering::Relaxed)
+        ));
+        let result = async {
+            tokio::fs::create_dir_all(parent).await?;
+            tokio::fs::write(&tmp, &bytes).await?;
+            tokio::fs::rename(&tmp, path).await
+        }
+        .await;
+        if let Err(error) = result {
+            let _ = tokio::fs::remove_file(&tmp).await;
+            tracing::debug!(%error, "eval embedding cache write failed");
+        }
+    }
+}
+
+fn hex_digest(bytes: &[u8]) -> String {
+    hex(&Sha256::digest(bytes))
+}
+
+fn hex(bytes: &[u8]) -> String {
+    bytes.iter().map(|b| format!("{b:02x}")).collect()
+}
+
+/// sha256 of `text` with every `YYYY-MM-DDTHH:MM:SS[.fff]Z` replaced by a
+/// fixed marker, so a replay's wall-clock times do not change the key.
+fn text_key(text: &str) -> String {
+    let bytes = text.as_bytes();
+    let mut hasher = Sha256::new();
+    let (mut start, mut i) = (0, 0);
+    while i < bytes.len() {
+        match timestamp_len(&bytes[i..]) {
+            Some(len) => {
+                hasher.update(&bytes[start..i]);
+                hasher.update(b"\0ts\0");
+                i += len;
+                start = i;
+            }
+            None => i += 1,
+        }
+    }
+    hasher.update(&bytes[start..]);
+    hex(&hasher.finalize())
+}
+
+/// Length of an RFC 3339 UTC timestamp at the start of `b`, if one is there.
+fn timestamp_len(b: &[u8]) -> Option<usize> {
+    const SHAPE: &[u8; 19] = b"dddd-dd-ddTdd:dd:dd";
+    if b.len() < SHAPE.len() + 1 {
+        return None;
+    }
+    let shaped = SHAPE.iter().zip(b).all(|(want, got)| match want {
+        b'd' => got.is_ascii_digit(),
+        _ => want == got,
+    });
+    if !shaped {
+        return None;
+    }
+    let mut len = SHAPE.len();
+    if b[len] == b'.' {
+        let digits = b[len + 1..]
+            .iter()
+            .take_while(|c| c.is_ascii_digit())
+            .count();
+        if digits == 0 {
+            return None;
+        }
+        len += 1 + digits;
+    }
+    (b.get(len) == Some(&b'Z')).then_some(len + 1)
+}
+
+#[async_trait]
+impl Embedder for EvalEmbeddingCache {
+    fn provider(&self) -> &'static str {
+        self.inner.provider()
+    }
+
+    fn model(&self) -> &str {
+        self.inner.model()
+    }
+
+    fn model_identity(&self) -> String {
+        self.inner.model_identity()
+    }
+
+    fn dim(&self) -> u32 {
+        self.inner.dim()
+    }
+
+    async fn embed(&self, text: &str) -> LlmResult<Vec<f32>> {
+        self.inner.embed(text).await
+    }
+
+    async fn embed_document(&self, text: &str) -> LlmResult<Vec<f32>> {
+        let path = self.entry_path(text);
+        if let Some(vector) = self.load(&path).await {
+            return Ok(vector);
+        }
+        let vector = self.inner.embed_document(text).await?;
+        self.store(&path, &vector).await;
+        Ok(vector)
+    }
+
+    async fn embed_query(&self, text: &str) -> LlmResult<Vec<f32>> {
+        self.inner.embed_query(text).await
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::sync::atomic::AtomicUsize;
+
+    use super::*;
+
+    /// Counts calls per method; vectors encode the text length so a wrong
+    /// entry would be visible.
+    struct Counting {
+        model: &'static str,
+        dim: u32,
+        documents: AtomicUsize,
+        queries: AtomicUsize,
+    }
+
+    impl Counting {
+        fn new(model: &'static str, dim: u32) -> Arc<Self> {
+            Arc::new(Self {
+                model,
+                dim,
+                documents: AtomicUsize::new(0),
+                queries: AtomicUsize::new(0),
+            })
+        }
+
+        fn vector(&self, text: &str) -> Vec<f32> {
+            (0..self.dim)
+                .map(|i| text.len() as f32 + i as f32)
+                .collect()
+        }
+    }
+
+    #[async_trait]
+    impl Embedder for Counting {
+        fn provider(&self) -> &'static str {
+            "counting"
+        }
+        fn model(&self) -> &str {
+            self.model
+        }
+        fn dim(&self) -> u32 {
+            self.dim
+        }
+        async fn embed(&self, text: &str) -> LlmResult<Vec<f32>> {
+            Ok(self.vector(text))
+        }
+        async fn embed_document(&self, text: &str) -> LlmResult<Vec<f32>> {
+            self.documents.fetch_add(1, Ordering::SeqCst);
+            Ok(self.vector(text))
+        }
+        async fn embed_query(&self, text: &str) -> LlmResult<Vec<f32>> {
+            self.queries.fetch_add(1, Ordering::SeqCst);
+            Ok(self.vector(text))
+        }
+    }
+
+    #[tokio::test]
+    async fn second_run_serves_documents_from_disk_and_keeps_queries_live() {
+        let root = tempfile::tempdir().unwrap();
+        let first_inner = Counting::new("m", 4);
+        let first = EvalEmbeddingCache::new(first_inner.clone(), root.path());
+        let stored = first.embed_document("haystack session").await.unwrap();
+
+        // A fresh wrapper stands in for the next benchmark server process.
+        let second_inner = Counting::new("m", 4);
+        let second = EvalEmbeddingCache::new(second_inner.clone(), root.path());
+        assert_eq!(
+            second.embed_document("haystack session").await.unwrap(),
+            stored
+        );
+        assert_eq!(second_inner.documents.load(Ordering::SeqCst), 0);
+
+        // A different text is a miss, and queries are never cached.
+        second.embed_document("another session").await.unwrap();
+        second.embed_query("haystack session").await.unwrap();
+        second.embed_query("haystack session").await.unwrap();
+        assert_eq!(second_inner.documents.load(Ordering::SeqCst), 1);
+        assert_eq!(second_inner.queries.load(Ordering::SeqCst), 2);
+        assert_eq!(second.model_identity(), "m");
+        assert_eq!((second.provider(), second.dim()), ("counting", 4));
+    }
+
+    #[test]
+    fn replay_wall_clock_times_do_not_change_the_key() {
+        let page = |t: &str| {
+            format!("- **started_at:** {t}\n- `stop` @ {t} — stop\n[session date: 2023/07/12]")
+        };
+        assert_eq!(
+            text_key(&page("2026-11-04T17:26:19Z")),
+            text_key(&page("2026-11-05T09:01:02.123456Z"))
+        );
+        // Any other text change, including the dataset's own dates, misses.
+        assert_ne!(
+            text_key(&page("2026-11-04T17:26:19Z")),
+            text_key(&page("2026-11-04T17:26:19Z").replace("2023/07/12", "2023/07/13"))
+        );
+        // Near-misses of the timestamp shape stay part of the key.
+        for (a, b) in [
+            ("2026-11-04T17:26:19", "2026-11-05T17:26:19"),
+            ("2026-11-04 17:26:19Z", "2026-11-05 17:26:19Z"),
+            ("2026-11-04T17:26:19.Z", "2026-11-05T17:26:19.Z"),
+        ] {
+            assert_ne!(text_key(a), text_key(b), "{a}");
+        }
+        assert_ne!(text_key("a"), text_key("b"));
+    }
+
+    #[tokio::test]
+    async fn model_or_dimension_change_misses_the_cache() {
+        let root = tempfile::tempdir().unwrap();
+        let original = EvalEmbeddingCache::new(Counting::new("m", 4), root.path());
+        original.embed_document("text").await.unwrap();
+
+        for inner in [Counting::new("other-model", 4), Counting::new("m", 8)] {
+            let cache = EvalEmbeddingCache::new(inner.clone(), root.path());
+            let vector = cache.embed_document("text").await.unwrap();
+            assert_eq!(vector.len(), inner.dim as usize);
+            assert_eq!(inner.documents.load(Ordering::SeqCst), 1);
+        }
+    }
+
+    #[tokio::test]
+    async fn a_truncated_entry_is_recomputed_and_rewritten() {
+        let root = tempfile::tempdir().unwrap();
+        let inner = Counting::new("m", 4);
+        let cache = EvalEmbeddingCache::new(inner.clone(), root.path());
+        let path = cache.entry_path("text");
+        std::fs::create_dir_all(path.parent().unwrap()).unwrap();
+        std::fs::write(&path, [0u8; 6]).unwrap();
+
+        let vector = cache.embed_document("text").await.unwrap();
+        assert_eq!(vector, inner.vector("text"));
+        assert_eq!(inner.documents.load(Ordering::SeqCst), 1);
+        assert_eq!(std::fs::read(&path).unwrap().len(), 16);
+    }
+}

crates/ai-memory-cli/src/commands/mod.rs

@@ -50,6 +50,7 @@ pub mod data_purge;
 pub mod delete_page;
 pub mod doctor;
 pub mod embed;
+mod eval_embedding_cache;
 pub mod export_okf;
 pub mod finalize_session;
 pub mod forget_sweep;

crates/ai-memory-cli/src/commands/serve.rs

@@ -49,6 +49,7 @@ use secrecy::ExposeSecret;
 use tokio_util::sync::CancellationToken;
 use tracing::info;
 
+use super::eval_embedding_cache::{EvalEmbeddingCache, eval_embedding_cache_dir};
 use crate::cli::{ServeArgs, TransportKind};
 use crate::config::{AuthSettings, AutoImproveSettings, Config, MaintenanceSettings};
 use ai_memory_mcp::auth::{AuthState, require_bearer};
@@ -3160,6 +3161,17 @@ async fn configure_embedder(
     } else {
         embedder
     };
+    // Outermost, so a cached document vector skips the capacity queue too.
+    let embedder: Arc<dyn Embedder> = match eval_embedding_cache_dir() {
+        Some(root) => {
+            info!(
+                root = %root.display(),
+                "benchmark document-embedding cache enabled (eval harness only)"
+            );
+            Arc::new(EvalEmbeddingCache::new(embedder, &root))
+        }
+        None => embedder,
+    };
     Ok((wiki.with_embedder(embedder.clone()), Some(embedder)))
 }
 

docs/benchmarks/README.md

@@ -267,6 +267,60 @@ ai-memory-eval retrieval \
   --server-env AI_MEMORY_EVAL_SKIP_WIKI_CHECKPOINTS=1
 ```
 
+### Reuse haystack embeddings across runs (local mode)
+
+Local-embeddings ingest is nearly all CPU time spent embedding haystack
+pages, and the haystack is the same on every run. The harness therefore
+keeps each document vector it computes in `evals/.cache/embeddings/`
+(gitignored) and hands the stored vector back on the next run. The key is
+the embedder's provider, model identity and dimension plus the sha256 of
+the text embedded, so a different model or dimension never reads another's
+vectors. The hash skips RFC 3339 UTC timestamps (`2026-11-04T17:26:19Z`).
+The server stamps each replayed hook event with its own clock and the session
+page lists those times, so without the mask no page text repeats between runs:
+an unmasked key stored a new vector for every page on the second run. A cached run
+therefore ranks pages with the vector the first run computed, for a text that
+differed only in those wall-clock times. Any other change to a page's text,
+the dataset's own dates included, is a miss and embeds live. Only document
+embeddings are cached: query embeddings, ranking and fusion run live, so a
+cached run still measures the code under test.
+
+The harness turns the cache on for every local-embeddings server it starts,
+through the eval-only server variable `AI_MEMORY_EVAL_EMBEDDING_CACHE_DIR`.
+Like the checkpoint variable above, no product path sets it, the server reads
+it once at startup, and it is absent from `config.toml`. Zero-LLM runs have no
+embedder and never use it. A report whose document vectors came from the cache
+says so on a `document embeddings: eval cache` line and in `embedding_cache`
+in `report.json`.
+
+Pass `--no-embedding-cache` for a fully cold run, and delete
+`evals/.cache/embeddings/` (or run cold) after a change to the embedding code
+itself: the key names the model, not the code that turns text into a vector.
+
+Measured on `--stratify 40 --embeddings local` (230 questions, checkpoints
+skipped), on the 32-thread benchmark box while other builds kept its load
+average between 10 and 90, so read the wall clock as indicative:
+
+| run | wall clock | ingest p50 per question | query p50 / p95 | vectors embedded |
+|---|---:|---:|---:|---:|
+| cold, filling the cache | 1,015 s | 22.5 s | 123 / 264 ms | 11,012 |
+| cached | 132 s | 3.2 s | 70 / 116 ms | 0 |
+| cached again | 135 s | 3.2 s | 66 / 124 ms | 0 |
+| cold, `--no-embedding-cache` | 963 s | 21.2 s | 113 / 294 ms | 11,012 |
+
+Both cached runs matched the run that filled the cache on every question's
+`hit@k`, `recall@k`, `recall-all@k` and retrieved sessions. Only context
+tokens moved, by at most 3 tokens on 3 questions, because snippets carry the
+replay's wall-clock times. Three cold runs of the same tree differed pairwise
+on 10 to 12 questions' `hit@k` and 19 to 23 questions' `recall@k`, while
+overall `hit@5` stayed within 0.935-0.943. That spread is the local-mode
+variance tracked in #1854. A cached run reproduces the one cold run that
+filled the cache, so two builds compared on the same cache differ only where
+the builds do. Query latency also drops in a cached run, because queries no
+longer compete with embedding work, so a cached run's latency and ingest
+numbers are not comparable with a cold run's. Published release numbers come
+from a cold run (AGENTS.md, project maintenance rules).
+
 ### Run the two modes back to back, not at the same time
 
 Two harness processes are fully independent (separate server, temp data

evals/README.md

@@ -360,6 +360,12 @@ Output: a per-slice table on stdout plus
 provenance (commit, dataset sha, hardware, per-question scores).
 When publishing a baseline, copy the markdown into `docs/benchmarks/`.
 
+`--embeddings local` runs keep haystack document vectors in
+`evals/.cache/embeddings/` (gitignored) and reuse them on the next run, so a
+repeat run skips most of its embedding work; query embeddings stay live.
+`--no-embedding-cache` runs cold. See
+[Reuse haystack embeddings across runs](../docs/benchmarks/README.md#reuse-haystack-embeddings-across-runs-local-mode).
+
 ### Bundled coding-session retrieval corpus
 
 `--dataset-variant coding-v1` selects four synthetic questions in

evals/src/retrieval/mod.rs

@@ -124,6 +124,12 @@ pub struct RetrievalArgs {
     #[arg(long)]
     keep_data_dir: bool,
 
+    /// Embed every haystack document live instead of reusing vectors cached
+    /// under `evals/.cache/embeddings/` by earlier local-embeddings runs.
+    /// Query embeddings are never cached either way.
+    #[arg(long)]
+    no_embedding_cache: bool,
+
     // ---- baseline config knobs ----
     /// Baseline embedding mode: `none` (zero-LLM, vectors off, the
     /// default) or `local` (in-process all-MiniLM-L6-v2, vectors on; the
@@ -238,6 +244,10 @@ enum Engine {
     Grep,
 }
 
+/// Where local-embeddings runs keep haystack document vectors between runs,
+/// keyed by provider, model, dimension and text hash (gitignored).
+const EMBEDDING_CACHE_DIR: &str = "evals/.cache/embeddings";
+
 /// One resolved server + query configuration.
 struct Knobs {
     /// Short label for reports (`baseline` / `candidate`).
@@ -246,6 +256,8 @@ struct Knobs {
     /// `zero-llm` or `local-embeddings`, for report provenance.
     mode: &'static str,
     models_root: Option<PathBuf>,
+    /// Document-embedding cache root handed to the server (local mode only).
+    embedding_cache: Option<PathBuf>,
     reranker: bool,
     env: Vec<(String, String)>,
     query_args: serde_json::Map<String, serde_json::Value>,
@@ -302,6 +314,7 @@ impl Knobs {
             embeddings: embeddings_kind,
             mode,
             models_root,
+            embedding_cache: None,
             reranker,
             env,
             query_args,
@@ -311,10 +324,19 @@ impl Knobs {
         })
     }
 
+    /// Point a local-embeddings server at the document-vector cache; a
+    /// zero-LLM server has no embedder, so it never gets one.
+    fn use_embedding_cache(&mut self, root: &std::path::Path) {
+        if self.embeddings == EvalEmbeddings::Local {
+            self.embedding_cache = Some(root.to_path_buf());
+        }
+    }
+
     fn launch_config(&self) -> LaunchConfig<'_> {
         LaunchConfig {
             embeddings: self.embeddings,
             models_root: self.models_root.as_deref(),
+            embedding_cache: self.embedding_cache.as_deref(),
             reranker: self.reranker,
             env: &self.env,
         }
@@ -437,7 +459,7 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {
         );
     }
 
-    let baseline = Knobs::parse(
+    let mut baseline = Knobs::parse(
         "baseline",
         &args.embeddings,
         args.reranker,
@@ -446,7 +468,7 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {
         args.ingest,
         args.date_prefix,
     )?;
-    let candidate = if args.ab_mode() {
+    let mut candidate = if args.ab_mode() {
         let embeddings = args
             .candidate_embeddings
             .clone()
@@ -464,6 +486,14 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {
         None
     };
 
+    if !args.no_embedding_cache {
+        let root = std::path::absolute(EMBEDDING_CACHE_DIR)
+            .context("resolving the embedding cache directory")?;
+        for knobs in std::iter::once(&mut baseline).chain(candidate.as_mut()) {
+            knobs.use_embedding_cache(&root);
+        }
+    }
+
     // Fetch the local embedding model once if any config needs it.
     let needs_local = baseline.embeddings == EvalEmbeddings::Local
         || candidate
@@ -735,6 +765,7 @@ async fn run_config(run: &RunContext<'_>, knobs: &Knobs) -> Result<report::Repor
         dataset_sha256: run.variant.sha256,
         mode: knobs.mode,
         config: knobs.summary.clone(),
+        embedding_cache: knobs.embedding_cache.is_some(),
         questions_scored: scores.len(),
         abstention_excluded: run.abstention_n,
         ks: args.ks.clone(),
@@ -800,6 +831,7 @@ fn run_grep(
         dataset_sha256: variant.sha256,
         mode: "grep",
         config: "engine=grep (whole raw session, term overlap, no server)".into(),
+        embedding_cache: false,
         questions_scored: scores.len(),
         abstention_excluded: abstention_n,
         ks: args.ks.clone(),
@@ -887,4 +919,31 @@ mod tests {
         assert!(k.summary.contains("ingest=whole-session"));
         assert!(k.summary.contains("date_prefix=session-start"));
     }
+
+    #[test]
+    fn embedding_cache_reaches_only_local_embedding_servers() {
+        let root = std::path::Path::new("/cache");
+        let parse = |embeddings| {
+            Knobs::parse(
+                "baseline",
+                embeddings,
+                false,
+                &[],
+                &[],
+                IngestMode::Hooks,
+                DatePrefix::Turn,
+            )
+            .unwrap()
+        };
+        let mut local = parse("local");
+        assert_eq!(local.launch_config().embedding_cache, None);
+        local.use_embedding_cache(root);
+        assert_eq!(local.launch_config().embedding_cache, Some(root));
+        // The published config line stays comparable with earlier runs.
+        assert_eq!(local.summary, "embeddings=local reranker=off");
+
+        let mut fts = parse("none");
+        fts.use_embedding_cache(root);
+        assert_eq!(fts.launch_config().embedding_cache, None);
+    }
 }

evals/src/retrieval/report.rs

@@ -28,6 +28,10 @@ pub struct Report {
     /// Human-readable knob summary for this config (e.g.
     /// `embeddings=local reranker=off`).
     pub config: String,
+    /// Haystack document vectors came from the eval embedding cache when
+    /// present (query embeddings are always live).
+    #[serde(skip_serializing_if = "std::ops::Not::not")]
+    pub embedding_cache: bool,
     pub questions_scored: usize,
     pub abstention_excluded: usize,
     pub ks: Vec<usize>,
@@ -291,7 +295,11 @@ pub fn to_markdown(r: &Report) -> String {
         r.dataset, r.generated_at
     ));
     md.push_str(&provenance_md(r));
-    md.push_str(&format!("- mode: {}\n- config: {}\n\n", r.mode, r.config));
+    md.push_str(&format!("- mode: {}\n- config: {}\n", r.mode, r.config));
+    if r.embedding_cache {
+        md.push_str("- document embeddings: eval cache (`evals/.cache/embeddings`)\n");
+    }
+    md.push('\n');
     md.push_str(&slices_table_md(r));
     md.push_str(&qa_table_md(r));
     md.push_str(&extras_md(r));
@@ -498,6 +506,7 @@ mod tests {
             dataset_sha256: crate::retrieval::dataset::LONGMEMEVAL_S_SHA256,
             mode: "zero-llm",
             config: config.into(),
+            embedding_cache: false,
             questions_scored: 2,
             abstention_excluded: 1,
             ks: vec![5],

evals/src/retrieval/server.rs

@@ -49,6 +49,9 @@ pub struct LaunchConfig<'a> {
     pub embeddings: EvalEmbeddings,
     /// When `embeddings == Local`, the models root to seed the server from.
     pub models_root: Option<&'a std::path::Path>,
+    /// Document-embedding cache root (`AI_MEMORY_EVAL_EMBEDDING_CACHE_DIR`),
+    /// `None` for a cold run.
+    pub embedding_cache: Option<&'a std::path::Path>,
     /// Turn the post-RRF reranker on (`AI_MEMORY_RERANKER=llm`). Requires
     /// an LLM provider to be supplied through `env` — the harness does not
     /// configure one, so a bare `reranker=true` with no provider is a no-op
@@ -126,6 +129,12 @@ impl EvalServer {
                 cmd.env("AI_MEMORY_EMBEDDING_PROVIDER", "local");
             }
         }
+        // Eval-only server hook: reuse haystack document vectors across runs.
+        // An inherited value would silently warm a run meant to be cold.
+        match config.embedding_cache {
+            Some(root) => cmd.env("AI_MEMORY_EVAL_EMBEDDING_CACHE_DIR", root),
+            None => cmd.env_remove("AI_MEMORY_EVAL_EMBEDDING_CACHE_DIR"),
+        };
         if config.reranker {
             // Only `llm` is a supported reranker; it needs a provider,
             // which the config must supply via `env`.