修复长期记忆向量链路:投影入队 + 集合名三侧同源 + 召回按客户过滤
语义召回"恒空"的根因分四层,本提交修掉投递层与读取层(另两层——重试计数
门禁、episode 不投 rebuild 事件——已在前两个提交修复)。
R2 投递层:dispatch_profile_rebuild 只做图投影,没有任何 Milvus 投递,
长期记忆的向量从未被写入过(实测客户 9001 在 memory_sync_outbox 里 0 行)。
新增 _enqueue_memory_vector_projection(),在画像重建后投
MemorySyncOutbox(target_store="milvus")。走事件而不是同步写,是为了拿
Outbox 的重试/退避/死信,且不把 embedding 的网络等待拖进事务。
⚠️ payload 必须带 version(正整数):适配器 _coerce_profile_version 缺它
直接抛 ValueError(实测踩到,事件立刻 failed)。
R4 读取层:写的集合与读的集合不是同一个 ——
写(MilvusProfileProjection)用 "user_long_term_memory_v1",
读(bootstrap.get_vector_memory_adapter)与删(projection_cleanup_service)
却用 settings.milvus_collection = "jr_memory",而该集合从未被创建。
⇒ 召回:MilvusClient 构造不校验集合存在,适配器"构造成功"但每次 search
抛异常被 VectorMemoryAdapter 吞成 degraded → 召回恒
degraded_reasons=('milvus_unavailable',)、向量命中恒 0 条;
⇒ 清理:jr_memory 不在集合列表 → 走 vector_collection_absent 分支 →
报清理成功但一个向量都没删,陈旧向量永久留存。
修法:PROFILE_COLLECTION 成为唯一常量,读/删两侧直接引用它;
并删除 Settings.milvus_collection 配置项、清掉 .env.example 的
MILVUS_COLLECTION —— 写侧从来没读过它,一个只在契约一侧生效的配置项
比没有配置项更危险(Settings 的 extra="ignore" 会让其他环境残留的该
变量被安全忽略)。
顺带:
- 语义检索把客户过滤下推到 Milvus(filter="customer_id == N")。此前不带
过滤,别家客户的命中会白占 limit 名额,稀释本客户的召回条数。
- upsert 在 sources 为空时先返回,不再无条件 load_collection ——
"本来就没有可写内容"不该被记成投递失败(10001/10002 那两条事件即如此
重试 5 次进死信)。
回归守卫:tests/unit/infrastructure/test_memory_vector_collection_consistency.py
断言读侧与删侧用的都是 PROFILE_COLLECTION,且被删掉的配置项不得回归。
这个缺陷能活下来,正是因为两侧单测全绿而接缝无人守。
验证(走生产装配、进程内调用,未重启你正在跑的 API 窗口):
读侧集合打印 user_long_term_memory_v1(修前为 jr_memory);
召回 degraded=False / reasons=() / sources 含 milvus,且排序随 query 语义
变化(投资期限→horizon 0.288 > risk_level 0.201;风险偏好→risk_level 0.287);
query="进取型" 双通道合并且 vector_score=0.9987;query=None 走 mysql 全量。
pytest tests/unit tests/contract → 1432 passed, 2 skipped, 1 failed
(唯一失败是同事正在改的投顾页面,与记忆链路无关)。
文档:docs/演示用/记忆召回恒空-根因与修复-2026-09-14.md(新增,四层根因+证据)、
docs/37-记忆投影链路实现说明.md(补集合名三侧契约)。
This commit is contained in:
@@ -0,0 +1,122 @@
|
||||
"""长期记忆向量集合名必须"写、读、删"三侧同源。
|
||||
|
||||
2026-09-14 实测故障(三侧各写各的,接缝无人守):
|
||||
|
||||
- **写**(`MilvusProfileProjection`)用模块常量 `PROFILE_COLLECTION`
|
||||
= `user_long_term_memory_v1`;
|
||||
- **读**(`bootstrap.get_vector_memory_adapter`)用 `settings.milvus_collection`
|
||||
= `jr_memory` —— **该集合在本仓从未被创建**;
|
||||
- **删**(`ProjectionCleanupService._cleanup_vector`)同读侧。
|
||||
|
||||
后果是两条静默故障:语义召回每次 `search` 抛异常 → 降级 `milvus_unavailable`
|
||||
→ 召回恒空;记忆删除读到集合不存在 → 走 `vector_collection_absent` 分支 →
|
||||
**报清理成功但一个向量都没删**。而三侧各自的单测全绿。
|
||||
|
||||
本文件守的就是这条接缝:任何一侧换集合名,这里必须红。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from app.core.config import Settings
|
||||
from app.infrastructure.milvus_profile_projection import PROFILE_COLLECTION
|
||||
from app.service import projection_cleanup_service
|
||||
from app.service.agent import bootstrap
|
||||
from app.service.projection_cleanup_service import ProjectionCleanupService
|
||||
|
||||
MEMORY_UUID = "187e3b7c-0000-4000-8000-000000000001"
|
||||
|
||||
|
||||
class FakeMilvusClient:
|
||||
"""记录构造参数与删除调用的 Milvus 客户端替身;不发网络请求。"""
|
||||
|
||||
instances: list[FakeMilvusClient] = []
|
||||
collections: list[str] = []
|
||||
|
||||
def __init__(self, *, uri: str, token: str | None = None) -> None:
|
||||
self.uri = uri
|
||||
self.token = token
|
||||
self.deletes: list[dict[str, Any]] = []
|
||||
FakeMilvusClient.instances.append(self)
|
||||
|
||||
def list_collections(self) -> list[str]:
|
||||
return list(FakeMilvusClient.collections)
|
||||
|
||||
def delete(self, *, collection_name: str, filter: str) -> None: # noqa: A002
|
||||
self.deletes.append({"collection_name": collection_name, "filter": filter})
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_fake_client() -> None:
|
||||
FakeMilvusClient.instances = []
|
||||
FakeMilvusClient.collections = [PROFILE_COLLECTION]
|
||||
|
||||
|
||||
class _FakeSettings:
|
||||
milvus_uri = "http://milvus:19530"
|
||||
milvus_token = ""
|
||||
|
||||
|
||||
def test_retired_collection_setting_is_not_reintroduced() -> None:
|
||||
"""`milvus_collection` 配置项必须保持删除状态。
|
||||
|
||||
它只在契约的读/删两侧生效、写侧从不读它 —— 一个"看着权威但只有一半生效"的
|
||||
配置项,正是本次故障的成因。重新加回来会让同一个坑再出现一次。
|
||||
"""
|
||||
assert "milvus_collection" not in Settings.model_fields
|
||||
|
||||
|
||||
def test_recall_adapter_reads_the_projection_collection(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""读侧集合名 == 写侧常量,而不是任意配置项。"""
|
||||
monkeypatch.setattr(bootstrap, "get_settings", lambda: _FakeSettings())
|
||||
monkeypatch.setattr("pymilvus.MilvusClient", FakeMilvusClient)
|
||||
bootstrap.get_vector_memory_adapter.cache_clear()
|
||||
try:
|
||||
adapter = bootstrap.get_vector_memory_adapter()
|
||||
finally:
|
||||
# 必须清缓存:否则被 monkeypatch 污染的实例会泄漏给同会话后续测试。
|
||||
bootstrap.get_vector_memory_adapter.cache_clear()
|
||||
|
||||
assert adapter is not None
|
||||
assert adapter.collection == PROFILE_COLLECTION
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cleanup_deletes_from_the_projection_collection(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""删侧必须真的去同一集合删,而不是因为查了另一个集合名就"报成功"。"""
|
||||
monkeypatch.setattr(projection_cleanup_service, "get_settings", lambda: _FakeSettings())
|
||||
monkeypatch.setattr("pymilvus.MilvusClient", FakeMilvusClient)
|
||||
|
||||
details: list[str] = []
|
||||
cleaned = await ProjectionCleanupService()._cleanup_vector(MEMORY_UUID, details)
|
||||
|
||||
assert cleaned is True
|
||||
assert details == ["vector_cleaned"]
|
||||
assert FakeMilvusClient.instances[-1].deletes == [{
|
||||
"collection_name": PROFILE_COLLECTION,
|
||||
"filter": f'memory_uuid == "{MEMORY_UUID}"',
|
||||
}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cleanup_reports_honestly_when_collection_is_absent(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""集合确实不存在时如实报 `vector_collection_absent`(合法降级,不是伪造成功)。"""
|
||||
FakeMilvusClient.collections = []
|
||||
monkeypatch.setattr(projection_cleanup_service, "get_settings", lambda: _FakeSettings())
|
||||
monkeypatch.setattr("pymilvus.MilvusClient", FakeMilvusClient)
|
||||
|
||||
details: list[str] = []
|
||||
cleaned = await ProjectionCleanupService()._cleanup_vector(MEMORY_UUID, details)
|
||||
|
||||
assert cleaned is True
|
||||
assert details == ["vector_collection_absent"]
|
||||
assert FakeMilvusClient.instances[-1].deletes == []
|
||||
@@ -31,15 +31,24 @@ QUERY = "基金"
|
||||
|
||||
|
||||
class StubVectorClient:
|
||||
"""Milvus 客户端替身:返回嵌套命中或抛连接错误,不发网络请求。"""
|
||||
"""Milvus 客户端替身:返回嵌套命中或抛连接错误,不发网络请求。
|
||||
|
||||
记录 `filter` 是为了守住"语义检索必须按客户下推过滤"这条约束——不过滤的话
|
||||
别家客户的命中会白占 `limit` 名额,本客户的召回条数被稀释。
|
||||
"""
|
||||
|
||||
def __init__(self, rows: list[Any] | None = None, error: Exception | None = None) -> None:
|
||||
self.rows: list[Any] = list(rows or [])
|
||||
self.error = error
|
||||
self.calls: list[int] = []
|
||||
self.filters: list[str | None] = []
|
||||
|
||||
def search(self, collection_name: str, data: list[list[float]], limit: int) -> Any:
|
||||
def search(
|
||||
self, collection_name: str, data: list[list[float]], limit: int,
|
||||
filter: str | None = None,
|
||||
) -> Any:
|
||||
self.calls.append(limit)
|
||||
self.filters.append(filter)
|
||||
if self.error is not None:
|
||||
raise self.error
|
||||
return [self.rows]
|
||||
@@ -193,6 +202,8 @@ async def test_recall_merges_structured_and_vector_channels() -> None:
|
||||
assert vector.confidence == pytest.approx(0.63)
|
||||
assert vector.evidence["vector_score"] == pytest.approx(0.9)
|
||||
assert client.calls == [10]
|
||||
# 语义检索必须把客户过滤下推到向量库,否则别家客户的命中会白占 limit 名额。
|
||||
assert client.filters == [f"customer_id == {CUSTOMER_ID}"]
|
||||
|
||||
|
||||
async def test_vector_hit_content_is_reloaded_from_mysql_authority() -> None:
|
||||
|
||||
Reference in New Issue
Block a user