知识库三项收口:向量-元数据对账 + 导入侧幂等 + 过期行向量清理入口
① 只读对账 tools/reconcile_knowledge_vectors.py
按集合列出:孤儿向量 / 死向量 / 缺向量 / 重复正文 / 低信息量碎片 / 纯标题。
关键口径:非数字 id(FAQ-0013 这类语义 id)是灌库脚本有意写进 Milvus 的,
单独归类、不建议删;向量数取自 query 实际行数,不用 get_collection_stats
(后者含已软删未 compaction 的行)。
② 导入侧幂等:同 source_file + 集合重传 = 覆盖上一版
app/service/knowledge_ingest_service.py 新增 _supersede_previous_version:
把上一版 active 行置为 expired,并逐行投 knowledge.vector_delete_requested
(与本次入库同事务)。写入侧只认 active 而检索侧不看 status,旧向量不清掉
会继续参与排序、和同题活块抢答。
顺带修掉一个真 bug:改为先判 chunks 非空再下线 —— 否则传一份解析出 0 块的
文档会把上一版下架、新版一行没写,这份文档在检索侧凭空消失。
③ 清理入口:POST /api/v1/knowledge/{knowledge_id}/vector-cleanups
给历史上"被别的途径置为 expired、从未投过删除事件"的行补投向量清理。
DELETE 对已过期行返回 404 的口径保持不变(重复删除静默成功会让调用方
分不清"这次真下线了"和"早就过期了"),因此新开一个语义明确的端点:
不存在 404 / 仍是 active 422(请改用 DELETE)/ 已 expired 200 并回传事件名。
配套 tools/purge_expired_knowledge_vectors.py(默认 dry-run)批量驱动该端点。
文档:docs/演示用/知识库向量对账与清理-2026-09-15.md(含真机验证输出),
并对 docs/演示用/知识库问答诊断-2026-09-14.md 做两处更正 —— 实测孤儿向量 0 条、
那 175 行历史副本从来没有向量(不参与排序),当时的差额来自 get_collection_stats
把已软删行算进去。
新发现(未修,需业务拍板):661 条向量里 451 条正文不到 40 字,是灌库时把
markdown 表格/标题切碎产生的碎片。「风险评估问卷怎么评分」实测前 4 名是 4 条
一模一样的 19 字碎片(gap 0.0024),真正 2828 字的答案排第 5 → 客服必然转人工。
属灌库切分缺陷,补内容救不了,也不应靠放宽 MIN_GAP 解决。
验证:pytest tests/unit tests/contract → 1500 passed, 2 skipped, 0 failed;
mypy app → 3 个错全在组员文件中(与本次改动无关);ruff 本次改动文件 0 错。
真机端到端:重传 → 旧行 expired + 删除事件 published + 旧向量已从 Milvus 删除;
两个问句回归仍正常回答(r1到r5 gap 0.0766;申购确认 0.8453)。
This commit is contained in:
@@ -36,6 +36,7 @@ PROTECTED_POST = [
|
||||
"/api/v1/agent-runs/run-x/cancellations",
|
||||
"/api/v1/conversation-messages/1/feedback",
|
||||
"/api/v1/knowledge/upload",
|
||||
"/api/v1/knowledge/1/vector-cleanups",
|
||||
]
|
||||
|
||||
PROTECTED_DELETE = [
|
||||
|
||||
@@ -43,17 +43,29 @@ class FakeParser:
|
||||
|
||||
|
||||
class FakeSession:
|
||||
"""记录每条带参数的语句;`SELECT LAST_INSERT_ID()` 按预设序列返回。"""
|
||||
"""记录每条带参数的语句;`SELECT LAST_INSERT_ID()` 按预设序列返回。
|
||||
|
||||
def __init__(self, ids: list[Any] | None = None) -> None:
|
||||
`previous_ids` 是"同一 source_file + 集合上已经 active 的行"(导入侧幂等的替身输入):
|
||||
`_supersede_previous_version` 用 `scalars()` 读它们,本替身直接把它当查询结果返回。
|
||||
"""
|
||||
|
||||
def __init__(self, ids: list[Any] | None = None, previous_ids: list[int] | None = None) -> None:
|
||||
self._ids = list(ids if ids is not None else [])
|
||||
self.previous_ids = list(previous_ids if previous_ids is not None else [])
|
||||
self.statements: list[tuple[str, dict[str, Any]]] = []
|
||||
self.commits = 0
|
||||
|
||||
async def execute(self, statement: Any, params: Any = None) -> None:
|
||||
self.statements.append((str(statement), dict(params or {})))
|
||||
# SQLAlchemy 构造式语句(UPDATE ... WHERE id IN (...))的绑定值不在 `params` 里,
|
||||
# 而是编译进语句本身;不取出来就看不见"到底把哪些行改成了什么状态"。
|
||||
values = dict(params or {}) or _compiled_params(statement)
|
||||
self.statements.append((str(statement), values))
|
||||
return None
|
||||
|
||||
async def scalars(self, statement: Any, params: Any = None) -> Any:
|
||||
self.statements.append((str(statement), dict(params or {}) or _compiled_params(statement)))
|
||||
return list(self.previous_ids)
|
||||
|
||||
async def scalar(self, statement: Any, params: Any = None) -> Any:
|
||||
self.statements.append((str(statement), dict(params or {})))
|
||||
return self._ids.pop(0) if self._ids else None
|
||||
@@ -62,6 +74,14 @@ class FakeSession:
|
||||
self.commits += 1
|
||||
|
||||
|
||||
def _compiled_params(statement: Any) -> dict[str, Any]:
|
||||
"""取构造式语句编译后的绑定参数(纯 SQL 字符串没有可编译对象)。"""
|
||||
compile_ = getattr(statement, "compile", None)
|
||||
if compile_ is None:
|
||||
return {}
|
||||
return dict(compile_().params)
|
||||
|
||||
|
||||
class RecordingEmbedder:
|
||||
async def embed(self, endpoints: Any, text: str, *, max_attempts: int = 2) -> Any:
|
||||
raise AssertionError("入库服务不得自己调用 embedding(向量由 Outbox Worker 负责)")
|
||||
@@ -84,9 +104,10 @@ def _service(
|
||||
ids: list[Any] | None = None,
|
||||
chunks: list[ParsedChunk] | None = None,
|
||||
created_by: int | None = 9003,
|
||||
previous_ids: list[int] | None = None,
|
||||
) -> tuple[KnowledgeIngestService, FakeSession, FakeStorage, FakeParser]:
|
||||
"""默认给 2 个 chunk 配 2 个自增 id(`LAST_INSERT_ID()` 的替身)。"""
|
||||
session = FakeSession(ids if ids is not None else [11, 12])
|
||||
session = FakeSession(ids if ids is not None else [11, 12], previous_ids)
|
||||
storage = FakeStorage()
|
||||
parser = FakeParser(chunks if chunks is not None else _chunks())
|
||||
service = KnowledgeIngestService(
|
||||
@@ -290,10 +311,13 @@ async def test_last_insert_id_is_read_once_per_chunk_in_the_same_statement_order
|
||||
"INSERT INTO fin_knowledge_meta" if sql.startswith("INSERT INTO fin_knowledge_meta")
|
||||
else "SELECT LAST_INSERT_ID()" if "LAST_INSERT_ID" in sql
|
||||
else "INSERT INTO domain_event_outbox" if sql.startswith("INSERT INTO domain_event_outbox")
|
||||
else "SELECT previous active ids" if "fin_knowledge_meta.id" in sql
|
||||
else sql
|
||||
for sql, _ in session.statements
|
||||
]
|
||||
assert shape == [
|
||||
# 导入侧幂等的第一步:查同 source_file + 集合的 active 旧版(本次为空,故没有 UPDATE)。
|
||||
"SELECT previous active ids",
|
||||
"INSERT INTO fin_knowledge_meta",
|
||||
"SELECT LAST_INSERT_ID()",
|
||||
"INSERT INTO domain_event_outbox",
|
||||
@@ -301,3 +325,96 @@ async def test_last_insert_id_is_read_once_per_chunk_in_the_same_statement_order
|
||||
"SELECT LAST_INSERT_ID()",
|
||||
"INSERT INTO domain_event_outbox",
|
||||
]
|
||||
|
||||
|
||||
# --- 导入侧幂等:同一份文档重传 = 覆盖上一版 ---------------------------------------
|
||||
|
||||
|
||||
def _vector_events(session: FakeSession, event_type: str) -> list[dict[str, Any]]:
|
||||
return [
|
||||
params
|
||||
for _, params in session.statements
|
||||
if params.get("event_type") == event_type
|
||||
]
|
||||
|
||||
|
||||
async def test_reingesting_the_same_file_expires_the_previous_version() -> None:
|
||||
"""重传同名文档必须**先下线上一版并逐块投出向量删除事件**。
|
||||
|
||||
不这么做的话,旧版向量会留在 Milvus 里继续参与排序(检索侧不看 status),
|
||||
表现就是"同一份内容有 N 个副本互相抢答"、客服的"领先次优 ≥0.07"门槛被永远卡死。
|
||||
"""
|
||||
service, session, _, _ = _service(ids=[11, 12], previous_ids=[3, 4, 5])
|
||||
|
||||
await service.ingest(filename="faq.md", content=b"x", knowledge_type="faq")
|
||||
|
||||
updates = [
|
||||
(sql, params) for sql, params in session.statements if sql.startswith("UPDATE")
|
||||
]
|
||||
assert len(updates) == 1
|
||||
_, params = updates[0]
|
||||
assert params["status"] == "expired"
|
||||
# `id.in_([...])` 编译后是一个列表参数(参数名由 SQLAlchemy 生成,不写死名字)。
|
||||
id_lists = [value for value in params.values() if isinstance(value, list)]
|
||||
assert id_lists == [[3, 4, 5]]
|
||||
|
||||
deletes = _vector_events(session, "knowledge.vector_delete_requested")
|
||||
assert [event["aggregate_id"] for event in deletes] == ["3", "4", "5"]
|
||||
assert [json.loads(event["payload"]) for event in deletes] == [
|
||||
{"knowledge_id": "3"}, {"knowledge_id": "4"}, {"knowledge_id": "5"}
|
||||
]
|
||||
assert {event["aggregate_type"] for event in deletes} == {"knowledge_meta"}
|
||||
# 新版自己的同步事件不受影响。
|
||||
assert [event["aggregate_id"]
|
||||
for event in _vector_events(session, "knowledge.vector_sync_requested")] == ["11", "12"]
|
||||
|
||||
|
||||
async def test_supersede_happens_before_the_new_rows_are_written() -> None:
|
||||
"""顺序是硬约束:先下线旧版(含投删除事件)、再写新版。
|
||||
|
||||
反过来的话,"新行插入失败"会留下一份"旧版已下线、新版没写成"的文档(检索侧彻底查不到);
|
||||
代价不对称,所以顺序要在测试里固定下来。
|
||||
"""
|
||||
service, session, _, _ = _service(ids=[11, 12], previous_ids=[3])
|
||||
|
||||
await service.ingest(filename="faq.md", content=b"x", knowledge_type="faq")
|
||||
|
||||
def kind(index: int) -> str:
|
||||
sql = session.statements[index][0]
|
||||
if sql.startswith("UPDATE"):
|
||||
return "update"
|
||||
if "fin_knowledge_meta.id" in sql:
|
||||
return "select_previous"
|
||||
if sql.startswith("INSERT INTO fin_knowledge_meta"):
|
||||
return "insert_row"
|
||||
if sql.startswith("INSERT INTO domain_event_outbox"):
|
||||
return "insert_event"
|
||||
return "other"
|
||||
|
||||
kinds = [kind(index) for index in range(len(session.statements))]
|
||||
assert kinds[0] == "select_previous"
|
||||
assert kinds.index("update") < kinds.index("insert_row")
|
||||
# 旧版那条删除事件也在新版第一行之前投出。
|
||||
delete_index = next(
|
||||
index for index, (_, params) in enumerate(session.statements)
|
||||
if params.get("event_type") == "knowledge.vector_delete_requested"
|
||||
)
|
||||
assert delete_index < kinds.index("insert_row")
|
||||
|
||||
|
||||
async def test_first_import_enqueues_no_vector_delete() -> None:
|
||||
"""首次导入(没有旧版)不得投出任何删除事件——否则会把别人的向量删掉。"""
|
||||
service, session, _, _ = _service(ids=[11, 12], previous_ids=[])
|
||||
|
||||
await service.ingest(filename="faq.md", content=b"x", knowledge_type="faq")
|
||||
|
||||
assert _vector_events(session, "knowledge.vector_delete_requested") == []
|
||||
assert not [sql for sql, _ in session.statements if sql.startswith("UPDATE")]
|
||||
|
||||
|
||||
async def test_empty_document_must_not_expire_the_previous_version() -> None:
|
||||
"""空文档(解析出 0 块)不得触发下线:否则"旧版下架了、新版一行没写",文档凭空消失。"""
|
||||
service, session, _, _ = _service(ids=[], chunks=[], previous_ids=[3, 4])
|
||||
|
||||
assert await service.ingest(filename="empty.md", content=b"", knowledge_type="faq") == []
|
||||
assert session.statements == []
|
||||
|
||||
@@ -402,6 +402,59 @@ async def test_archive_key_comes_from_the_knowledge_row() -> None:
|
||||
assert storage.archived == ["kb/faq/deadbeef-faq.md"]
|
||||
|
||||
|
||||
# --- ④ 补投向量清理(expired 行) --------------------------------------------
|
||||
|
||||
|
||||
async def test_cleanup_vector_enqueues_delete_for_an_expired_row() -> None:
|
||||
"""已过期的历史行必须能补投删除事件——这正是 `DELETE` 覆盖不到的那批行。"""
|
||||
row = meta_row(7, status=EXPIRED_STATUS)
|
||||
service, session, _, _ = build_service(rows=[row])
|
||||
|
||||
result = await service.cleanup_vector(context(), 7)
|
||||
|
||||
assert row.status == EXPIRED_STATUS # 不改状态:本动作只补投事件
|
||||
events = delete_events(session)
|
||||
assert len(events) == 1
|
||||
assert events[0].aggregate_id == "7"
|
||||
assert events[0].payload == {"knowledge_id": "7"}
|
||||
assert events[0].trace_id == "trace-task11"
|
||||
# 事件 id 必须是新的 uuid4:与历史事件同 id 会被 outbox 的唯一键吞掉,等于没投。
|
||||
assert events[0].event_id
|
||||
assert result == {"knowledge_id": 7, "status": EXPIRED_STATUS,
|
||||
"vector_delete_event": "knowledge.vector_delete_requested"}
|
||||
|
||||
|
||||
async def test_cleanup_vector_refuses_an_active_row() -> None:
|
||||
"""在用文档不得被"只删向量不标状态":那会造成"行是 active、向量已没"的不一致。"""
|
||||
service, session, _, _ = build_service(rows=[meta_row(7, status="active")])
|
||||
|
||||
with pytest.raises(ValidationAgentError):
|
||||
await service.cleanup_vector(context(), 7)
|
||||
|
||||
assert delete_events(session) == []
|
||||
|
||||
|
||||
async def test_cleanup_vector_unknown_id_is_not_found() -> None:
|
||||
service, session, _, _ = build_service(rows=[meta_row(7, status=EXPIRED_STATUS)])
|
||||
|
||||
with pytest.raises(GenericResourceNotFoundError):
|
||||
await service.cleanup_vector(context(), 999)
|
||||
|
||||
assert delete_events(session) == []
|
||||
|
||||
|
||||
async def test_cleanup_vector_requires_the_management_permission() -> None:
|
||||
service, _, _, _ = build_service(rows=[meta_row(7, status=EXPIRED_STATUS)])
|
||||
|
||||
def explode() -> None:
|
||||
raise AssertionError("未授权请求不得打开数据库会话")
|
||||
|
||||
service._session_factory = explode # type: ignore[assignment]
|
||||
|
||||
with pytest.raises(ForbiddenAgentError):
|
||||
await service.cleanup_vector(context(permissions=()), 7)
|
||||
|
||||
|
||||
# --- 装配与路由 -------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -420,10 +473,12 @@ def test_router_registers_the_three_required_paths() -> None:
|
||||
assert ("/api/v1/knowledge/upload", "POST") in paths
|
||||
assert ("/api/v1/knowledge/list", "GET") in paths
|
||||
assert ("/api/v1/knowledge/{knowledge_id}", "DELETE") in paths
|
||||
# 运维补口:给已过期的历史行补投向量清理。
|
||||
assert ("/api/v1/knowledge/{knowledge_id}/vector-cleanups", "POST") in paths
|
||||
|
||||
|
||||
def test_every_endpoint_depends_on_the_authentication_gate() -> None:
|
||||
"""硬约束:三个端点都必须依赖 `build_request_context`,禁止匿名入口。"""
|
||||
"""硬约束:每个端点都必须依赖 `build_request_context`,禁止匿名入口。"""
|
||||
from app.api.controllers.knowledge_management import router
|
||||
|
||||
for route in router.routes:
|
||||
|
||||
Reference in New Issue
Block a user