fix(memory-projection): 订正 outbox 取值口径并接通画像投影链路
背景:memory_sync_outbox 这条链此前**完全没有消费者**,且生产端照 docs/00 §6.4.6
写成大写 MILVUS/NEO4J + 中文「待处理」,而消费端按 target_store 的**值**分派 handler、
且只领 status in {pending, failed} —— 两个条件都不满足,事件任何消费者都领不到、
永久滞留且不报错(唯一键 (event_uuid, target_store) 对大小写无约束,MySQL 也不报错)。
根因是代码与测试都硬编码字面量,所以测试跟着一起错、谁也没拦住。
订正
- profile_generation_service:取值改为全仓一致的小写(milvus/neo4j/upsert/pending)
- 测试改为引用常量并断言消费端契约,不再硬编码(硬编码是本次跑偏的直接原因)
- 新增契约回归测试:断言大写值分派不到 handler、会进死信,谁改回大写立刻红
- 新增 tools/normalize_memory_sync_outbox.py:订正历史脏行(默认 dry-run、幂等)
接通投影链路(此前零消费者)
- 新增 Milvus 集合 user_long_term_memory_v1 及建集合工具(幂等、不覆盖已有集合)
- 新增 MilvusProfileProjection / MilvusProfileVectorClient,并修掉移植带来的两处必炸点:
customer_id 由「必须 int」放宽为接受数字字符串(本仓所有生产者都写 str,
不放宽则每个事件必然失败);不可投影的 memory_key 由「整批 raise」改为跳过留痕
(否则一条 constraint: 记忆毒死该客户整批,而受控词表 13 个键里有 7 个不满足前缀)
- 新增 MemorySyncOutboxWorker(领取/指数退避/死信骨架保留原样)并接入 WorkerRuntime
- milvus → 向量投影;neo4j → 复用主干 ProfileGraphProjectionService(方案 A,
不引入第二套投影,避免同一事实在图中两种说法、违反主干既有的只投影已确认事实的不变式)
- 生产端从 memory_unit(status=active) 组装 memory_sources,随事件带上确定快照
- 前置移植 conversation_privacy:写外部存储前脱敏手机号/证件号/银行卡等
验证
- 新增 17 个单测;全量 2 failed, 1307 passed, 2 skipped
(2 个失败为既有环境项:断言请求体中文原文而 httpx 序列化成 \uXXXX,非本次引入)
- mypy app → 0 错(227 文件);audit_schema → 89 张业务表无缺失/意外,未改动表结构
- 真机:真实 embedding(1024 维) + 真实 Milvus 写入并回读通过
- 整合链路(测试记忆 → 生产端组装 → outbox → 消费端投递 → Milvus 回读)通过,
且 MySQL 已回滚、Milvus 无残留
文档
- 新增 docs/32-记忆投影链路实现说明.md:真实口径、根因、契约与验证证据(供接手)
- AGENTS.md:新增该易错点;新增 Windows 中文输出乱码的正确命令(-X utf8);
校正测试基线与 mypy 文件数
未做:未改 docs/00 基线、未动数据库迁移、未改投顾线代码、未启动常驻 Worker。
遗留:投顾线两处生产者的 payload 缺 memory_sources,会被消费至死信,待架构师确认是否投影。
This commit is contained in:
@@ -22,6 +22,7 @@ from app.service.agent.bootstrap import (
|
||||
get_memory_cache_adapter,
|
||||
get_memory_embedding_service,
|
||||
get_milvus_knowledge_writer,
|
||||
get_milvus_profile_vector_client,
|
||||
get_model_service,
|
||||
)
|
||||
from app.service.agent.executor import AgentExecutor
|
||||
@@ -91,6 +92,7 @@ class WorkerRuntime:
|
||||
knowledge_writer: Any = _UNSET,
|
||||
knowledge_embedder: Any = _UNSET,
|
||||
knowledge_endpoint_resolver: Any = _UNSET,
|
||||
profile_vector_client: Any = _UNSET,
|
||||
) -> None:
|
||||
self.factory = factory if factory is not None else get_agent_factory()
|
||||
self.settings = settings or get_settings()
|
||||
@@ -138,6 +140,18 @@ class WorkerRuntime:
|
||||
)
|
||||
# 降级告警只打一次:dispatch 是轮询热路径,每轮一条 warning 会把日志淹掉。
|
||||
self._knowledge_degraded_logged = False
|
||||
# 画像投影(`memory_sync_outbox`)消费装配。
|
||||
#
|
||||
# 与 `knowledge_writer` 同一取向:客户端构造**惰性**(不连 Milvus),
|
||||
# `milvus_uri` 未配置时显式降级为不注册 handler(事件留 pending、可观测、可重放),
|
||||
# 绝不伪造同步成功。
|
||||
self.profile_vector_client = (
|
||||
get_milvus_profile_vector_client()
|
||||
if profile_vector_client is _UNSET
|
||||
else profile_vector_client
|
||||
)
|
||||
self.profile_endpoint_resolver = self.knowledge_endpoint_resolver
|
||||
self._profile_degraded_logged = False
|
||||
# episode 聚合是低频批处理,按轮次节流而不是每轮都查。
|
||||
self._episode_rounds = 0
|
||||
|
||||
@@ -408,6 +422,14 @@ class WorkerRuntime:
|
||||
|
||||
async def run_once(self) -> bool:
|
||||
dispatched = await self.dispatch_batch() > 0
|
||||
# 画像投影消费:与领域事件同一轮次内处理。失败只告警,不影响 run 的处理与
|
||||
# 轮询节奏——事件仍在库里,下一轮照常重试(退避由 worker 自己记在 next_retry_at)。
|
||||
try:
|
||||
projected = await self.consume_profile_projections() > 0
|
||||
except Exception:
|
||||
logger.warning("profile projection consumption failed", exc_info=True)
|
||||
projected = False
|
||||
dispatched = dispatched or projected
|
||||
self._episode_rounds += 1
|
||||
if self._episode_rounds % EPISODE_INTERVAL_ROUNDS == 0:
|
||||
# 会话片段聚合:内部幂等(content_hash 唯一键),失败只告警,
|
||||
@@ -431,6 +453,85 @@ class WorkerRuntime:
|
||||
return dispatched
|
||||
return await self.execute(run_id) or dispatched
|
||||
|
||||
async def consume_profile_projections(self, *, limit: int = 20) -> int:
|
||||
"""消费 `memory_sync_outbox` 的画像投影事件,返回本次处理条数。
|
||||
|
||||
两个目标存储的分工(**方案 A**:以架构师主干为主线,不引入第二套 Neo4j 投影):
|
||||
|
||||
- `milvus` → 写入长期记忆向量集合 `user_long_term_memory_v1`
|
||||
(此前**完全没有消费者**,事件永久滞留);
|
||||
- `neo4j` → 复用主干 `ProfileGraphProjectionService`。主干已由
|
||||
`profile.rebuild_requested` 事件驱动同一条链,图投影是 `MERGE` 幂等的,
|
||||
因此这里再投一次不产生重复节点/关系,只用于把 outbox 行的投递状态收敛掉。
|
||||
|
||||
为什么不让 handler 自己 commit:事务边界与 `dispatch_batch` 一致,
|
||||
由本方法按条提交;单条失败由 `MemorySyncOutboxWorker` 内部转成
|
||||
`failed`+退避或死信,不冒泡打断本轮其余事件。
|
||||
"""
|
||||
if self.profile_vector_client is None:
|
||||
# 显式降级:不注册 handler 就交给 worker 判死信是**错的**(那是把配置缺失
|
||||
# 伪装成投递失败)。这里直接不消费,事件保持 pending,由启动日志提示。
|
||||
if not self._profile_degraded_logged:
|
||||
self._profile_degraded_logged = True
|
||||
logger.warning(
|
||||
"profile projection disabled: milvus profile vector client unavailable; "
|
||||
"memory_sync_outbox events stay pending"
|
||||
)
|
||||
return 0
|
||||
# 收窄到局部变量:闭包内访问 self 属性时 mypy 无法保留上面的 None 判定。
|
||||
vector_client = self.profile_vector_client
|
||||
|
||||
from app.infrastructure.milvus_profile_projection import MilvusProfileProjection
|
||||
from app.worker.memory_sync_outbox_worker import MemorySyncOutboxWorker
|
||||
|
||||
async def project_milvus(payload: dict[str, Any]) -> None:
|
||||
projection = MilvusProfileProjection(vector_client, self._profile_embed)
|
||||
await projection.upsert(payload)
|
||||
|
||||
async def project_neo4j(payload: dict[str, Any]) -> None:
|
||||
raw_customer_id = payload.get("customer_id")
|
||||
# 显式 isinstance 而不是 `in (None, "")`:后者不做类型收窄,mypy 无法确认
|
||||
# int() 的入参类型;同时也把"客户号必须是数字"这一契约写在类型检查里。
|
||||
if not isinstance(raw_customer_id, (int, str)) or raw_customer_id == "":
|
||||
raise RecoverableAgentError("profile projection payload has no customer_id")
|
||||
customer_id = int(raw_customer_id)
|
||||
# 延迟导入:与 dispatch_profile_rebuild 同一理由,避免模块级循环依赖。
|
||||
from app.service.profile_graph_projection_service import (
|
||||
ProfileGraphProjectionService,
|
||||
)
|
||||
|
||||
async with SessionFactory() as session:
|
||||
outcome = await ProfileGraphProjectionService(
|
||||
session, self.relationships
|
||||
).project_customer(customer_id)
|
||||
if outcome.degraded:
|
||||
# 图库不可用:如实抛出,让 worker 走失败/退避,而不是记成已投递。
|
||||
raise RecoverableAgentError(f"graph projection degraded: {outcome.reason}")
|
||||
|
||||
worker = MemorySyncOutboxWorker(
|
||||
{"milvus": project_milvus, "neo4j": project_neo4j}
|
||||
)
|
||||
handled = 0
|
||||
for _ in range(max(1, limit)):
|
||||
if not await worker.run_once():
|
||||
break
|
||||
handled += 1
|
||||
return handled
|
||||
|
||||
async def _profile_embed(self, text: str) -> list[float]:
|
||||
"""向量化一条记忆正文;端点走与知识向量化同一套已批准端点解析。
|
||||
|
||||
不复用 `bootstrap._embed_text`:那是模块私有函数,跨模块引用私有名会把
|
||||
两处的耦合藏起来。这里用同一组公开装配(端点解析器 + embedding 服务)。
|
||||
"""
|
||||
endpoints = await self.profile_endpoint_resolver.resolve(
|
||||
agent_type="memory_recall", task_type="embedding"
|
||||
)
|
||||
if not endpoints:
|
||||
raise RecoverableAgentError("没有可用的 embedding 端点,无法投影长期记忆")
|
||||
execution = await self.knowledge_embedder.embed(endpoints, text)
|
||||
return list(execution.vector)
|
||||
|
||||
async def aggregate_episodes(self, *, customer_limit: int = 50) -> int:
|
||||
"""把已静默的会话片段聚合为 episode,返回新写入的片段数。
|
||||
|
||||
|
||||
Reference in New Issue
Block a user