wangjianlong_0626
8a0cbab636
背景：memory_sync_outbox 这条链此前**完全没有消费者**，且生产端照 docs/00 §6.4.6
写成大写 MILVUS/NEO4J + 中文「待处理」，而消费端按 target_store 的**值**分派 handler、
且只领 status in {pending, failed} —— 两个条件都不满足，事件任何消费者都领不到、
永久滞留且不报错（唯一键 (event_uuid, target_store) 对大小写无约束，MySQL 也不报错）。
根因是代码与测试都硬编码字面量，所以测试跟着一起错、谁也没拦住。
订正
- profile_generation_service：取值改为全仓一致的小写（milvus/neo4j/upsert/pending）
- 测试改为引用常量并断言消费端契约，不再硬编码（硬编码是本次跑偏的直接原因）
- 新增契约回归测试：断言大写值分派不到 handler、会进死信，谁改回大写立刻红
- 新增 tools/normalize_memory_sync_outbox.py：订正历史脏行（默认 dry-run、幂等）
接通投影链路（此前零消费者）
- 新增 Milvus 集合 user_long_term_memory_v1 及建集合工具（幂等、不覆盖已有集合）
- 新增 MilvusProfileProjection / MilvusProfileVectorClient，并修掉移植带来的两处必炸点：
customer_id 由「必须 int」放宽为接受数字字符串（本仓所有生产者都写 str，
不放宽则每个事件必然失败）；不可投影的 memory_key 由「整批 raise」改为跳过留痕
（否则一条 constraint: 记忆毒死该客户整批，而受控词表 13 个键里有 7 个不满足前缀）
- 新增 MemorySyncOutboxWorker（领取/指数退避/死信骨架保留原样）并接入 WorkerRuntime
- milvus → 向量投影；neo4j → 复用主干 ProfileGraphProjectionService（方案 A，
不引入第二套投影，避免同一事实在图中两种说法、违反主干既有的只投影已确认事实的不变式）
- 生产端从 memory_unit(status=active) 组装 memory_sources，随事件带上确定快照
- 前置移植 conversation_privacy：写外部存储前脱敏手机号/证件号/银行卡等
验证
- 新增 17 个单测；全量 2 failed, 1307 passed, 2 skipped
（2 个失败为既有环境项：断言请求体中文原文而 httpx 序列化成 \uXXXX，非本次引入）
- mypy app → 0 错（227 文件）；audit_schema → 89 张业务表无缺失/意外，未改动表结构
- 真机：真实 embedding(1024 维) + 真实 Milvus 写入并回读通过
- 整合链路（测试记忆 → 生产端组装 → outbox → 消费端投递 → Milvus 回读）通过，
且 MySQL 已回滚、Milvus 无残留
文档
- 新增 docs/32-记忆投影链路实现说明.md：真实口径、根因、契约与验证证据（供接手）
- AGENTS.md：新增该易错点；新增 Windows 中文输出乱码的正确命令（-X utf8）；
校正测试基线与 mypy 文件数
未做：未改 docs/00 基线、未动数据库迁移、未改投顾线代码、未启动常驻 Worker。
遗留：投顾线两处生产者的 payload 缺 memory_sources，会被消费至死信，待架构师确认是否投影。
192 lines
8.4 KiB
Python
192 lines
8.4 KiB
Python
"""Milvus 长期记忆投影适配器。
|
||
|
||
只写入已经审核的 `memory_sources`，不接受画像快照整体冒充单条记忆。
|
||
|
||
来源：同事 `ZSY_develop` 分支（`app/infrastructure/milvus_profile_projection.py`），
|
||
本文件在其基础上做了**两处契约放宽**，均为"避免整批失败"，不改变写入语义：
|
||
|
||
1. **`customer_id` 接受 int 或数字字符串**。原实现要求 `isinstance(customer_id, int)`，
|
||
而本仓**所有** outbox 生产者写的都是 `str(customer_id)`
|
||
（`profile_generation_service`、投顾线 `profile_governance_service` 与
|
||
`risk_questionnaire_service` 三处皆然）。不放宽则每个事件必然失败、重试 5 次后进死信。
|
||
2. **不可投影的 `memory_key` 跳过而非整批报错**。受控词表
|
||
（`app/service/memory_taxonomy.py`）共 13 个键，其中 `constraint:*`（3 个）与
|
||
`profile:*`（4 个）不以 `preference:`/`goal:` 开头。原实现对第一个不合规的键直接
|
||
`raise`，一条 `constraint:` 记忆就会毒死该客户整批同步；现改为跳过并留痕，
|
||
使"能投影的照常写入"。
|
||
"""
|
||
|
||
import logging
|
||
from collections.abc import Awaitable, Callable
|
||
from datetime import UTC, datetime
|
||
from typing import Any, Protocol
|
||
from uuid import UUID
|
||
|
||
from app.core.conversation_privacy import sanitize_customer_service_message
|
||
from app.core.errors import RecoverableAgentError
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
PROFILE_COLLECTION = "user_long_term_memory_v1"
|
||
VECTOR_DIM = 1024
|
||
|
||
#: 可投影的键前缀。其余受控键（`constraint:*` / `profile:*`）属结构化字段，
|
||
#: 与向量召回不是同一用途，因此不进长期记忆向量集合。
|
||
PROJECTABLE_KEY_PREFIXES: tuple[str, ...] = ("preference:", "goal:")
|
||
|
||
|
||
class MilvusProfileClient(Protocol):
|
||
async def query(self, **kwargs: Any) -> list[dict[str, Any]]: ...
|
||
|
||
async def upsert(self, **kwargs: Any) -> Any: ...
|
||
|
||
|
||
EmbeddingProvider = Callable[[str], Awaitable[list[float]]]
|
||
|
||
|
||
class MilvusProfileProjection:
|
||
"""按记忆 UUID 幂等写入长期记忆向量。"""
|
||
|
||
def __init__(
|
||
self,
|
||
client: MilvusProfileClient,
|
||
embed: EmbeddingProvider,
|
||
*,
|
||
collection: str = PROFILE_COLLECTION,
|
||
) -> None:
|
||
self._client = client
|
||
self._embed = embed
|
||
self._collection = collection
|
||
|
||
async def upsert(self, payload: dict[str, Any]) -> None:
|
||
customer_id, profile_version, sources = self._normalize(payload)
|
||
load_collection = getattr(self._client, "load_collection", None)
|
||
if load_collection is not None:
|
||
await load_collection(collection_name=self._collection)
|
||
rows: list[dict[str, Any]] = []
|
||
for source in sources:
|
||
vector = await self._embed(source["content"])
|
||
if len(vector) != VECTOR_DIM:
|
||
raise RecoverableAgentError("画像向量维度不一致")
|
||
existing = await self._client.query(
|
||
collection_name=self._collection,
|
||
filter=f'memory_uuid == "{source["memory_uuid"]}"',
|
||
output_fields=["memory_uuid", "version", "customer_id"],
|
||
)
|
||
# 幂等：库里已有更新版本时不回退覆盖。
|
||
if existing and int(existing[0].get("version", 0)) > source["version"]:
|
||
continue
|
||
rows.append({
|
||
"memory_uuid": source["memory_uuid"],
|
||
"customer_id": customer_id,
|
||
"content": source["content"],
|
||
"embedding": vector,
|
||
"memory_type": source["memory_type"],
|
||
"memory_key": source["memory_key"],
|
||
"confidence": source["confidence"],
|
||
"version": source["version"],
|
||
"status": "active",
|
||
"valid_until_ts": source["valid_until_ts"],
|
||
"updated_at_ts": source["updated_at_ts"],
|
||
})
|
||
if rows:
|
||
await self._client.upsert(collection_name=self._collection, data=rows)
|
||
# 全部被跳过时也要留一条记录：否则"零写入"与"没跑"在日志上无法区分。
|
||
logger.info(
|
||
"milvus profile projection: customer_id=%s profile_version=%s written=%s",
|
||
customer_id,
|
||
profile_version,
|
||
len(rows),
|
||
)
|
||
|
||
@staticmethod
|
||
def _coerce_customer_id(raw: Any) -> int:
|
||
"""接受 int 或数字字符串。
|
||
|
||
生产端统一写 `str(customer_id)`（本仓三处生产者皆然），若严格要求 int，
|
||
所有事件都会失败。仅接受**纯数字**字符串，非数字一律拒绝，
|
||
避免把 uuid 之类当成客户号写进向量库。
|
||
"""
|
||
if isinstance(raw, bool):
|
||
raise ValueError("customer_id is invalid")
|
||
if isinstance(raw, int):
|
||
value = raw
|
||
elif isinstance(raw, str) and raw.strip().isdigit():
|
||
value = int(raw.strip())
|
||
else:
|
||
raise ValueError("customer_id is invalid")
|
||
if value <= 0:
|
||
raise ValueError("customer_id is invalid")
|
||
return value
|
||
|
||
@staticmethod
|
||
def _coerce_profile_version(payload: dict[str, Any]) -> int:
|
||
"""兼容两种键名：`profile_version`（本适配器契约）与 `version`（本仓生产端）。"""
|
||
raw = payload.get("profile_version", payload.get("version"))
|
||
if isinstance(raw, bool) or not isinstance(raw, int) or raw <= 0:
|
||
raise ValueError("profile_version is invalid")
|
||
return raw
|
||
|
||
@classmethod
|
||
def _normalize(
|
||
cls,
|
||
payload: dict[str, Any],
|
||
) -> tuple[int, int, list[dict[str, Any]]]:
|
||
customer_id = cls._coerce_customer_id(payload.get("customer_id"))
|
||
profile_version = cls._coerce_profile_version(payload)
|
||
sources = payload.get("memory_sources")
|
||
if not isinstance(sources, list):
|
||
raise ValueError("memory_sources is invalid")
|
||
normalized: list[dict[str, Any]] = []
|
||
skipped: list[str] = []
|
||
now = int(datetime.now(UTC).timestamp())
|
||
for source in sources:
|
||
if not isinstance(source, dict):
|
||
raise ValueError("memory source is invalid")
|
||
required = [
|
||
source.get(name)
|
||
for name in ("memory_uuid", "memory_key", "content", "memory_type")
|
||
]
|
||
if not all(isinstance(value, str) and value.strip() for value in required):
|
||
raise ValueError("memory source fields are invalid")
|
||
try:
|
||
memory_uuid = str(UUID(str(source["memory_uuid"])))
|
||
except ValueError as exc:
|
||
raise ValueError("memory_uuid is invalid") from exc
|
||
memory_key = str(source["memory_key"]).strip()
|
||
if not memory_key.startswith(PROJECTABLE_KEY_PREFIXES):
|
||
# 跳过而非整批失败：`constraint:*` / `profile:*` 是结构化事实，
|
||
# 不进向量召回。一条不可投影的键不得毒死同一客户其余记忆。
|
||
skipped.append(memory_key)
|
||
continue
|
||
confidence = source.get("confidence")
|
||
version = source.get("version")
|
||
if not isinstance(confidence, (int, float)) or not 0 <= confidence <= 1:
|
||
raise ValueError("memory confidence is invalid")
|
||
if not isinstance(version, int) or version <= 0:
|
||
raise ValueError("memory version is invalid")
|
||
valid_until = source.get("valid_until")
|
||
valid_until_ts = None
|
||
if isinstance(valid_until, str) and valid_until:
|
||
try:
|
||
valid_until_ts = int(datetime.fromisoformat(valid_until).timestamp())
|
||
except ValueError as exc:
|
||
raise ValueError("memory valid_until is invalid") from exc
|
||
normalized.append({
|
||
"memory_uuid": memory_uuid,
|
||
"memory_key": memory_key,
|
||
"content": sanitize_customer_service_message(str(source["content"])).strip(),
|
||
"memory_type": str(source["memory_type"]).strip(),
|
||
"confidence": float(confidence),
|
||
"version": version,
|
||
"valid_until_ts": valid_until_ts,
|
||
"updated_at_ts": now,
|
||
})
|
||
if skipped:
|
||
logger.info(
|
||
"milvus profile projection: skipped %s non-projectable memory keys %s",
|
||
len(skipped),
|
||
sorted(set(skipped)),
|
||
)
|
||
return customer_id, profile_version, normalized
|