背景:memory_sync_outbox 这条链此前**完全没有消费者**,且生产端照 docs/00 §6.4.6
写成大写 MILVUS/NEO4J + 中文「待处理」,而消费端按 target_store 的**值**分派 handler、
且只领 status in {pending, failed} —— 两个条件都不满足,事件任何消费者都领不到、
永久滞留且不报错(唯一键 (event_uuid, target_store) 对大小写无约束,MySQL 也不报错)。
根因是代码与测试都硬编码字面量,所以测试跟着一起错、谁也没拦住。
订正
- profile_generation_service:取值改为全仓一致的小写(milvus/neo4j/upsert/pending)
- 测试改为引用常量并断言消费端契约,不再硬编码(硬编码是本次跑偏的直接原因)
- 新增契约回归测试:断言大写值分派不到 handler、会进死信,谁改回大写立刻红
- 新增 tools/normalize_memory_sync_outbox.py:订正历史脏行(默认 dry-run、幂等)
接通投影链路(此前零消费者)
- 新增 Milvus 集合 user_long_term_memory_v1 及建集合工具(幂等、不覆盖已有集合)
- 新增 MilvusProfileProjection / MilvusProfileVectorClient,并修掉移植带来的两处必炸点:
customer_id 由「必须 int」放宽为接受数字字符串(本仓所有生产者都写 str,
不放宽则每个事件必然失败);不可投影的 memory_key 由「整批 raise」改为跳过留痕
(否则一条 constraint: 记忆毒死该客户整批,而受控词表 13 个键里有 7 个不满足前缀)
- 新增 MemorySyncOutboxWorker(领取/指数退避/死信骨架保留原样)并接入 WorkerRuntime
- milvus → 向量投影;neo4j → 复用主干 ProfileGraphProjectionService(方案 A,
不引入第二套投影,避免同一事实在图中两种说法、违反主干既有的只投影已确认事实的不变式)
- 生产端从 memory_unit(status=active) 组装 memory_sources,随事件带上确定快照
- 前置移植 conversation_privacy:写外部存储前脱敏手机号/证件号/银行卡等
验证
- 新增 17 个单测;全量 2 failed, 1307 passed, 2 skipped
(2 个失败为既有环境项:断言请求体中文原文而 httpx 序列化成 \uXXXX,非本次引入)
- mypy app → 0 错(227 文件);audit_schema → 89 张业务表无缺失/意外,未改动表结构
- 真机:真实 embedding(1024 维) + 真实 Milvus 写入并回读通过
- 整合链路(测试记忆 → 生产端组装 → outbox → 消费端投递 → Milvus 回读)通过,
且 MySQL 已回滚、Milvus 无残留
文档
- 新增 docs/32-记忆投影链路实现说明.md:真实口径、根因、契约与验证证据(供接手)
- AGENTS.md:新增该易错点;新增 Windows 中文输出乱码的正确命令(-X utf8);
校正测试基线与 mypy 文件数
未做:未改 docs/00 基线、未动数据库迁移、未改投顾线代码、未启动常驻 Worker。
遗留:投顾线两处生产者的 payload 缺 memory_sources,会被消费至死信,待架构师确认是否投影。
172 lines
6.5 KiB
Python
172 lines
6.5 KiB
Python
"""画像生成的数据访问层(Repository)。
|
||
|
||
按 MVC+S 边界:**Service 不得直接建 Session / 直接查 Model**,所有库交互收口在这里。
|
||
|
||
只做两件事:
|
||
1. **只读**组装画像所需的权威事实(`docs/00` §6.4.6:画像生成只允许读取**已提交的**业务事实
|
||
和状态为**有效**的中期记忆);
|
||
2. **写入**画像版本与两条跨存储同步事件(同事务,由调用方管理事务)。
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
from datetime import datetime
|
||
from typing import Any
|
||
|
||
from sqlalchemy import text
|
||
from sqlalchemy.ext.asyncio import AsyncSession
|
||
|
||
from app.model.memory import MemorySyncOutbox
|
||
|
||
#: 画像快照:查询客户主表。
|
||
_SQL_PROFILE = text("""
|
||
SELECT customer_id, investor_type, investment_horizon, preferred_asset_class,
|
||
trading_frequency, total_asset, behavior_score, risk_tags, last_active_at
|
||
FROM fin_customer_profile
|
||
WHERE customer_id = :customer_id
|
||
""")
|
||
|
||
#: 风险测评:取**最新一条**,含有效期(用于判定是否过期)。
|
||
_SQL_ASSESSMENT = text("""
|
||
SELECT id, questionnaire_version, investor_type, total_score, assessed_at, valid_until
|
||
FROM fin_risk_assessment
|
||
WHERE customer_id = :customer_id
|
||
ORDER BY assessed_at DESC, id DESC
|
||
LIMIT 1
|
||
""")
|
||
|
||
#: 当前画像版本(取 `is_current=1` 那一条)。
|
||
_SQL_CURRENT_SNAPSHOT = text("""
|
||
SELECT id, profile_uuid, version, snapshot, snapshot_hash, generation_basis
|
||
FROM profile_snapshots
|
||
WHERE customer_id = :customer_id AND is_current = 1
|
||
LIMIT 1
|
||
""")
|
||
|
||
#: 下一版本号 = 当前最大版本 + 1
|
||
#: (`uk_profile_snapshot_version (customer_id, version)` 要求版本号按客户唯一)。
|
||
_SQL_NEXT_VERSION = text("""
|
||
SELECT COALESCE(MAX(version), 0) + 1
|
||
FROM profile_snapshots WHERE customer_id = :customer_id
|
||
""")
|
||
|
||
#: 该客户**有效**的长期记忆(投影到 Milvus 长期记忆向量集合的数据源)。
|
||
#:
|
||
#: 只取 `status='active'`:失效/被取代的记忆不应再进入向量召回,否则投顾会召回
|
||
#: 已过期偏好。字段与 `MilvusProfileProjection` 的 `memory_sources` 契约一一对应。
|
||
_SQL_ACTIVE_MEMORIES = text("""
|
||
SELECT memory_uuid, memory_key, content, memory_type, confidence, version, valid_until
|
||
FROM memory_unit
|
||
WHERE customer_id = :customer_id AND status = 'active'
|
||
ORDER BY id
|
||
""")
|
||
|
||
#: 旧画像置非当前(`current_customer_id` 有唯一键,必须先置 0 才能插新的当前版本)。
|
||
_SQL_CLEAR_CURRENT = text("""
|
||
UPDATE profile_snapshots
|
||
SET is_current = 0, current_customer_id = NULL, updated_at = :now
|
||
WHERE customer_id = :customer_id AND is_current = 1
|
||
""")
|
||
|
||
_SQL_INSERT_SNAPSHOT = text("""
|
||
INSERT INTO profile_snapshots
|
||
(profile_uuid, customer_id, version, snapshot, generation_basis, snapshot_hash,
|
||
is_current, current_customer_id, generated_at, created_at, updated_at)
|
||
VALUES
|
||
(:profile_uuid, :customer_id, :version, :snapshot, :generation_basis, :snapshot_hash,
|
||
1, :customer_id, :now, :now, :now)
|
||
""")
|
||
|
||
|
||
class ProfileRepository:
|
||
def __init__(self, session: AsyncSession) -> None:
|
||
self._session = session
|
||
|
||
# --- 只读事实 ---------------------------------------------------------
|
||
|
||
async def profile_row(self, customer_id: int) -> dict[str, Any] | None:
|
||
result = await self._session.execute(_SQL_PROFILE, {"customer_id": customer_id})
|
||
row = result.mappings().first()
|
||
return dict(row) if row is not None else None
|
||
|
||
async def latest_assessment(self, customer_id: int) -> dict[str, Any] | None:
|
||
result = await self._session.execute(_SQL_ASSESSMENT, {"customer_id": customer_id})
|
||
row = result.mappings().first()
|
||
return dict(row) if row is not None else None
|
||
|
||
async def current_snapshot(self, customer_id: int) -> dict[str, Any] | None:
|
||
result = await self._session.execute(_SQL_CURRENT_SNAPSHOT, {"customer_id": customer_id})
|
||
row = result.mappings().first()
|
||
return dict(row) if row is not None else None
|
||
|
||
async def active_memories(self, customer_id: int) -> list[dict[str, Any]]:
|
||
"""该客户所有 `status='active'` 的长期记忆,按 id 稳定排序。"""
|
||
result = await self._session.execute(
|
||
_SQL_ACTIVE_MEMORIES, {"customer_id": customer_id}
|
||
)
|
||
return [dict(row) for row in result.mappings().all()]
|
||
|
||
async def next_version(self, customer_id: int) -> int:
|
||
return int(await self._session.scalar(_SQL_NEXT_VERSION, {"customer_id": customer_id}))
|
||
|
||
# --- 写入(调用方管事务) ---------------------------------------------
|
||
|
||
async def clear_current(self, customer_id: int, *, now: datetime) -> None:
|
||
"""把旧画像的 `is_current` 置 0 —— 必须在插入新版本**之前**调用。
|
||
|
||
`uk_profile_snapshot_current` 建在 `current_customer_id` 上(唯一),
|
||
所以同一客户同时只能有一条 `is_current=1`;顺序颠倒会撞唯一键。
|
||
"""
|
||
await self._session.execute(_SQL_CLEAR_CURRENT, {"customer_id": customer_id, "now": now})
|
||
|
||
async def insert_snapshot(
|
||
self,
|
||
*,
|
||
customer_id: int,
|
||
version: int,
|
||
profile_uuid: str,
|
||
snapshot_json: str,
|
||
generation_basis_json: str,
|
||
snapshot_hash: str,
|
||
now: datetime,
|
||
) -> None:
|
||
await self._session.execute(_SQL_INSERT_SNAPSHOT, {
|
||
"profile_uuid": profile_uuid,
|
||
"customer_id": customer_id,
|
||
"version": version,
|
||
"snapshot": snapshot_json,
|
||
"generation_basis": generation_basis_json,
|
||
"snapshot_hash": snapshot_hash,
|
||
"now": now,
|
||
})
|
||
|
||
def add_sync_event(
|
||
self,
|
||
*,
|
||
event_uuid: str,
|
||
target_store: str,
|
||
aggregate_uuid: str,
|
||
aggregate_version: int,
|
||
payload: dict[str, Any],
|
||
now: datetime,
|
||
status: str,
|
||
operation: str,
|
||
) -> None:
|
||
"""追加一条跨存储同步事件。
|
||
|
||
唯一键 `uk_memory_sync_event (event_uuid, target_store)`:**同一 `event_uuid` 允许
|
||
对两个目标库各写一条**,因此 Milvus 与 Neo4j 共用同一个 `event_uuid`。
|
||
"""
|
||
self._session.add(MemorySyncOutbox(
|
||
event_uuid=event_uuid,
|
||
aggregate_type="profile",
|
||
aggregate_uuid=aggregate_uuid,
|
||
aggregate_version=aggregate_version,
|
||
target_store=target_store,
|
||
operation=operation,
|
||
payload=payload,
|
||
status=status,
|
||
retry_count=0,
|
||
created_at=now,
|
||
))
|