基金转换 T-11：工具与 SQL 汇总去重（FR-C15 / R-d）
convert 在 core_trade 落两条流水（转出 redeem + 转入 subscribe，共享 convert_group_id）， 所有金额聚合读取方必须只计一次，否则同笔转换金额翻倍。 落地（改码 3 处 + 新增 1 脚本 + 测试 3 文件） - rules.py：_amount_view → 公开 amount_view（提升而非复制别名，全仓唯一金额聚合口径）； docstring 补「跨模块共用」说明 - core_tools.py：query_recent_trades 的 sum_amount 改走 amount_view（FR-C15）； items / total_count 保持全量不变——两条流水是真实的两笔权益变动，读到两条是对的 - core_ro.py：list_holdings 加 h.qty > 0（convert 转出全部份额留下的 qty=0 归零行不是持仓）； sum_trades_on_date 加 convert 去重条件（R-d），组内只计转出端 - 新增 tests/test_core_tools.py（3 条：汇总不翻倍 / 跨口径一致性 / 持仓不返回 qty=0 行） - tests/test_core_ro_sum.py 追加 1 条 convert 去重用例（既有断言零改动） - 新增 scripts/dev/verify_convert_tools.py：真库验证脚本（MySQL 8.0.46） 执行期裁定 2 条（已留痕） - _amount_view 提升为公开而非复制：一个函数两个名字会漂移（自检第 13 问） - SQL 条件从 IS NULL 扩为 IS NULL OR = ''：Python 的 if not gid 把空串当无组， 而 SQL 里 '' IS NULL 恒假。真库实证：只写 IS NULL → 合计数 400000（正确 450000，漏算 50000）。 不加则 RISK-002 漏算，且 sqlite 单测若只造 NULL 数据永远发现不了 - 等价性边界：组内无 redeem 时 SQL 丢整组、amount_view 保留首条；由 R-b 保证不可达 验证 - pytest -q → 718 passed / 3 skipped（基线 714 加 4，零回归） - 突变验证 3 组均精准命中：去掉去重条件（2 红）/ 去掉 qty>0（1 红）/ 汇总不走 amount_view（2 红） - 真库 verify_convert_tools.py 14/14，隔离数据零残留 - 复跑受影响真库脚本零回归：T-8 31/31、T-7 35/35、T-10 20/20
This commit is contained in:
@@ -6,7 +6,7 @@
|
||||
`convert_service._run_engine` 走 `ImportError` 分支**静默跳过**；T-8 落地后同一笔转换会
|
||||
**真实出单 + 写 L3 + 落审计**。真库跑一遍才能确认没有连带破坏（审计条数、L3 写入、单落库）。
|
||||
2. **`convert_group_id` 的 NULL 值域**：MySQL 里普通交易该列为 NULL、convert 两条为同一串值；
|
||||
`_amount_view` 的 `if not gid` 必须在真库值域下成立（NULL / 空串都不得误聚合）。
|
||||
`amount_view` 的 `if not gid` 必须在真库值域下成立（NULL / 空串都不得误聚合）。
|
||||
3. **DECIMAL 精度**：`core_trade.amount` 在 MySQL 是 DECIMAL，进 Python 后与生产纯函数
|
||||
逐项比对（sqlite 用 REAL 无此保证，需 `float()` 绑定）。
|
||||
4. **JSON 列反解**：`risk_alert.payload` 真库落库后反解出的 `events` 长度必须为 2。
|
||||
@@ -249,7 +249,7 @@ def main() -> int:
|
||||
core_engine = get_engine("jinrong_core", role="admin")
|
||||
agent_engine = get_engine("jinrong_agent", role="admin")
|
||||
print("=" * 60)
|
||||
print("T-8 真库验证：规则引擎改造（_amount_view 去重 + process_convert_event 出单）")
|
||||
print("T-8 真库验证：规则引擎改造（amount_view 去重 + process_convert_event 出单）")
|
||||
print("=" * 60)
|
||||
|
||||
cleanup(core_engine, agent_engine)
|
||||
|
||||
@@ -0,0 +1,240 @@
|
||||
"""T-11 真 MySQL 验证脚本：金额汇总去重 + 归零行过滤（DoD 逐条断言）。
|
||||
|
||||
**为什么 sqlite 单测全绿还不够**
|
||||
|
||||
T-11 的核心是把「一次转换只计一次金额」这个口径，从 Python（`rules.amount_view`）
|
||||
复制到 SQL（`core_ro.sum_trades_on_date`）。跨语言的两份实现最容易漂移，
|
||||
**在 NULL 值域上尤其危险**：
|
||||
|
||||
- `amount_view` 判的是 Python 的 `if not gid` —— **NULL 与空串都算「无组」**；
|
||||
- 若 SQL 只写 `convert_group_id IS NULL`，**空串会被漏掉**（三值逻辑里 `'' IS NULL` 恒为假），
|
||||
该笔普通交易从当日累计里凭空消失 → RISK-002 少算、且**测试很难发现**
|
||||
（sqlite 单测若只造 NULL 数据，这条分支永远不被执行）。
|
||||
|
||||
于是本脚本专门造一笔 `convert_group_id = ''` 的真库数据，并**同时跑两条 SQL 做对照**
|
||||
（只 `IS NULL` vs `IS NULL OR = ''`），把「差异是真实存在的」变成可打印的证据。
|
||||
|
||||
同时验证 `list_holdings` 的 `qty > 0` 在 MySQL `DECIMAL(18,4)` 下的比较语义
|
||||
（convert 转出全部份额后 `qty = 0.0000` 的台账留痕行必须被排除）。
|
||||
|
||||
用法：
|
||||
python scripts/dev/verify_convert_tools.py # 建隔离数据 → 跑断言 → 清理
|
||||
|
||||
注意：
|
||||
- 全部数据用 **TOOLT 前缀**（客户 `CUST-TOOLT` / 产品 `PROD-TOOLT*`），跑完清理干净，不碰既有种子；
|
||||
- 建/清数据走 `role="admin"`（需 DELETE，R-e）；被测路径走生产同款仓储
|
||||
（`CoreReadOnlyRepository` ro 读 + `core_tools` 同款函数）；
|
||||
- 退出码 1 = 有断言不一致（供 CI / 人工判定）。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from datetime import datetime, time, timedelta
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from app.config.settings import settings # noqa: E402
|
||||
from app.repository.core_ro import CoreReadOnlyRepository # noqa: E402
|
||||
from app.tool.core_tools import query_holdings, query_recent_trades # noqa: E402
|
||||
from app.utils.db import dispose_engines, get_engine # noqa: E402
|
||||
|
||||
CUSTOMER = "CUST-TOOLT"
|
||||
PROD_A = "PROD-TOOLTA" # 正常持仓（qty > 0）
|
||||
PROD_B = "PROD-TOOLTB" # 归零行（qty = 0，convert 转出全部后的台账留痕）
|
||||
GROUP = "G-TOOLT-1"
|
||||
|
||||
# 金额设计（每一项都能区分「对 / 未去重 / 只判 IS NULL」三种实现）：
|
||||
AMT_OUT = "300000" # convert 转出端 → 应计入
|
||||
AMT_IN = "300000" # convert 转入端 → 同组，不应计入
|
||||
AMT_PLAIN = "100000" # 普通赎回（gid = NULL）→ 应计入
|
||||
AMT_EMPTY = "50000" # gid = 空串 → 视为无组，**应计入**（区分 ='' 分支的关键）
|
||||
EXPECTED_SUM = Decimal("450000") # 300000 + 100000 + 50000
|
||||
SUM_IF_NO_DEDUP = Decimal("750000") # 未去重：再 + 300000
|
||||
SUM_IF_NULL_ONLY = Decimal("400000") # 只写 IS NULL：丢掉空串那笔 50000
|
||||
|
||||
_passed = 0
|
||||
_failed = 0
|
||||
|
||||
|
||||
def check(name: str, actual, expected) -> None:
|
||||
"""逐条断言并打印（与既有 verify_convert_*.py 同款输出）。"""
|
||||
global _passed, _failed
|
||||
ok = actual == expected
|
||||
if ok:
|
||||
_passed += 1
|
||||
else:
|
||||
_failed += 1
|
||||
flag = "✅" if ok else "❌"
|
||||
print(f" {flag} {name}: 实际 {actual!r}" + ("" if ok else f" / 期望 {expected!r}"))
|
||||
|
||||
|
||||
def dec(value, places: str = "0.01") -> Decimal:
|
||||
"""真库读回是 `Decimal`（MySQL DECIMAL）→ 统一量化后比较。"""
|
||||
return Decimal(str(value)).quantize(Decimal(places))
|
||||
|
||||
|
||||
def q1(engine, sql: str, **params):
|
||||
with engine.connect() as conn:
|
||||
return conn.execute(text(sql), params).scalar_one_or_none()
|
||||
|
||||
|
||||
# ── 数据准备 / 清理 ─────────────────────────────────────────────────
|
||||
def cleanup(engine) -> None:
|
||||
"""按 FK 依赖倒序删：trade → holding → product → customer（全部按前缀）。"""
|
||||
with engine.begin() as conn:
|
||||
conn.execute(
|
||||
text("DELETE FROM core_trade WHERE customer_id LIKE :p"), {"p": f"{CUSTOMER}%"}
|
||||
)
|
||||
conn.execute(
|
||||
text("DELETE FROM core_holding WHERE customer_id LIKE :p"), {"p": f"{CUSTOMER}%"}
|
||||
)
|
||||
conn.execute(
|
||||
text("DELETE FROM core_product WHERE product_id LIKE :p"), {"p": "PROD-TOOLT%"}
|
||||
)
|
||||
conn.execute(
|
||||
text("DELETE FROM core_customer WHERE customer_id LIKE :p"), {"p": f"{CUSTOMER}%"}
|
||||
)
|
||||
|
||||
|
||||
def seed(engine, now: datetime) -> None:
|
||||
today = now.date()
|
||||
with engine.begin() as conn:
|
||||
conn.execute(
|
||||
text(
|
||||
"INSERT INTO core_customer (customer_id, display_name, open_date) "
|
||||
"VALUES (:c, 'T11真库验证', :d)"
|
||||
),
|
||||
{"c": CUSTOMER, "d": today},
|
||||
)
|
||||
for pid, name in [(PROD_A, "T11正常持仓基金"), (PROD_B, "T11归零基金")]:
|
||||
conn.execute(
|
||||
text(
|
||||
# product_type 是真库 ENUM('money','bond','mixed',...) 的中文值非法
|
||||
"INSERT INTO core_product (product_id, product_name, min_risk_code, "
|
||||
"product_type, can_subscribe, can_redeem) "
|
||||
"VALUES (:p, :n, 'R3', 'mixed', 1, 1)"
|
||||
),
|
||||
{"p": pid, "n": name},
|
||||
)
|
||||
# 持仓：一条正常 + 一条归零（qty = 0，行必须保留 —— 台账留痕，不物理删除）
|
||||
conn.execute(
|
||||
text(
|
||||
"INSERT INTO core_holding (customer_id, product_id, qty, cost_amount, "
|
||||
"market_value, pnl_pct, as_of) VALUES "
|
||||
"(:c, :pa, 1000.0000, 1000.00, 1200.00, 0.2000, :d), "
|
||||
"(:c, :pb, 0.0000, 0.00, 0.00, 0.0000, :d)"
|
||||
),
|
||||
{"c": CUSTOMER, "pa": PROD_A, "pb": PROD_B, "d": today},
|
||||
)
|
||||
# 流水：一组 convert（两条共享 gid）+ 普通赎回（NULL）+ 空串 gid 申购
|
||||
rows = [
|
||||
("TX-TOOLT-OUT", PROD_A, "redeem", AMT_OUT, GROUP),
|
||||
("TX-TOOLT-IN", PROD_B, "subscribe", AMT_IN, GROUP),
|
||||
("TX-TOOLT-PLAIN", PROD_A, "redeem", AMT_PLAIN, None),
|
||||
("TX-TOOLT-EMPTY", PROD_A, "subscribe", AMT_EMPTY, ""),
|
||||
]
|
||||
for tid, pid, ttype, amt, gid in rows:
|
||||
conn.execute(
|
||||
text(
|
||||
"INSERT INTO core_trade (trade_id, customer_id, product_id, trade_type, "
|
||||
"amount, qty, convert_group_id, trade_status, traded_at) "
|
||||
"VALUES (:tid, :c, :p, :tt, :amt, 100.0000, :gid, 'confirmed', :t)"
|
||||
),
|
||||
{"tid": tid, "c": CUSTOMER, "p": pid, "tt": ttype, "amt": amt, "gid": gid, "t": now},
|
||||
)
|
||||
|
||||
|
||||
# ── 被测路径断言 ────────────────────────────────────────────────────
|
||||
def run_checks(engine, now: datetime) -> None:
|
||||
core_ro = CoreReadOnlyRepository(engine=get_engine(settings.mysql_core_database, "ro"))
|
||||
today = now.date()
|
||||
day_start = datetime.combine(today, time.min)
|
||||
|
||||
print("\n[A] 真库值域确认（前提）")
|
||||
gid_out = q1(
|
||||
engine, "SELECT convert_group_id FROM core_trade WHERE trade_id = 'TX-TOOLT-OUT'"
|
||||
)
|
||||
gid_plain = q1(
|
||||
engine, "SELECT convert_group_id FROM core_trade WHERE trade_id = 'TX-TOOLT-PLAIN'"
|
||||
)
|
||||
gid_empty = q1(
|
||||
engine, "SELECT convert_group_id FROM core_trade WHERE trade_id = 'TX-TOOLT-EMPTY'"
|
||||
)
|
||||
check("convert 转出端 gid 非空", gid_out, GROUP)
|
||||
check("普通交易 gid 为 NULL", gid_plain, None)
|
||||
check("空串 gid 真库存成空串（非 NULL）", gid_empty, "")
|
||||
|
||||
print("\n[B] sum_trades_on_date 去重（R-d）")
|
||||
actual_sum = core_ro.sum_trades_on_date(CUSTOMER, today)
|
||||
check("去重后当日合计", dec(actual_sum), EXPECTED_SUM)
|
||||
check("不等于未去重的值（去重真的生效）", dec(actual_sum) != SUM_IF_NO_DEDUP, True)
|
||||
|
||||
print("\n[C] 对照组：只写 IS NULL 会漏掉空串那笔（证明 ='' 分支必要）")
|
||||
null_only = q1(
|
||||
engine,
|
||||
"SELECT COALESCE(SUM(amount), 0) FROM core_trade "
|
||||
"WHERE customer_id = :c AND trade_type IN ('subscribe','redeem') "
|
||||
"AND trade_status = 'confirmed' "
|
||||
"AND traded_at >= :s AND traded_at < :e "
|
||||
"AND (convert_group_id IS NULL OR trade_type = 'redeem')",
|
||||
c=CUSTOMER,
|
||||
s=day_start,
|
||||
e=day_start + timedelta(days=1),
|
||||
)
|
||||
check("只判 IS NULL 的结果（少算空串那笔）", dec(null_only), SUM_IF_NULL_ONLY)
|
||||
check("与正确口径差额 = 空串那笔金额", dec(actual_sum) - dec(null_only), Decimal(AMT_EMPTY))
|
||||
|
||||
print("\n[D] core_tools.query_recent_trades 汇总去重（FR-C15）")
|
||||
res = query_recent_trades(CUSTOMER, days=1, core_ro=core_ro)
|
||||
check("明细保持全量（4 条）", res["total_count"], 4)
|
||||
check("sum_amount 不翻倍", dec(res["sum_amount"]), EXPECTED_SUM)
|
||||
check("跨口径一致：Tool 汇总 == 仓储 SQL 汇总", dec(res["sum_amount"]), dec(actual_sum))
|
||||
|
||||
print("\n[E] core_tools.query_holdings 归零行过滤")
|
||||
holdings = query_holdings(CUSTOMER, core_ro=core_ro)
|
||||
check("持仓条数（归零行被排除）", holdings["total_count"], 1)
|
||||
check("返回的是正常持仓产品", [r["product_id"] for r in holdings["items"]], [PROD_A])
|
||||
check("合计市值不含归零行", dec(holdings["sum_market_value"]), Decimal("1200"))
|
||||
|
||||
print("\n[F] 底表复核：归零行确实还在库里（过滤发生在查询侧，不是删除）")
|
||||
zero_rows = q1(
|
||||
engine,
|
||||
"SELECT COUNT(*) FROM core_holding WHERE customer_id = :c AND qty <= 0",
|
||||
c=CUSTOMER,
|
||||
)
|
||||
check("core_holding 归零行仍保留", zero_rows, 1)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
now = datetime.now()
|
||||
engine = get_engine(settings.mysql_core_database, "admin")
|
||||
print("T-11 真库验证：金额汇总去重（FR-C15 / R-d）+ 归零行过滤")
|
||||
print(f"客户={CUSTOMER} 产品={PROD_A}/{PROD_B} 交易日={now.date()}")
|
||||
cleanup(engine)
|
||||
try:
|
||||
seed(engine, now)
|
||||
run_checks(engine, now)
|
||||
finally:
|
||||
cleanup(engine)
|
||||
left = q1(
|
||||
engine,
|
||||
"SELECT (SELECT COUNT(*) FROM core_trade WHERE customer_id LIKE 'CUST-TOOLT%') "
|
||||
"+ (SELECT COUNT(*) FROM core_holding WHERE customer_id LIKE 'CUST-TOOLT%') "
|
||||
"+ (SELECT COUNT(*) FROM core_product WHERE product_id LIKE 'PROD-TOOLT%') "
|
||||
"+ (SELECT COUNT(*) FROM core_customer WHERE customer_id LIKE 'CUST-TOOLT%')",
|
||||
)
|
||||
print(f"\n清理后残留行数：{left}")
|
||||
dispose_engines()
|
||||
|
||||
print(f"\n结果：{_passed} 通过 / {_failed} 失败")
|
||||
return 1 if _failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user