feat(data): 取真实费率并回填 fin_product；发现 510300 的身份问题
## 取费率
项目原有的行情适配器只有 names/quotes/history，没有费率。东财也没有可用的 JSON 接口
（`FundArchivesDatas.aspx?type=jjfl` 实测正文只有一句 `var apidata=`），
所以给 EastmoneyFundAdapter 加了 fetch_fees：抓 f10 基金概况页
（fundf10.eastmoney.com/jbgk_{code}.html）去标签后匹配字段名，取管理费/托管费/
销售服务费/申赎费，并顺带带回基金全称。取不到的基金不进返回（而不是给全 None），
免得调用方把"没抓到"当成"该基金确实没有费率"。
实测 20/20 全部取到。
## 回填
新增 tools/backfill_product_fees.py，默认 dry-run、默认只补 NULL：
- 只补空值：写入 38 个字段（19 只 × 管理费/托管费），已执行
- 已有值一律不动 —— `510300` 库里的 0.5000/0.1000 保持原样，要改需显式 --overwrite
- 写入走单事务
回填前 grep 确认：`fin_product.management_fee_rate` / `custodian_fee_rate`
**目前没有任何业务代码读取**（advisor 域那两个 fee 字段属于另一张表
advisor_product_contract），所以这次回填不改变任何现有行为。
要让客户真的看到费率，还需要接口与前端读它 —— 那不在本次范围。
## 两个数据问题（都指向 510300）
1) **它不是南方基金的产品**。数据源返回的基金全称是
「华泰柏瑞沪深300交易型开放式指数证券投资基金」，而库里 fund_manager 写「南方基金」。
其余 19 只全称都是"南方…"开头，只有它例外。
影响面不只是知识库：MarketQuoteSyncService 按 fund_manager='南方基金' 筛选同步对象，
所以它会被当成自家产品一起同步、一起展示。
2) **它的费率是照抄的**。库里 0.5000/0.1000，真实 0.15/0.05；而 0.50/0.10 恰好是
159329/159382/159511/588890 这四只的真实费率。结合它的净值快照时间是当天
（其余 19 只停在 09-11）、净值取整数 4.500000，判断是演示用途的人为设定。
这两条都由用户决定怎么处理，本次只记录、不擅自改（510300 的 fund_manager、净值、费率
三处原值都没动）。
## 其他
- tools/fetch_live_quotes.py 增加费率核对段落与 --no-fees（费率要串行抓 45KB/只，较慢）
- docs/42 用真实费率重写 1.2 节与新增 4.3 真实费率清单，3.1 节标出 510300 的身份问题
- docs/42 的数据缺口表更新：费率不再是缺口
门禁：ruff 通过 / mypy 249 文件 0 错 / 单元+契约 1377 passed 2 skipped。
This commit is contained in:
@@ -0,0 +1,144 @@
|
||||
"""把真实费率回填到 `fin_product`（默认 dry-run，只有 `--apply` 才写库）。
|
||||
|
||||
## 为什么需要它
|
||||
|
||||
`fin_product.management_fee_rate` / `custodian_fee_rate` 在本机 **20 只里 19 只为空**，
|
||||
客户问"管理费多少"只能转人工。费率可以从东方财富 f10 基金概况页取到
|
||||
（`EastmoneyFundAdapter.fetch_fees`），本脚本负责把它落到库里。
|
||||
|
||||
## 安全设计
|
||||
|
||||
1. **默认 dry-run**：只打印将执行的 UPDATE，不写库。要真正写入必须显式给 `--apply`。
|
||||
2. **默认只补空值**：`NULL` 才写，**已有值一律不动** —— 因为已有值可能是人为设定
|
||||
（本机 `510300` 就是：库里 0.5000/0.1000，真实 0.15/0.05）。要覆盖必须再给 `--overwrite`。
|
||||
3. **`--overwrite` 也只改这一只这一类**：它不会顺手把别的东西一起改掉。
|
||||
4. 写入走单个事务，任一失败整体回滚。
|
||||
|
||||
## 用法
|
||||
|
||||
python tools/backfill_product_fees.py # 看将要改什么
|
||||
python tools/backfill_product_fees.py --apply # 只补空值
|
||||
python tools/backfill_product_fees.py --apply --overwrite # 连已有值一起改成真实值
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import sys
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from app.infrastructure.db import SessionFactory # noqa: E402
|
||||
from app.infrastructure.fund_market_adapter import EastmoneyFundAdapter # noqa: E402
|
||||
|
||||
if hasattr(sys.stdout, "reconfigure"):
|
||||
sys.stdout.reconfigure(errors="replace") # type: ignore[union-attr]
|
||||
|
||||
#: 与 `MarketQuoteSyncService.sync` 同一批产品。
|
||||
PRODUCT_QUERY = """
|
||||
SELECT id, product_code, product_name, management_fee_rate, custodian_fee_rate
|
||||
FROM fin_product
|
||||
WHERE fund_manager = '南方基金'
|
||||
AND exchange_code IN ('SSE', 'SZSE')
|
||||
AND status = '上市'
|
||||
ORDER BY product_code
|
||||
"""
|
||||
|
||||
#: 库字段 ↔ 数据源字段。
|
||||
FEE_FIELDS = (
|
||||
("management_fee_rate", "management_fee_rate"),
|
||||
("custodian_fee_rate", "custodian_fee_rate"),
|
||||
)
|
||||
|
||||
|
||||
def _same(left: object, right: object) -> bool:
|
||||
if left is None or right is None:
|
||||
return False
|
||||
try:
|
||||
return abs(float(str(left)) - float(str(right))) < 1e-9
|
||||
except (TypeError, ValueError):
|
||||
return str(left) == str(right)
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="把真实费率回填到 fin_product")
|
||||
parser.add_argument("--apply", action="store_true", help="真正写库（默认只看不改）")
|
||||
parser.add_argument(
|
||||
"--overwrite", action="store_true",
|
||||
help="连已有值也改（默认只补 NULL，不动人为设定过的值）",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
async with SessionFactory() as session:
|
||||
rows = (await session.execute(text(PRODUCT_QUERY))).mappings().all()
|
||||
|
||||
adapter = EastmoneyFundAdapter()
|
||||
fees = await adapter.fetch_fees([str(row["product_code"]) for row in rows])
|
||||
print(f"产品 {len(rows)} 只；取到真实费率 {len(fees)} 只\n")
|
||||
|
||||
updates: list[tuple[int, str, str, object]] = []
|
||||
conflicts: list[str] = []
|
||||
for row in rows:
|
||||
code = str(row["product_code"])
|
||||
live = fees.get(code)
|
||||
if not live:
|
||||
continue
|
||||
for column, key in FEE_FIELDS:
|
||||
live_value = live.get(key)
|
||||
if live_value is None:
|
||||
continue
|
||||
stored = row[column]
|
||||
if stored is not None and not _same(stored, live_value):
|
||||
conflicts.append(
|
||||
f"{code} {row['product_name']} 的 {column}：库里 {stored} → 真实 {live_value}"
|
||||
)
|
||||
if not args.overwrite:
|
||||
continue
|
||||
if stored is not None and not args.overwrite:
|
||||
continue
|
||||
if _same(stored, live_value):
|
||||
continue
|
||||
updates.append((int(row["id"]), code, column, live_value))
|
||||
|
||||
if conflicts:
|
||||
print("与库里已有值冲突（默认不动，需要 --overwrite 才覆盖）：")
|
||||
for line in conflicts:
|
||||
print(f" * {line}")
|
||||
print()
|
||||
|
||||
if not updates:
|
||||
print("没有需要回填的字段。")
|
||||
return 0
|
||||
|
||||
print(f"将执行 {len(updates)} 条 UPDATE：")
|
||||
for _, code, column, value in updates:
|
||||
print(f" · {code} {column} = {value}")
|
||||
|
||||
if not args.apply:
|
||||
print("\n[dry-run] 未写库。加 --apply 执行。")
|
||||
return 0
|
||||
|
||||
now = datetime.now(UTC).replace(tzinfo=None)
|
||||
async with SessionFactory() as session, session.begin():
|
||||
for product_id, _, column, value in updates:
|
||||
# 列名来自本模块常量，不是外部输入；值走参数绑定。
|
||||
await session.execute(
|
||||
text(
|
||||
f"UPDATE fin_product SET {column} = :value, updated_at = :now "
|
||||
"WHERE id = :product_id"
|
||||
),
|
||||
{"value": value, "now": now, "product_id": product_id},
|
||||
)
|
||||
print(f"\n已写入 {len(updates)} 个字段。")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(asyncio.run(main()))
|
||||
+123
-42
@@ -1,27 +1,34 @@
|
||||
"""拉取场内基金的真实行情并核对库里的快照。
|
||||
"""拉取场内基金的真实行情与费率，并核对库里的快照。
|
||||
|
||||
## 数据源
|
||||
|
||||
走项目自带的 `EastmoneyFundAdapter`（`app/infrastructure/fund_market_adapter.py`），
|
||||
不自己拼 HTTP、不引第三方行情库：
|
||||
|
||||
· 实时/收盘价 `https://push2.eastmoney.com/api/qt/ulist.np/get`
|
||||
· 历史净值 `https://api.fund.eastmoney.com/f10/lsjz`
|
||||
· 历史净值 `api.fund.eastmoney.com/f10/lsjz`
|
||||
· 费率 `fundf10.eastmoney.com/jbgk_{code}.html`（管理费/托管费/销售服务费/申赎费）
|
||||
· 实时行情 `push2.eastmoney.com/api/qt/ulist.np/get`（见适配器 `fetch_quotes`）
|
||||
|
||||
同一套适配器也是下单链路（`TradeService` 取 `quote_price`/`quote_source`）
|
||||
与 `MarketQuoteSyncService`（东财 + 腾讯双源）在用的，所以这里拿到的值与成交价同源。
|
||||
|
||||
## 用法
|
||||
|
||||
python tools/fetch_live_quotes.py # 与库里快照逐只对比
|
||||
python tools/fetch_live_quotes.py --json # 只输出 JSON（供生成知识条目等）
|
||||
python tools/fetch_live_quotes.py # 净值 + 费率，与库里逐只对比
|
||||
python tools/fetch_live_quotes.py --no-fees # 跳过费率（费率要串行抓 45KB/只，慢）
|
||||
python tools/fetch_live_quotes.py --json # 输出 JSON（供生成知识条目等）
|
||||
|
||||
## 为什么要做对比而不是直接覆盖
|
||||
|
||||
`fin_product.current_nav` 是**快照缓存**，正常情况下与数据源一致（本机 20 只里 18 只
|
||||
完全一致，只是小数位表示不同）。**不一致的那两只才是重点**：它们要么被人为改过
|
||||
（演示需要），要么字段口径本身有问题 —— 直接覆盖会把线索一起抹掉。
|
||||
所以本脚本只**报告**差异，不写库；要不要刷新由 `ProductHistorySyncService` 决定。
|
||||
`fin_product.current_nav` / `management_fee_rate` 都是**快照缓存**。多数情况下与数据源
|
||||
一致，**不一致的那几只才是重点**：它们要么被人为改过（演示需要），要么字段本身填错。
|
||||
直接覆盖会把线索一起抹掉。所以本脚本只**报告**差异、不写库
|
||||
（刷新快照属于 `ProductHistorySyncService` / `MarketQuoteSyncService` 的职责）。
|
||||
|
||||
本机实测的两处不一致（2026-09-13），都是这么发现的：
|
||||
· `510300` 库里净值 4.500000、真实 4.5794（且库里快照时间是当天，其余 19 只停在 09-11）
|
||||
· `510300` 库里管理费 0.5000 / 托管费 0.1000、真实 0.15 / 0.05
|
||||
—— 而 `0.50/0.10` 恰好是另外几只 ETF 的真实值，像是照抄过来的
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -49,7 +56,7 @@ if hasattr(sys.stdout, "reconfigure"):
|
||||
PRODUCT_QUERY = """
|
||||
SELECT product_code, product_name, exchange_code, product_category, risk_level,
|
||||
lot_size, min_amount, current_nav, current_nav_at,
|
||||
management_fee_rate, custodian_fee_rate
|
||||
management_fee_rate, custodian_fee_rate, transaction_fee_rate
|
||||
FROM fin_product
|
||||
WHERE fund_manager = '南方基金'
|
||||
AND exchange_code IN ('SSE', 'SZSE')
|
||||
@@ -57,8 +64,28 @@ PRODUCT_QUERY = """
|
||||
ORDER BY product_code
|
||||
"""
|
||||
|
||||
#: 库里已有值的费率字段 ↔ 数据源字段（用于逐项核对）。
|
||||
FEE_STORED_TO_LIVE = {
|
||||
"management_fee_rate": "management_fee_rate",
|
||||
"custodian_fee_rate": "custodian_fee_rate",
|
||||
}
|
||||
|
||||
async def collect() -> list[dict[str, object]]:
|
||||
|
||||
def _text(value: object) -> str | None:
|
||||
return None if value is None else str(value)
|
||||
|
||||
|
||||
def _same(left: object, right: object) -> bool:
|
||||
"""比较两个费率/净值文本，容忍 `0.5` 与 `0.5000` 这种表示差异。"""
|
||||
if left is None or right is None:
|
||||
return False
|
||||
try:
|
||||
return abs(float(str(left)) - float(str(right))) < 1e-9
|
||||
except (TypeError, ValueError):
|
||||
return str(left) == str(right)
|
||||
|
||||
|
||||
async def collect(*, with_fees: bool = True) -> list[dict[str, object]]:
|
||||
async with SessionFactory() as session:
|
||||
rows = (await session.execute(text(PRODUCT_QUERY))).mappings().all()
|
||||
|
||||
@@ -68,41 +95,44 @@ async def collect() -> list[dict[str, object]]:
|
||||
*(adapter.fetch_history(code, date.today()) for code in codes),
|
||||
return_exceptions=True,
|
||||
)
|
||||
# 费率走 f10 概况页：约 45KB/只、且适配器内部是串行抓取（不对公开接口并发施压），
|
||||
# 因此只在需要时拉 —— `--no-fees` 可跳过。
|
||||
fees: dict[str, dict[str, object]] = {}
|
||||
if with_fees:
|
||||
fees = await adapter.fetch_fees(codes)
|
||||
|
||||
records: list[dict[str, object]] = []
|
||||
for row, history in zip(rows, histories, strict=True):
|
||||
code = str(row["product_code"])
|
||||
if isinstance(history, BaseException):
|
||||
history = {"degraded": True, "error": type(history).__name__}
|
||||
live_nav = history.get("nav")
|
||||
stored_nav = row["current_nav"]
|
||||
records.append({
|
||||
"product_code": str(row["product_code"]),
|
||||
live_fee = fees.get(code) or {}
|
||||
record: dict[str, object] = {
|
||||
"product_code": code,
|
||||
"product_name": str(row["product_name"]),
|
||||
"exchange_code": str(row["exchange_code"]),
|
||||
"product_category": str(row["product_category"]),
|
||||
"risk_level": str(row["risk_level"]),
|
||||
"lot_size": str(row["lot_size"]),
|
||||
"min_amount": str(row["min_amount"]),
|
||||
"management_fee_rate": (
|
||||
None if row["management_fee_rate"] is None else str(row["management_fee_rate"])
|
||||
),
|
||||
"custodian_fee_rate": (
|
||||
None if row["custodian_fee_rate"] is None else str(row["custodian_fee_rate"])
|
||||
),
|
||||
"live_nav": None if live_nav is None else str(live_nav),
|
||||
"live_nav_date": (
|
||||
None if history.get("nav_date") is None else str(history["nav_date"])
|
||||
),
|
||||
"live_change_pct": (
|
||||
None if history.get("daily_change") is None else str(history["daily_change"])
|
||||
),
|
||||
"live_nav": _text(live_nav),
|
||||
"live_nav_date": _text(history.get("nav_date")),
|
||||
"live_change_pct": _text(history.get("daily_change")),
|
||||
"stored_nav": str(stored_nav),
|
||||
"stored_nav_at": str(row["current_nav_at"]),
|
||||
"matches_stored": (
|
||||
live_nav is not None and float(live_nav) == float(stored_nav)
|
||||
),
|
||||
"matches_stored": _same(live_nav, stored_nav),
|
||||
"degraded": bool(history.get("degraded")),
|
||||
})
|
||||
}
|
||||
for stored_field, live_field in FEE_STORED_TO_LIVE.items():
|
||||
record[f"stored_{stored_field}"] = _text(row[stored_field])
|
||||
record[f"live_{live_field}"] = _text(live_fee.get(live_field))
|
||||
record["live_service_fee_rate"] = _text(live_fee.get("service_fee_rate"))
|
||||
record["live_subscribe_fee_rate"] = _text(live_fee.get("subscribe_fee_rate"))
|
||||
record["live_redeem_fee_rate"] = _text(live_fee.get("redeem_fee_rate"))
|
||||
record["live_full_name"] = _text(live_fee.get("full_name"))
|
||||
records.append(record)
|
||||
return records
|
||||
|
||||
|
||||
@@ -110,9 +140,13 @@ def render(records: list[dict[str, object]]) -> None:
|
||||
ok = [item for item in records if not item["degraded"]]
|
||||
same = [item for item in ok if item["matches_stored"]]
|
||||
diff = [item for item in ok if not item["matches_stored"]]
|
||||
print("=" * 100)
|
||||
print("净值核对（数据源 vs 库里快照）")
|
||||
print("=" * 100)
|
||||
print(f"场内产品 {len(records)} 只；取到真实行情 {len(ok)} 只"
|
||||
f"（与库里一致 {len(same)}、不一致 {len(diff)}）\n")
|
||||
header = f"{'代码':<8}{'名称':<26}{'真实净值':>10}{'涨跌%':>8}{'净值日期':>12} {'库里净值':>11} 核对"
|
||||
header = (f"{'代码':<8}{'名称':<26}{'真实净值':>10}{'涨跌%':>8}{'净值日期':>12}"
|
||||
f" {'库里净值':>11} 核对")
|
||||
print(header)
|
||||
print("-" * len(header))
|
||||
for item in records:
|
||||
@@ -128,24 +162,71 @@ def render(records: list[dict[str, object]]) -> None:
|
||||
f"{str(item['live_nav_date']):>12} {str(item['stored_nav']):>11} {verdict}"
|
||||
)
|
||||
|
||||
if diff:
|
||||
print("\n不一致的条目（需要人工判断，不要直接覆盖）：")
|
||||
for item in diff:
|
||||
print(f" · {item['product_code']} {item['product_name']}")
|
||||
print(f" 真实 {item['live_nav']}（{item['live_nav_date']}）"
|
||||
f" vs 库里 {item['stored_nav']}（快照于 {item['stored_nav_at']}）")
|
||||
fee_rows = [item for item in records if item.get("live_management_fee_rate") is not None]
|
||||
if fee_rows:
|
||||
print()
|
||||
print("=" * 100)
|
||||
print("费率核对（数据源 vs 库里字段）")
|
||||
print("=" * 100)
|
||||
fee_header = (f"{'代码':<8}{'名称':<24}{'真实管理费':>11}{'库里':>8}"
|
||||
f"{'真实托管费':>11}{'库里':>8} 核对")
|
||||
print(fee_header)
|
||||
print("-" * len(fee_header))
|
||||
fee_diff = []
|
||||
for item in fee_rows:
|
||||
live_manage = item["live_management_fee_rate"]
|
||||
stored_manage = item["stored_management_fee_rate"]
|
||||
if stored_manage is None:
|
||||
verdict = "库里为空"
|
||||
elif _same(live_manage, stored_manage) and _same(
|
||||
item["live_custodian_fee_rate"], item["stored_custodian_fee_rate"]
|
||||
):
|
||||
verdict = "一致"
|
||||
else:
|
||||
verdict = "**不一致**"
|
||||
fee_diff.append(item)
|
||||
print(
|
||||
f"{str(item['product_code']):<8}{str(item['product_name'])[:22]:<24}"
|
||||
f"{str(live_manage):>11}{str(stored_manage):>8}"
|
||||
f"{str(item['live_custodian_fee_rate']):>11}"
|
||||
f"{str(item['stored_custodian_fee_rate']):>8} {verdict}"
|
||||
)
|
||||
empty = [item for item in fee_rows if item["stored_management_fee_rate"] is None]
|
||||
print(f"\n真实费率全部取到 {len(fee_rows)} 只；库里为空 {len(empty)} 只、"
|
||||
f"与真实值冲突 {len(fee_diff)} 只")
|
||||
|
||||
missing = [item["product_code"] for item in records if item["degraded"]]
|
||||
if missing:
|
||||
print(f"\n取数失败：{missing}")
|
||||
if diff or fee_diff:
|
||||
print()
|
||||
print("需要人工判断的条目（不要直接覆盖）：")
|
||||
for item in diff:
|
||||
print(f" · 净值 {item['product_code']} {item['product_name']}："
|
||||
f"真实 {item['live_nav']}（{item['live_nav_date']}） vs "
|
||||
f"库里 {item['stored_nav']}（{item['stored_nav_at']}）")
|
||||
for item in fee_diff:
|
||||
print(f" · 费率 {item['product_code']} {item['product_name']}："
|
||||
f"真实 管理 {item['live_management_fee_rate']} / 托管 "
|
||||
f"{item['live_custodian_fee_rate']} vs 库里 管理 "
|
||||
f"{item['stored_management_fee_rate']} / 托管 "
|
||||
f"{item['stored_custodian_fee_rate']}")
|
||||
|
||||
full_names = [
|
||||
(str(item["product_code"]), item["live_full_name"])
|
||||
for item in records if item.get("live_full_name")
|
||||
]
|
||||
if full_names:
|
||||
print()
|
||||
print("数据源返回的基金全称（可用于核对 fund_manager 标注是否正确）：")
|
||||
for code, name in full_names:
|
||||
print(f" · {code} {name}")
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="拉取场内基金真实行情并核对库里快照")
|
||||
parser = argparse.ArgumentParser(description="拉取场内基金真实行情与费率，核对库里快照")
|
||||
parser.add_argument("--json", action="store_true", help="只输出 JSON")
|
||||
parser.add_argument("--no-fees", action="store_true", help="跳过费率抓取（较快）")
|
||||
args = parser.parse_args()
|
||||
|
||||
records = await collect()
|
||||
records = await collect(with_fees=not args.no_fees)
|
||||
if args.json:
|
||||
print(json.dumps(records, ensure_ascii=False, indent=1))
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user