场外运营与资料处理
运营工作台
集中查看收件箱、识别任务与规则状态,所有场外运营流程与场内模拟交易数据严格隔离。
diff --git a/README.md b/README.md new file mode 100644 index 0000000..7597fdf --- /dev/null +++ b/README.md @@ -0,0 +1,486 @@ +# 南方基金智能业务平台 + +本项目是一个基于 FastAPI 和 MVC+S 架构的金融业务 Agent 平台,当前包含以下三项核心业务: + +1. 场外基金申购/赎回运营流程 +2. 产品推介材料与宣传海报生成 +3. 金融自然语言转 SQL(NL2SQL)查询 + +项目中的场外基金运营、产品推介材料与场内基金模拟交易相互隔离。场外业务使用独立的 +`offsite_fund_*` 数据表和接口,不写入场内交易表。 + +## 一、技术栈 + +- Python `3.13` +- FastAPI、Uvicorn +- SQLAlchemy 2.x、Alembic +- MySQL、Redis +- Pydantic v2 +- JWT + RBAC +- `python-pptx`、OpenPyXL、Pillow、Matplotlib +- 可选:阿里云 DocMind/OCR、DeepSeek、SMTP、Milvus、Neo4j + +## 二、项目结构 + +```text +group_fqcd_jr/ +├── app/ +│ ├── api/controllers/ # HTTP 路由和请求参数校验 +│ ├── api/schemas/ # 请求/响应模型 +│ ├── core/ # 业务契约、配置和公共规则 +│ ├── model/ # SQLAlchemy 数据模型 +│ ├── service/ # 业务编排、规则和数据访问 +│ ├── infrastructure/ # 数据库、缓存和外部基础设施适配 +│ └── worker/ # 异步 Worker +├── data/ # 测试与演示数据集 +├── docs/ # 架构、接口、数据库和业务说明 +├── tests/ # 单元、契约、集成和环境测试 +├── tools/ # 迁移、种子、检查和测试工具 +├── alembic/ # 数据库迁移 +├── hq.py # 南方基金行情数据适配模块 +├── nl2sql_yc.py # 金融 NL2SQL 兼容入口和查询逻辑 +├── .env.example # 环境变量模板 +└── README.md +``` + +## 三、环境准备 + +项目要求 Python `>=3.13,<3.14`。建议使用独立虚拟环境: + +```powershell +conda create -n jr_py313 python=3.13 +conda activate jr_py313 +pip install -r requirements.txt +``` + +复制配置文件: + +```powershell +Copy-Item .env.example .env +``` + +然后根据本机环境修改 `.env`,至少配置: + +```text +MYSQL_DSN +JWT_PRIVATE_KEY_PATH +JWT_PUBLIC_KEY_PATH +``` + +`.env`、JWT 私钥、模型密钥和真实外部服务凭据禁止提交到 Git。 + +## 四、启动服务 + +### 4.1 数据库迁移 + +请确保 MySQL 已启动,并且 `.env` 中的数据库连接可用: + +```powershell +alembic upgrade head +python tools/audit_schema.py +``` + +### 4.2 启动 HTTP 服务 + +```powershell +python -m uvicorn app.main:app --host 127.0.0.1 --port 8099 +``` + +服务地址: + +```text +http://127.0.0.1:8099 +``` + +OpenAPI 文档: + +```text +http://127.0.0.1:8099/docs +``` + +### 4.3 启动 Worker + +需要异步处理 Agent 运行或场外邮件时,另开终端: + +```powershell +python -m app.worker +``` + +Worker 是否启用场外收信,取决于: + +```text +OFFSITE_MAIL_WORKER_ENABLED +OFFSITE_IMAP_ENABLED +OFFSITE_WORKER_USER_ID +``` + +## 五、功能一:场外基金申购/赎回 + +### 5.1 功能说明 + +场外流程面向运营人员,主要处理: + +```text +邮件接收 +→ 附件保存 +→ OCR/文档识别 +→ 结构化字段提取 +→ 申购/赎回规则计算 +→ NL2SQL 查询资金和持仓数据 +→ 运营人员确认 +→ 创建通知 +→ 发送或 dry-run +→ 审计和统计 +``` + +支持的单据类型: + +- `subscription`:申购 +- `redemption`:赎回 +- `summary`:汇总材料 +- `other`:其他材料 + +系统保留原始邮件、附件、识别结果、查询记录、规则结果、人工修正记录和审计记录。 +人工修正不会覆盖 Agent 原始识别值。 + +### 5.2 主要接口 + +基础路径: + +```text +/api/v1/offsite-fund +``` + +| 方法 | 路径 | 用途 | +|---|---|---| +| `GET` | `/mails` | 分页查询场外邮件 | +| `GET` | `/mails/{mail_id}` | 查看邮件详情 | +| `POST` | `/mails/{mail_id}/deletions` | 软删除邮件 | +| `GET` | `/mails/{mail_id}/recognition-fields` | 查看 OCR 识别字段 | +| `PUT` | `/mails/{mail_id}/recognition-fields` | 保存 OCR 字段人工修正 | +| `GET` | `/documents/{task_id}/nl2sql-fields` | 查看 NL2SQL 返回字段 | +| `PUT` | `/documents/{task_id}/nl2sql-fields` | 保存 NL2SQL 字段人工修正 | +| `GET` | `/documents/{task_id}/rule-results` | 查看规则判定结果 | +| `POST` | `/documents/{task_id}/rule-results/recalculations` | 使用已有结果重新判定 | +| `GET` | `/mailbox-status` | 查看收件游标状态 | +| `POST` | `/mailbox-status/recoveries` | 恢复被阻塞的收件游标 | +| `GET` | `/attachments/{attachment_id}/file` | 预览或下载原始附件 | +| `POST` | `/documents/{task_id}/confirmations` | 运营人员确认单据 | +| `POST` | `/documents/{task_id}/recognition-retries` | 重试识别异常单据 | +| `POST` | `/documents/{task_id}/notifications` | 创建运营通知 | +| `POST` | `/notifications/{notification_id}/send` | 发送通知 | +| `POST` | `/settlement-statistics/recalculate` | 重算结算统计 | + +触发场外单据 NL2SQL 核对的接口为: + +```text +POST /api/tasks/{task_id}/trigger-agent-nl2sql +``` + +### 5.3 权限与安全 + +- 读取权限:`offsite:read` +- 写入权限:`offsite:write` +- 确认权限:`offsite:confirm` +- NL2SQL 权限:`offsite:nl2sql` +- 通知权限:`offsite:notify` +- 请求中的 `operator_id` 必须与 JWT 身份一致 +- 所有写操作都进行权限校验、操作人校验、审计和状态流转校验 +- 通知发送支持幂等,SMTP 默认关闭并使用 dry-run + +## 六、功能二:产品推介材料与宣传海报生成 + +### 6.1 功能说明 + +该模块用于新产品的结构化资料整理和宣传材料生成。当前生成链路为: + +```text +创建任务 +→ 填写产品、经理、团队、策略、费用、业绩和风险资料 +→ 上传经理照片、业绩数据、来源证据或模板 +→ 合规检查 +→ 生成 PPTX、宣传海报和可选 PDF +→ 审核 +→ 发送给指定投顾 +``` + +支持的输入附件: + +- `manager_photo`:基金经理照片 +- `performance_data`:CSV/XLSX 业绩数据 +- `source_evidence`:来源证据 +- `template_file`:材料模板 + +支持的输出格式: + +- `pptx` +- `poster` +- `pdf`,需要配置 LibreOffice/`soffice` 等转换器 + +模块会执行风险披露、费用结构、业绩数据、输入完整性和生成文本合规检查。 +合规阻断时返回具体 findings 和补齐建议,不继续生成材料。 + +### 6.2 主要接口 + +基础路径: + +```text +/api/v1/fund-promotion-materials +``` + +| 方法 | 路径 | 用途 | +|---|---|---| +| `POST` | `/` | 创建推介材料任务 | +| `PUT` | `/{task_no}/inputs` | 保存结构化输入资料 | +| `POST` | `/{task_no}/attachments` | 上传照片、业绩文件或证据 | +| `POST` | `/{task_no}/generations` | 生成推介材料 | +| `GET` | `/{task_no}/compliance-checks` | 查看合规检查结果 | +| `GET` | `/{task_no}` | 查询任务和已审核材料 | +| `POST` | `/{task_no}/reviews` | 审核、驳回或要求修改 | +| `POST` | `/{task_no}/deliveries` | 发送给投顾 | + +创建任务和写入操作必须提供请求头: + +```text +Idempotency-Key: <唯一请求键> +``` + +### 6.3 权限与状态 + +主要权限: + +- `promotion:write` +- `promotion:read` +- `promotion:review` +- `promotion:deliver` + +材料任务状态包括: + +```text +draft +→ input_ready +→ generating +→ generated +→ pending_review +→ approved +→ sent +``` + +异常状态包括: + +```text +compliance_failed +rejected +failed +``` + +生成的文件默认保存到: + +```text +data/promotion_materials/ +``` + +## 七、功能三:金融 NL2SQL + +### 7.1 功能说明 + +NL2SQL 将自然语言问题转换为经过权限、业务域、表白名单和只读校验的 SQL。 +业务 Agent 不直接连接数据库,而是通过公共只读工具调用: + +```text +业务 Agent +→ BaseAgent.call_tool +→ ToolExecutor +→ query_financial_data +→ 只读金融数据表 +``` + +工具名称: + +```text +query_financial_data +``` + +权限码: + +```text +financial:nl2sql:read +``` + +### 7.2 支持范围 + +当前纳入查询范围的主要表: + +```text +sys_customer_assignment +fin_customer_profile +fin_risk_assessment +fin_product +fin_fee_rule +fin_market_price +fin_nav_history +fin_holding +fin_transaction +fin_sim_order +fin_sim_account +fin_cash_ledger +client_facing_content +``` + +支持: + +- 基金净值、行情和历史收益查询 +- 费率和费用查询 +- 客户风险测评和画像查询 +- 持仓、交易、订单、账户和资金流水查询 +- 历史范围查询 +- 最多跨三个业务域查询 +- 结果行数限制,默认 50,最大 200 +- SQL dry-run + +不支持或会被拒绝: + +- 非白名单表 +- 非 `SELECT` 查询 +- 未通过客户数据范围校验的查询 +- 当前快照表的历史时点查询 +- 模糊且未确认查询范围、指标口径或时间条件的问题 + +### 7.3 调用示例 + +```python +result = await self.call_tool( + "query_financial_data", + { + "question": "查询基金代码 000001 最近 30 天的净值", + "dry_run": False, + "limit": 50, + }, + intent="financial_query", + context=context, +) +``` + +模糊问题会返回: + +```json +{ + "status": "need_confirmation", + "message": "请确认查询范围、指标口径和时间条件。" +} +``` + +工具结果包含查询计划、SQL、参数、权限检查和执行摘要,并进入统一审计和运行持久化链路。 + +## 八、测试数据集 + +测试数据位于: + +```text +data/ +``` + +当前数据集概况: + +| 类型 | 数量 | 用途 | +|---|---:|---| +| `.eml` | 13 | 场外邮件及附件识别 | +| `.csv` / `.xlsx` | 8 | 推介材料业绩数据 | +| `.png` / `.jpg` | 35 | 经理照片、宣传图和测试图片 | +| `.pptx` | 10 | 推介材料结构检查 | +| `.pdf` | 9 | PDF 文件有效性检查 | + +测试数据只用于本地测试和演示,不应直接作为生产业务数据导入。 + +## 九、测试与代码检查 + +### 9.1 运行完整测试 + +普通单元和契约测试: + +```powershell +python -m pytest tests/unit tests/contract -q -p no:cacheprovider +``` + +使用独立 MySQL 测试库并执行测试数据隔离: + +```powershell +python -m tools.run_tests_on_test_db tests -p no:cacheprovider +``` + +### 9.2 运行三项功能定向测试 + +```powershell +python -m pytest ` + tests/integration/test_offsite_fund_api.py ` + tests/integration/test_offsite_nl2sql_fields.py ` + tests/integration/test_offsite_notification_send.py ` + tests/integration/test_promotion_material_api.py ` + tests/unit/service/test_suitability_service.py ` + tests/contract/test_financial_nl2sql_tool_contract.py ` + -q -p no:cacheprovider +``` + +### 9.3 代码质量检查 + +```powershell +python -m ruff check app tests tools alembic +python -m mypy app +git diff --check +``` + +### 9.4 测试基线 + +最近一次项目回归记录: + +```text +关键三项功能定向测试:85 passed +全量自动化测试:559 passed, 1 skipped +``` + +Redis、真实 IMAP/OCR/DeepSeek/SMTP、Neo4j、Milvus 和浏览器 E2E 是否能执行, +取决于本机服务和凭据配置。外部服务关闭或不可用时,测试使用本地数据集、Mock 或 dry-run。 + +## 十、配置开关 + +场外邮件和外部识别: + +```text +OFFSITE_IMAP_ENABLED=false +OFFSITE_MAIL_WORKER_ENABLED=false +OFFSITE_OCR_ENABLED=false +OFFSITE_DEEPSEEK_ENABLED=false +OFFSITE_SMTP_ENABLED=false +OFFSITE_SMTP_DRY_RUN=true +``` + +产品推介材料: + +```text +PROMOTION_MATERIAL_STORAGE_DIR=data/promotion_materials +PROMOTION_PDF_ENABLED=false +PROMOTION_PDF_CONVERTER_PATH= +``` + +启用真实服务前,请先完成网络、凭据、权限、数据脱敏和回滚方案确认。 + +## 十一、安全与开发约束 + +- 所有业务请求都应经过统一 JWT、RBAC、客户范围和审计链路。 +- 不在业务 Agent 中自行读取密钥、直接连接数据库或绕过 `ToolExecutor`。 +- 场外业务不能写入场内交易表。 +- 数据库结构只能通过 Alembic 迁移修改。 +- 不删除、重命名或复用已有表和字段;历史结构不足时新增字段、新表或兼容读写。 +- 所有写接口应设计幂等键、状态机和重复提交保护。 +- 真实外部服务默认关闭,测试优先使用本地数据、Mock 和 dry-run。 + +## 十二、相关文档 + +- `docs/MVC/MVC架构.md`:项目架构和目录职责 +- `docs/00-新数据库基线设计.md`:数据库业务基线 +- `docs/05-接口文档.md`:接口权威说明 +- `docs/14-Agent组员统一接入说明书.md`:Agent 接入规范 +- `docs/15-金融NL2SQL工具接入说明.md`:NL2SQL 工具规范 +- `product_promotion_to_do_list.md`:推介材料模块开发记录 +- `测试报告/2026-09-12.md`:三项功能数据集测试报告 + diff --git a/app/api/schemas/offsite_fund.py b/app/api/schemas/offsite_fund.py index 58dd682..2a027da 100644 --- a/app/api/schemas/offsite_fund.py +++ b/app/api/schemas/offsite_fund.py @@ -15,7 +15,7 @@ class OffsiteConfirmRequest(BaseModel): class OffsiteRecalculateRequest(BaseModel): model_config = ConfigDict(extra="forbid") - fund_code: str = Field(min_length=1, max_length=32) + fund_code: str | None = Field(default=None, max_length=32) application_date: str = Field(min_length=8, max_length=32) diff --git a/app/service/offsite_document_recognition_adapter.py b/app/service/offsite_document_recognition_adapter.py index 4b3b9e2..7c157e5 100644 --- a/app/service/offsite_document_recognition_adapter.py +++ b/app/service/offsite_document_recognition_adapter.py @@ -56,6 +56,7 @@ EXTRACTED_FIELD_NAMES: tuple[str, ...] = ( "申请日期", "代销机构", "申购金额", + "币种", "金额单位", "赎回份额", "最新净值", @@ -486,7 +487,9 @@ class OffsiteDocumentRecognitionAdapter: try: body = await self._call_deepseek(source, ocr) document_type = _document_type(body.get("document_type")) - fields = _recognized_fields(body) + fields = normalize_recognition_fields( + _recognized_fields(body), ocr.ocr_text + ) confidence = _confidence_map(_mapping_dict(body, "field_confidence")) page_evidence = _mapping_dict(body, "page_evidence") or ocr.page_evidence missing_fields = tuple( @@ -539,6 +542,10 @@ class OffsiteDocumentRecognitionAdapter: "page_evidence六个顶层字段。" "extracted_fields必须是对象,字段名只能使用下面列出的中文字段;" "无法从原文确认的字段填null,不得猜测或编造。" + "申购金额只能填写数字本身,不得包含人民币、RMB、CNY、元、万元、" + "货币符号或千分位逗号;币种单独填写到币种字段,金额单位单独填写到金额单位字段。" + "例如原文“人民币5,000元”必须输出申购金额“5000”、币种“人民币”、" + "金额单位“元”。" "文档中的产品代码、产品编号、基金产品代码均映射为基金代码;" "基金代码不要求固定六位,必须按原文保留。" "document_type只能是summary、subscription、redemption、other。" @@ -547,7 +554,7 @@ class OffsiteDocumentRecognitionAdapter: '{"基金代码":null,"基金名称":null,"账户标识":null,' '"投资者名称":null,"客户标识":null,"申请编号":null,' '"申请日期":null,"代销机构":null,"申购金额":null,' - '"金额单位":null,"赎回份额":null,"最新净值":null,' + '"币种":null,"金额单位":null,"赎回份额":null,"最新净值":null,' '"基金最新总份额":null,"申请前持有份额":null,' '"当前最新可用份额":null},' '"field_confidence":{},"missing_fields":[],' @@ -718,7 +725,7 @@ def _extract_simple_fields(text: str) -> dict[str, object]: ) if value: fields[name] = value - return fields + return normalize_recognition_fields(fields, text) def _find_label_value(text: str, label: str) -> str | None: @@ -729,6 +736,97 @@ def _find_label_value(text: str, label: str) -> str | None: return matched.group(1).strip() +_CURRENCY_MARKERS: tuple[tuple[str, str], ...] = ( + ("人民币", "人民币"), + ("RMB", "人民币"), + ("CNY", "人民币"), + ("¥", "人民币"), + ("¥", "人民币"), + ("美元", "美元"), + ("USD", "美元"), + ("$", "美元"), + ("港币", "港币"), + ("HKD", "港币"), + ("欧元", "欧元"), + ("EUR", "欧元"), + ("日元", "日元"), + ("JPY", "日元"), +) + + +def normalize_recognition_fields( + fields: Mapping[str, object], source_text: str = "" +) -> dict[str, object]: + """统一识别字段格式,保证金额与币种、单位分开保存。""" + normalized = dict(fields) + raw_amount = normalized.get("申购金额") + if raw_amount is None or str(raw_amount).strip() == "": + return normalized + + amount_text = str(raw_amount).strip().replace(",", "").replace(",", "") + currency = _normalize_currency(normalized.get("币种")) + if not currency: + currency = _currency_from_text(amount_text) + if not currency: + currency = _currency_from_amount_context(source_text) + if currency: + normalized["币种"] = currency + + unit = str(normalized.get("金额单位") or "").strip() + if unit: + if unit.endswith("万元"): + unit = "万元" + elif unit.endswith("元"): + unit = "元" + normalized["金额单位"] = unit + + for marker, _ in _CURRENCY_MARKERS: + amount_text = re.sub( + rf"^{re.escape(marker)}\s*", + "", + amount_text, + flags=re.IGNORECASE, + ) + amount_text = re.sub( + rf"\s*{re.escape(marker)}$", + "", + amount_text, + flags=re.IGNORECASE, + ) + if unit: + amount_text = re.sub(rf"\s*{re.escape(unit)}$", "", amount_text) + matched = re.search(r"-?(?:\d+(?:\.\d*)?|\.\d+)", amount_text) + if matched: + normalized["申购金额"] = matched.group(0) + return normalized + + +def _normalize_currency(value: object) -> str | None: + text = str(value or "").strip() + if not text: + return None + return _currency_from_text(text) or text + + +def _currency_from_text(text: str) -> str | None: + lowered = text.lower() + for marker, currency in _CURRENCY_MARKERS: + if marker.lower() in lowered: + return currency + return None + + +def _currency_from_amount_context(source_text: str) -> str | None: + if not source_text: + return None + matched = re.search( + r"申购金额\s*[::]?\s*([^\s,,;;]+)", + source_text, + flags=re.IGNORECASE, + ) + return _currency_from_text(matched.group(1)) if matched else None + + def _extract_docx_text(payload: bytes) -> str: try: with zipfile.ZipFile(BytesIO(payload)) as archive: diff --git a/app/service/offsite_fund_rules.py b/app/service/offsite_fund_rules.py index c571d91..69f6f99 100644 --- a/app/service/offsite_fund_rules.py +++ b/app/service/offsite_fund_rules.py @@ -13,6 +13,14 @@ TWENTY_PERCENT = Decimal("0.20") ONE_YUAN = Decimal("1") +def display_decimal(value: Decimal) -> str: + """把计算结果格式化成适合页面展示的十进制文本。""" + text = format(value, "f") + if "." in text: + text = text.rstrip("0").rstrip(".") + return text or "0" + + @dataclass(frozen=True) class RuleDecision: rule_code: str @@ -88,7 +96,13 @@ class OffsiteFundRuleEngine: document_value={"申购金额元": str(amount_yuan)}, database_value={"最新净值": str(nav), "基金最新总份额": str(total_fund_shares), "申请前持有份额": str(before)}, - calculation={"本次申购份额": str(current_shares), "申购后持有比例": str(ratio)}, + calculation={ + "本次申购份额": str(current_shares), + "申购后持有比例": str(ratio), + "实际值": f"{display_decimal(ratio * HUNDRED)}%", + "规则值": "≤ 20%", + "比较": f"{display_decimal(ratio * HUNDRED)}% ≤ 20%", + }, )) limit = total_fund_shares * TEN_PERCENT decisions.append(RuleDecision( @@ -97,7 +111,13 @@ class OffsiteFundRuleEngine: result="异常" if current_shares > limit else "正常", document_value={"申购金额元": str(amount_yuan)}, database_value={"最新净值": str(nav), "基金最新总份额": str(total_fund_shares)}, - calculation={"本次申购份额": str(current_shares), "份额上限": str(limit)}, + calculation={ + "本次申购份额": str(current_shares), + "份额上限": str(limit), + "实际值": str(current_shares), + "规则值": str(limit), + "比较": f"{current_shares} ≤ {limit}", + }, )) return decisions @@ -118,7 +138,12 @@ class OffsiteFundRuleEngine: result="异常" if value > TWENTY_PERCENT else "正常", document_value={"赎回份额": str(redemption_shares)}, database_value={"产品最新总份额": str(total_fund_shares)}, - calculation={"赎回比例": str(value)}, + calculation={ + "赎回比例": str(value), + "实际值": f"{display_decimal(value * HUNDRED)}%", + "规则值": "≤ 20%", + "比较": f"{display_decimal(value * HUNDRED)}% ≤ 20%", + }, ) if redemption_shares is None or available_quantity is None: available = self._unknown("redemption_available_quantity", "账户可用份额") @@ -129,7 +154,12 @@ class OffsiteFundRuleEngine: result="异常" if redemption_shares > available_quantity else "正常", document_value={"赎回份额": str(redemption_shares)}, database_value={"当前最新可用份额": str(available_quantity)}, - calculation={"是否超出可用份额": redemption_shares > available_quantity}, + calculation={ + "是否超出可用份额": redemption_shares > available_quantity, + "实际值": str(redemption_shares), + "规则值": str(available_quantity), + "比较": f"{redemption_shares} ≤ {available_quantity}", + }, ) return [ratio, available] @@ -144,7 +174,12 @@ class OffsiteFundRuleEngine: result=result, document_value={"申购金额元": str(amount_yuan) if amount_yuan is not None else None}, database_value={}, - calculation={"判断口径": "标准化申购金额 <= 1 元为异常"}, + calculation={ + "判断口径": "标准化申购金额 <= 1 元为异常", + "实际值": str(amount_yuan) if amount_yuan is not None else None, + "规则值": "> 1 元", + "比较": f"{amount_yuan} > 1 元" if amount_yuan is not None else None, + }, ) @staticmethod diff --git a/app/service/offsite_fund_service.py b/app/service/offsite_fund_service.py index d46e4cb..7304e7d 100644 --- a/app/service/offsite_fund_service.py +++ b/app/service/offsite_fund_service.py @@ -46,6 +46,7 @@ from app.service.offsite_document_recognition_adapter import ( OffsiteDocumentRecognitionAdapter, RecognitionSourceFile, StructuredRecognitionResult, + normalize_recognition_fields, ) from app.service.offsite_fund_rules import ( OffsiteFundRuleEngine, @@ -364,7 +365,9 @@ class OffsiteFundService: correction_fields = ( dict(correction.corrected_fields) if correction is not None else {} ) - return self._merge_corrections(attachment.extracted_fields, correction_fields) + return normalize_recognition_fields( + self._merge_corrections(attachment.extracted_fields, correction_fields) + ) @staticmethod def _merge_corrections( @@ -443,8 +446,8 @@ class OffsiteFundService: "status": attachment.status, "ocr_text": attachment.ocr_text, "extracted_fields": attachment.extracted_fields, - "effective_fields": cls._merge_corrections( - attachment.extracted_fields, correction_fields + "effective_fields": normalize_recognition_fields( + cls._merge_corrections(attachment.extracted_fields, correction_fields) ), "corrections": cls._correction_payload(correction), "has_correction": correction is not None, @@ -799,8 +802,9 @@ class OffsiteFundService: "message": f"附件 {attachment_id} 不属于该邮件", "data": {}, } + normalized_fields = normalize_recognition_fields(fields) cleaned = self._cleaned_corrections( - fields, EXTRACTED_FIELD_NAMES, attachment.extracted_fields + normalized_fields, EXTRACTED_FIELD_NAMES, attachment.extracted_fields ) self.session.add( OffsiteFieldCorrection( @@ -815,8 +819,8 @@ class OffsiteFundService: created_at=now, ) ) - effective = self._merge_corrections( - attachment.extracted_fields, cleaned + effective = normalize_recognition_fields( + self._merge_corrections(attachment.extracted_fields, cleaned) ) for document in documents_by_attachment.get(attachment_id, []): self._apply_recognition_fields(document, effective) @@ -1758,6 +1762,7 @@ class OffsiteFundService: def _apply_recognition_fields( document: OffsiteFundDocument, fields: Mapping[str, object] ) -> None: + fields = normalize_recognition_fields(fields) raw_date = fields.get("申请日期") document.fund_code = OffsiteFundService._text(fields.get("基金代码")) document.fund_name = OffsiteFundService._text(fields.get("基金名称")) @@ -1836,6 +1841,7 @@ class OffsiteFundService: self, task_id: str, decision: OperationDecision, operator_id: str, context: RequestContext, ) -> dict[str, object]: + task_id = task_id.strip() operator_error = self._operator_error(operator_id, context) if operator_error is not None: return operator_error @@ -1870,7 +1876,7 @@ class OffsiteFundService: return {"code": 0, "message": "ok", "data": {"task_id": task_id, "decision": decision}} async def recalculate_statistics( - self, fund_code: str, application_date: str, context: RequestContext + self, fund_code: str | None, application_date: str, context: RequestContext ) -> dict[str, object]: denied = self._permission_error(context, ("offsite:read", "offsite:write")) if denied is not None: @@ -1878,11 +1884,17 @@ class OffsiteFundService: target_date = parse_application_date(application_date) if target_date is None: return {"code": 422, "message": "申请日期格式不正确", "data": {}} - documents = (await self.session.execute(select(OffsiteFundDocument).where( - OffsiteFundDocument.fund_code == fund_code, + requested_fund_code = fund_code.strip() if fund_code else None + filters = [ OffsiteFundDocument.application_date == target_date, OffsiteFundDocument.operator_decision == "确认正常", - ))).scalars().all() + OffsiteFundDocument.fund_code.is_not(None), + ] + if requested_fund_code: + filters.append(OffsiteFundDocument.fund_code == requested_fund_code) + documents = (await self.session.execute( + select(OffsiteFundDocument).where(*filters) + )).scalars().all() task_ids = [document.task_id for document in documents] successful_normal_returns: set[str] = set() if task_ids: @@ -1899,6 +1911,42 @@ class OffsiteFundService: document for document in documents if document.task_id in successful_normal_returns ] + grouped: dict[str, list[OffsiteFundDocument]] = defaultdict(list) + for document in documents: + if document.fund_code: + grouped.setdefault(document.fund_code, []) + for row in rows: + if row.fund_code: + grouped[row.fund_code].append(row) + if requested_fund_code and requested_fund_code not in grouped: + grouped[requested_fund_code] = [] + statistics = [ + await self._build_settlement_statistic( + code, target_date, grouped[code], context + ) + for code in sorted(grouped) + ] + if requested_fund_code: + data = {**statistics[0], "items": statistics} + return {"code": 0, "message": "ok", "data": data} + return { + "code": 0, + "message": "ok", + "data": { + "application_date": target_date.isoformat(), + "fund_code": None, + "fund_count": len(statistics), + "items": statistics, + }, + } + + async def _build_settlement_statistic( + self, + fund_code: str, + target_date: date, + rows: Sequence[OffsiteFundDocument], + context: RequestContext, + ) -> dict[str, object]: subscription_total = sum( ((row.subscription_amount_yuan or Decimal("0")) for row in rows), Decimal("0"), @@ -1907,7 +1955,6 @@ class OffsiteFundService: ((row.redemption_shares or Decimal("0")) for row in rows), Decimal("0"), ) - agency_breakdown = self._agency_breakdown(rows) latest_nav = await self._query_latest_nav(fund_code, target_date, context) redemption_amount_yuan = ( redemption_total * latest_nav if latest_nav is not None else None @@ -1917,22 +1964,29 @@ class OffsiteFundService: if redemption_amount_yuan is not None else (subscription_total if redemption_total == 0 else None) ) - return {"code": 0, "message": "ok", "data": { - "fund_code": fund_code, "application_date": target_date.isoformat(), + return { + "fund_code": fund_code, + "application_date": target_date.isoformat(), "fund_name": next((row.fund_name for row in rows if row.fund_name), None), "subscription_amount_yuan": str(subscription_total), - "subscription_count": sum(1 for row in rows if row.document_type == "subscription"), + "subscription_count": sum( + 1 for row in rows if row.document_type == "subscription" + ), "redemption_shares": str(redemption_total), - "redemption_count": sum(1 for row in rows if row.document_type == "redemption"), + "redemption_count": sum( + 1 for row in rows if row.document_type == "redemption" + ), "latest_nav": str(latest_nav) if latest_nav is not None else None, "redemption_amount_yuan": ( - str(redemption_amount_yuan) if redemption_amount_yuan is not None else None + str(redemption_amount_yuan) + if redemption_amount_yuan is not None + else None ), "net_flow_amount_yuan": ( str(net_flow_amount_yuan) if net_flow_amount_yuan is not None else None ), - "agency_breakdown": agency_breakdown, - }} + "agency_breakdown": self._agency_breakdown(rows), + } async def trigger_agent_nl2sql( self, task_id: str, operator_id: str, manual_confirmed: bool, @@ -2020,6 +2074,9 @@ class OffsiteFundService: ) records.append({ "rule_code": rule_code, + "rule_name": dict( + self._offsite_rule_titles(document.document_type) + ).get(rule_code, rule_code), "status": query_status, "row_count": 1 if row is not None else 0, }) @@ -2050,6 +2107,7 @@ class OffsiteFundService: self, task_id: str, notification_type: str, operator_id: str, context: RequestContext, ) -> dict[str, object]: + task_id = task_id.strip() operator_error = self._operator_error(operator_id, context) if operator_error is not None: return operator_error @@ -2547,6 +2605,20 @@ class OffsiteFundService: error_message=None, created_at=now, updated_at=now, )) + @staticmethod + def _offsite_rule_titles( + document_type: str, + ) -> tuple[tuple[str, str], ...]: + if document_type == "redemption": + return ( + ("redemption_large_ratio", "赎回巨额比例"), + ("redemption_available_quantity", "账户可用份额"), + ) + return ( + ("subscription_holding_ratio", "申购后单一投资者持有比例"), + ("subscription_single_share_limit", "申购单笔份额上限"), + ) + @staticmethod def _nl2sql_questions(document: OffsiteFundDocument) -> list[tuple[str, str]]: if not document.fund_code: diff --git a/app/service/promotion_material_service.py b/app/service/promotion_material_service.py index 07eb380..bb9d22d 100644 --- a/app/service/promotion_material_service.py +++ b/app/service/promotion_material_service.py @@ -299,10 +299,22 @@ class PromotionMaterialService: context: RequestContext, ) -> dict[str, Any]: task = await self._task(session, task_no) + normalized_media_type = ( + media_type + or mimetypes.guess_type(filename)[0] + or "application/octet-stream" + ) + # 部分 Windows 浏览器或代理会把图片上传为通用二进制类型; + # 扩展名和 PIL 内容校验仍会继续约束文件,不能只信任请求头。 + if ( + attachment_type == "manager_photo" + and not normalized_media_type.startswith("image/") + ): + normalized_media_type = mimetypes.guess_type(filename)[0] or normalized_media_type self._validate_attachment( attachment_type, filename, - media_type, + normalized_media_type, len(payload), max_photo_size=self.max_photo_size, max_performance_size=self.max_performance_size, @@ -338,11 +350,7 @@ class PromotionMaterialService: task_no=task_no, attachment_type=attachment_type, filename=Path(filename).name, - media_type=( - media_type - or mimetypes.guess_type(filename)[0] - or "application/octet-stream" - ), + media_type=normalized_media_type, file_hash=digest, size_bytes=len(payload), file_path=str(destination), diff --git a/app/static/portal/common/api-client.js b/app/static/portal/common/api-client.js index d93549d..87fa792 100644 --- a/app/static/portal/common/api-client.js +++ b/app/static/portal/common/api-client.js @@ -55,7 +55,33 @@ const ENDPOINTS = Object.freeze({ ADVISOR_GOAL: { method: 'GET', path: '/api/v1/advisor/investment-goals/current' }, ADVISOR_ANALYSIS: { method: 'POST', path: '/api/v1/advisor/portfolio-analysis' }, OFFSITE_MAILS: { method: 'GET', path: '/api/v1/offsite-fund/mails' }, + OFFSITE_MAIL: { method: 'GET', path: '/api/v1/offsite-fund/mails/{mailId}' }, + OFFSITE_MAIL_DELETE: { method: 'POST', path: '/api/v1/offsite-fund/mails/{mailId}/deletions', idempotent: true }, + OFFSITE_RECOGNITION: { method: 'GET', path: '/api/v1/offsite-fund/mails/{mailId}/recognition-fields' }, + OFFSITE_RECOGNITION_SAVE: { method: 'PUT', path: '/api/v1/offsite-fund/mails/{mailId}/recognition-fields', idempotent: true }, + OFFSITE_NL2SQL_FIELDS: { method: 'GET', path: '/api/v1/offsite-fund/documents/{taskId}/nl2sql-fields' }, + OFFSITE_NL2SQL_FIELDS_SAVE: { method: 'PUT', path: '/api/v1/offsite-fund/documents/{taskId}/nl2sql-fields', idempotent: true }, + OFFSITE_RULE_RESULTS: { method: 'GET', path: '/api/v1/offsite-fund/documents/{taskId}/rule-results' }, + OFFSITE_RULE_RECALCULATE: { method: 'POST', path: '/api/v1/offsite-fund/documents/{taskId}/rule-results/recalculations', idempotent: true }, OFFSITE_MAILBOX: { method: 'GET', path: '/api/v1/offsite-fund/mailbox-status' }, + OFFSITE_MAILBOX_RECOVER: { method: 'POST', path: '/api/v1/offsite-fund/mailbox-status/recoveries', idempotent: true }, + OFFSITE_ATTACHMENT_FILE: { method: 'GET', path: '/api/v1/offsite-fund/attachments/{attachmentId}/file' }, + OFFSITE_CONFIRM: { method: 'POST', path: '/api/v1/offsite-fund/documents/{taskId}/confirmations', idempotent: true }, + OFFSITE_RECOGNITION_RETRY: { method: 'POST', path: '/api/v1/offsite-fund/documents/{taskId}/recognition-retries', idempotent: true }, + OFFSITE_NOTIFICATION_CREATE: { method: 'POST', path: '/api/v1/offsite-fund/documents/{taskId}/notifications', idempotent: true }, + OFFSITE_NOTIFICATION_SEND: { method: 'POST', path: '/api/v1/offsite-fund/notifications/{notificationId}/send', idempotent: true }, + OFFSITE_SETTLEMENT_RECALCULATE: { method: 'POST', path: '/api/v1/offsite-fund/settlement-statistics/recalculate', idempotent: true }, + OFFSITE_TRIGGER_NL2SQL: { method: 'POST', path: '/api/tasks/{taskId}/trigger-agent-nl2sql', idempotent: true }, + PROMOTION_CREATE: { method: 'POST', path: '/api/v1/fund-promotion-materials', idempotent: true }, + PROMOTION_TASK: { method: 'GET', path: '/api/v1/fund-promotion-materials/{taskNo}' }, + PROMOTION_INPUTS: { method: 'PUT', path: '/api/v1/fund-promotion-materials/{taskNo}/inputs', idempotent: true }, + PROMOTION_ATTACHMENT: { method: 'POST', path: '/api/v1/fund-promotion-materials/{taskNo}/attachments', formData: true, idempotent: true }, + PROMOTION_GENERATE: { method: 'POST', path: '/api/v1/fund-promotion-materials/{taskNo}/generations', idempotent: true }, + PROMOTION_CHECKS: { method: 'GET', path: '/api/v1/fund-promotion-materials/{taskNo}/compliance-checks' }, + PROMOTION_REVIEW: { method: 'POST', path: '/api/v1/fund-promotion-materials/{taskNo}/reviews', idempotent: true }, + PROMOTION_DELIVERY: { method: 'POST', path: '/api/v1/fund-promotion-materials/{taskNo}/deliveries', idempotent: true }, + AGENT_RUN_CREATE: { method: 'POST', path: '/api/v1/agent-runs', idempotent: true }, + AGENT_RUN: { method: 'GET', path: '/api/v1/agent-runs/{runId}' }, }); export class ApiError extends Error { @@ -67,6 +93,7 @@ export class ApiError extends Error { this.retryable = Boolean(options.retryable); this.fieldErrors = options.fieldErrors || []; this.traceId = options.traceId || ''; + this.payload = options.payload || null; } } @@ -114,13 +141,15 @@ async function request(endpointId, options = {}) { body: options.body === undefined || endpoint.method === 'GET' ? undefined : (endpoint.formData ? options.body : JSON.stringify(options.body)), + cache: 'no-store', signal: controller.signal, }); const payload = await response.json().catch(() => ({})); if (response.status === 401 && endpoint.auth !== false) clearAuthSession(); const responseTraceId = payload.meta?.trace_id || response.headers.get('X-Trace-ID') || traceId; document.documentElement.dataset.traceId = responseTraceId; - if (!response.ok || payload.error) { + const hasBusinessError = Object.prototype.hasOwnProperty.call(payload, 'code') && payload.code !== 0; + if (!response.ok || payload.error || hasBusinessError) { const detail = payload.error || {}; const validationDetail = Array.isArray(payload.detail) ? payload.detail @@ -129,14 +158,19 @@ async function request(endpointId, options = {}) { .join(';') : (typeof payload.detail === 'string' ? payload.detail : ''); const message = detail.message + || (hasBusinessError ? payload.message : '') || validationDetail || (response.status ? `请求失败(HTTP ${response.status})` : '请求未完成'); + const businessStatus = hasBusinessError && Number(payload.code) >= 400 + ? Number(payload.code) + : response.status; const error = new ApiError(message, { - code: detail.code, - status: response.status, + code: detail.code || (hasBusinessError ? `BUSINESS_${payload.code}` : undefined), + status: businessStatus, retryable: detail.retryable, fieldErrors: detail.field_errors, traceId: responseTraceId, + payload, }); if (response.status === 429 && attempt === 0) await wait(5000); else if (shouldRetry(error, attempt)) await wait(2000); @@ -213,10 +247,42 @@ async function stream(endpointId, body, options = {}) { if (buffer.trim()) dispatch(buffer); } +async function requestFile(endpointId, options = {}) { + const endpoint = ENDPOINTS[endpointId]; + if (!endpoint) throw new ApiError(`未注册端点 ${endpointId}`, { code: 'ENDPOINT_NOT_REGISTERED' }); + const traceId = crypto.randomUUID(); + const headers = { Accept: '*/*', 'X-Trace-ID': traceId, ...(options.headers || {}) }; + const token = getAccessToken(); + if (endpoint.auth !== false && token) headers.Authorization = `Bearer ${token}`; + const response = await fetch(`${pathFor(endpoint, options.pathParams)}${new URLSearchParams(options.query || {}).toString() ? `?${new URLSearchParams(options.query).toString()}` : ''}`, { + method: endpoint.method, + headers, + cache: 'no-store', + signal: options.signal, + }); + if (!response.ok) { + const payload = await response.json().catch(() => ({})); + const detail = payload.error || {}; + if (response.status === 401 && endpoint.auth !== false) clearAuthSession(); + throw new ApiError(detail.message || `文件请求失败(HTTP ${response.status})`, { + code: detail.code, + status: response.status, + traceId: payload.meta?.trace_id || response.headers.get('X-Trace-ID') || traceId, + }); + } + return { + blob: await response.blob(), + filename: response.headers.get('Content-Disposition') || '', + contentType: response.headers.get('Content-Type') || '', + traceId: response.headers.get('X-Trace-ID') || traceId, + }; +} + export const apiClient = Object.freeze({ get(endpointId, options = {}) { return request(endpointId, options); }, post(endpointId, body, options = {}) { return request(endpointId, { ...options, body }); }, upload(endpointId, formData, options = {}) { return request(endpointId, { ...options, body: formData, timeout: options.timeout || 30000 }); }, + file(endpointId, options = {}) { return requestFile(endpointId, options); }, stream, reportError(error) { window.dispatchEvent(new CustomEvent('portal:error', { detail: { message: error.message, traceId: error.traceId || '' } })); diff --git a/app/static/portal/common/formatters.js b/app/static/portal/common/formatters.js index 8b38886..2526077 100644 --- a/app/static/portal/common/formatters.js +++ b/app/static/portal/common/formatters.js @@ -12,12 +12,18 @@ export function formatPercent(value) { return `${number > 0 ? '+' : ''}${number.toFixed(2)}%`; } -export function formatDateTime(value) { +export function formatDateTime(value, includeSeconds = false) { if (!value) return '--'; const date = new Date(value); if (Number.isNaN(date.getTime())) return String(value); return new Intl.DateTimeFormat('zh-CN', { - year: 'numeric', month: '2-digit', day: '2-digit', hour: '2-digit', minute: '2-digit', hour12: false, + year: 'numeric', + month: '2-digit', + day: '2-digit', + hour: '2-digit', + minute: '2-digit', + ...(includeSeconds ? { second: '2-digit' } : {}), + hour12: false, }).format(date).replaceAll('/', '-'); } diff --git a/app/static/portal/common/layout/app-shell.js b/app/static/portal/common/layout/app-shell.js index 1574a12..4db5f7e 100644 --- a/app/static/portal/common/layout/app-shell.js +++ b/app/static/portal/common/layout/app-shell.js @@ -33,6 +33,9 @@ const ADVISOR_LINKS = [ const OPERATOR_LINKS = [ ['operator-dashboard', '运营工作台', '/portal/employee-operations/dashboard/'], + ['operator-offsite', '场外申赎', '/portal/employee-operations/offsite/'], + ['operator-promotion', '推介材料', '/portal/employee-operations/promotion/'], + ['operator-nl2sql', 'NL2SQL', '/portal/employee-operations/nl2sql/'], ['public-products', '公开产品', '/portal/guest/products/'], ]; @@ -121,19 +124,26 @@ export function mountShell({ active, mode = 'public' }) { window.location.assign('/portal/guest/home/?reason=signed-out'); }); const networkStatus = document.querySelector('[data-network-status]'); - window.addEventListener('offline', () => { - networkStatus.textContent = '网络已断开'; + let networkStatusTimer = 0; + const hideNetworkStatus = () => { + window.clearTimeout(networkStatusTimer); + networkStatus.classList.remove('network-status--visible'); + }; + const showNetworkStatus = (message, duration = 3600) => { + window.clearTimeout(networkStatusTimer); + networkStatus.textContent = message; networkStatus.classList.add('network-status--visible'); + if (duration > 0) networkStatusTimer = window.setTimeout(hideNetworkStatus, duration); + }; + window.addEventListener('offline', () => { + showNetworkStatus('网络已断开', 0); }); window.addEventListener('online', () => { - networkStatus.textContent = '网络已恢复,请点击刷新'; - networkStatus.classList.add('network-status--visible'); - window.setTimeout(() => networkStatus.classList.remove('network-status--visible'), 3000); + showNetworkStatus('网络已恢复,请点击刷新', 3000); }); window.addEventListener('portal:error', (event) => { const detail = event.detail || {}; - networkStatus.textContent = detail.message || '请求未完成'; - networkStatus.classList.add('network-status--visible'); + showNetworkStatus(detail.message || '请求未完成'); }); if (new URLSearchParams(window.location.search).get('reason') === 'signed-out') { networkStatus.textContent = '您已安全退出登录'; diff --git a/app/static/portal/employee-operations/dashboard/dashboard.css b/app/static/portal/employee-operations/dashboard/dashboard.css index 12102c5..2b1495e 100644 --- a/app/static/portal/employee-operations/dashboard/dashboard.css +++ b/app/static/portal/employee-operations/dashboard/dashboard.css @@ -9,4 +9,12 @@ .operator-status__card { padding: var(--space-4); background: var(--surface-soft); border-radius: var(--radius-sm); } .operator-status__card strong { display: block; margin-bottom: 5px; font-size: 18px; } .operator-status__card p { margin: 0; color: var(--muted); font-size: var(--fs-small); line-height: 1.65; } +.operator-module-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: var(--space-4); } +.operator-module-card { min-height: 148px; padding: var(--space-5); display: grid; align-content: start; gap: var(--space-2); color: var(--ink); background: var(--surface); border: 1px solid var(--line); border-radius: var(--radius-md); box-shadow: var(--shadow-card); transition: transform 180ms ease, border-color 180ms ease, box-shadow 180ms ease; } +.operator-module-card:hover, .operator-module-card:focus-visible { color: var(--ink); border-color: var(--brand); outline: 0; transform: translateY(-2px); box-shadow: var(--shadow-elevated); } +.operator-module-card__index { color: var(--brand-dark); font: 700 var(--fs-small)/1 Consolas, monospace; letter-spacing: .08em; } +.operator-module-card strong { font-size: 18px; } +.operator-module-card > span:last-child { color: var(--muted); font-size: var(--fs-small); line-height: 1.6; } +.heading-actions { display: flex; flex-wrap: wrap; gap: var(--space-2); align-items: center; } @media (max-width: 760px) { .operator-grid { grid-template-columns: 1fr; } } +@media (max-width: 860px) { .operator-module-grid { grid-template-columns: 1fr; } } diff --git a/app/static/portal/employee-operations/dashboard/index.html b/app/static/portal/employee-operations/dashboard/index.html index 07ce254..c9e0d04 100644 --- a/app/static/portal/employee-operations/dashboard/index.html +++ b/app/static/portal/employee-operations/dashboard/index.html @@ -1,3 +1,3 @@
场外运营与资料处理
集中查看收件箱、识别任务与规则状态,所有场外运营流程与场内模拟交易数据严格隔离。
场外运营与资料处理
集中查看收件箱、识别任务与规则状态,所有场外运营流程与场内模拟交易数据严格隔离。