Worker 失败原因不再只留类名:区分"可落库的固定文案"与"异常消息"
起因是查 Worker 运行状态时发现库里 373 条死信的 last_error 全是裸的 "ValueError"
(工具 tools/probe_worker_state.py,证据 docs/evidence/worker-state.json)。
dispatch(run not found)、dispatch_run_completed、dispatch_memory_extraction、
dispatch_profile_rebuild 抛的都是 ValueError,只记类名等于把"哪一处失败"也一起丢了。
但"直接存 str(exc)"是错的:tests/unit/worker/test_outbox_worker.py 那条
RuntimeError("credential=do-not-log") 断言异常消息不得落库 —— 它可能含凭据、SQL 或
客户标识。第一版改动就是这么写的,被这个测试当场拦下(这测试写得值)。
折中:
- 新增 OutboxHandlerError(继承 ValueError,这些失败本就是 ValueError 语义,保持
继承关系才不会改动既有的 except ValueError 行为与断言)。它的 reason 由代码写死、
不含任何请求数据,因此可以落库;
- safe_error_text:OutboxHandlerError → "类名: 固定文案"(截断 500 字符),
其余异常 → 仍只记类名;
- runtime.py 的 5 处 handler 失败改抛 OutboxHandlerError。
测试:新增"固定文案落库"用例;并把既有用例的断言收紧为 last_error == "RuntimeError"
(原先只断言"不含 do-not-log",太松,漏掉的情况测不出来)。
顺带产出 tools/probe_worker_state.py(只读):outbox / agent_run 各状态计数、按事件
类型分组、死信原因聚合。当前环境实测 pending 347、dead 373、published 410、
agent_run 无 queued/running。
门禁:ruff 干净 / mypy 138 文件 / 697 unit+contract / 33 integration。
This commit is contained in:
@@ -0,0 +1,135 @@
|
||||
{
|
||||
"outbox_columns": [
|
||||
"id",
|
||||
"event_id",
|
||||
"event_type",
|
||||
"aggregate_type",
|
||||
"aggregate_id",
|
||||
"trace_id",
|
||||
"payload",
|
||||
"status",
|
||||
"retry_count",
|
||||
"next_retry_at",
|
||||
"last_error",
|
||||
"occurred_at",
|
||||
"published_at",
|
||||
"created_at",
|
||||
"updated_at"
|
||||
],
|
||||
"agent_run_columns": [
|
||||
"id",
|
||||
"run_id",
|
||||
"idempotency_id",
|
||||
"session_id",
|
||||
"user_id",
|
||||
"agent_type",
|
||||
"trace_id",
|
||||
"request_message_id",
|
||||
"result_message_id",
|
||||
"status",
|
||||
"attempt_count",
|
||||
"worker_id",
|
||||
"locked_until",
|
||||
"error_code",
|
||||
"result_version",
|
||||
"cancel_requested_at",
|
||||
"started_at",
|
||||
"completed_at",
|
||||
"created_at",
|
||||
"updated_at"
|
||||
],
|
||||
"outbox_by_status": {
|
||||
"dead": 373,
|
||||
"failed": 3,
|
||||
"pending": 347,
|
||||
"published": 410
|
||||
},
|
||||
"agent_run_by_status": {
|
||||
"failed": 21,
|
||||
"succeeded": 184
|
||||
},
|
||||
"outbox_created_at_range": [
|
||||
"2026-09-09 06:55:06.787006",
|
||||
"2026-09-11 07:21:23.195457"
|
||||
],
|
||||
"outbox_total": 1133,
|
||||
"agent_run_total": 205,
|
||||
"outbox_by_event_type_status": [
|
||||
{
|
||||
"event_type": "agent.run_requested",
|
||||
"status": "dead",
|
||||
"count": 373
|
||||
},
|
||||
{
|
||||
"event_type": "agent.run_requested",
|
||||
"status": "pending",
|
||||
"count": 132
|
||||
},
|
||||
{
|
||||
"event_type": "agent.run_requested",
|
||||
"status": "published",
|
||||
"count": 130
|
||||
},
|
||||
{
|
||||
"event_type": "agent.run_completed",
|
||||
"status": "published",
|
||||
"count": 118
|
||||
},
|
||||
{
|
||||
"event_type": "memory.extraction_requested",
|
||||
"status": "published",
|
||||
"count": 106
|
||||
},
|
||||
{
|
||||
"event_type": "profile.rebuild_requested",
|
||||
"status": "pending",
|
||||
"count": 76
|
||||
},
|
||||
{
|
||||
"event_type": "agent.run_completed",
|
||||
"status": "pending",
|
||||
"count": 66
|
||||
},
|
||||
{
|
||||
"event_type": "memory.extraction_requested",
|
||||
"status": "pending",
|
||||
"count": 60
|
||||
},
|
||||
{
|
||||
"event_type": "config.cache_invalidate_requested",
|
||||
"status": "published",
|
||||
"count": 42
|
||||
},
|
||||
{
|
||||
"event_type": "profile.rebuild_requested",
|
||||
"status": "published",
|
||||
"count": 13
|
||||
},
|
||||
{
|
||||
"event_type": "config.cache_invalidate_requested",
|
||||
"status": "pending",
|
||||
"count": 13
|
||||
},
|
||||
{
|
||||
"event_type": "agent.run_requested",
|
||||
"status": "failed",
|
||||
"count": 3
|
||||
},
|
||||
{
|
||||
"event_type": "memory.invalidated",
|
||||
"status": "published",
|
||||
"count": 1
|
||||
}
|
||||
],
|
||||
"dead_last_errors": [
|
||||
{
|
||||
"last_error": "ValueError",
|
||||
"count": 372
|
||||
},
|
||||
{
|
||||
"last_error": "no handler registered",
|
||||
"count": 1
|
||||
}
|
||||
],
|
||||
"episode_probe_error": "ImportError: cannot import name 'MemoryEpisode' from 'app.model.memory' (C:\\Users\\Windows\\Desktop\\项目代码\\app\\model\\memory.py)"
|
||||
}
|
||||
Reference in New Issue
Block a user