update, translate skill.md
This commit is contained in:
@@ -1,116 +1,118 @@
|
||||
---
|
||||
name: metadata-validator
|
||||
description: Use when validating that a data requirement's candidate tables, fields, joins, and time columns actually exist in the project's metadata. Triggers include "validate metadata", "check if table exists", "field mapping", "join key validation", "确认表/字段", "校验 schema", "这个字段在哪张表", "口径对得上吗". Reads metadata files recursively across the workspace (JSON / Markdown / CSV / Excel / YAML / DDL / data dictionary) and produces a standardized Validation Result and Field Mapping.
|
||||
description: 当需要校验某个数据需求中候选的表、字段、join 关联和时间字段是否真实存在于工作区元数据中时使用。触发词包括"validate metadata"、"check if table exists"、"field mapping"、"join key validation"、"确认表/字段"、"校验 schema"、"这个字段在哪张表"、"口径对得上吗"。递归扫描工作区中的元数据文件(JSON / Markdown / CSV / Excel / YAML / DDL / 数据字典),并产出标准化的 Validation Result 和 Field Mapping。
|
||||
---
|
||||
|
||||
# Metadata Validator
|
||||
# Metadata Validator(元数据校验器)
|
||||
|
||||
## Overview
|
||||
## 概述(Overview)
|
||||
|
||||
Bridge between the requirement's *expected* tables/fields and the *actual* schema living in workspace metadata. Output is consumed verbatim by `logic-planner` and `sql-context-builder` — they must NOT re-validate.
|
||||
本 skill 在需求的"期望表/字段"和工作区元数据中的"实际 schema"之间架起桥梁。输出会被 `logic-planner` 和 `sql-context-builder` 原样消费,二者**不得**重复校验。
|
||||
|
||||
Core principle: **never guess a table name, field name, or join relationship.** If a candidate is not provably present in metadata, surface it as a `pending_question` and stop at status `NEED_USER_CONFIRMATION`.
|
||||
核心原则:**绝不臆测表名、字段名或 join 关系。** 如果某候选在元数据中无法证实存在,将其作为 `pending_question` 暴露,状态停在 `NEED_USER_CONFIRMATION`。
|
||||
|
||||
## When to Use
|
||||
## 何时使用(When to Use)
|
||||
|
||||
Use when:
|
||||
- `requirements-analysis` has produced candidate tables/fields and you need to confirm they exist in real metadata
|
||||
- The user asks "does this table/field exist", "which file describes table X", "map business field to physical column"
|
||||
- You need to detect join keys, time columns, or aggregation-suitable numeric columns
|
||||
- The user is about to write SQL and you must guard against hallucinated column names
|
||||
**使用场景:**
|
||||
|
||||
Do NOT use when:
|
||||
- The user has already supplied exact table+column names and confirmed them — go straight to `logic-planner`
|
||||
- The task is code generation without any data warehouse context
|
||||
- `requirements-analysis` 已经产出候选表/字段,需要确认它们在真实元数据中存在
|
||||
- 用户询问"这张表/这个字段是否存在"、"表 X 的元数据在哪份文件里"、"业务字段映射到哪个物理列"
|
||||
- 需要检测 join key、时间列、可用于聚合的数值列
|
||||
- 即将写 SQL,必须防止出现幻觉字段名
|
||||
|
||||
## Inputs
|
||||
**不要使用场景:**
|
||||
|
||||
1. From `requirements-analysis`:
|
||||
- `candidate_tables`: list of table names mentioned or implied
|
||||
- `candidate_fields`: list of business field names with their proposed table
|
||||
- `business_logic`: short summary of what is being computed
|
||||
2. Workspace metadata: any file under the working directory matching metadata conventions (see below)
|
||||
- 用户已经提供并确认了精确的表名+列名 — 直接进入 `logic-planner`
|
||||
- 任务是无数据仓库上下文的纯代码生成
|
||||
|
||||
## Workflow
|
||||
## 输入(Inputs)
|
||||
|
||||
1. 来自 `requirements-analysis` 的:
|
||||
- `candidate_tables`: 用户提到或隐含的表名列表
|
||||
- `candidate_fields`: 业务字段名及其建议所在表
|
||||
- `business_logic`: 计算逻辑的简短描述
|
||||
2. 工作区元数据:当前工作目录下任何符合元数据约定的文件(见下文)
|
||||
|
||||
## 工作流(Workflow)
|
||||
|
||||
```dot
|
||||
digraph metadata_validator {
|
||||
"Receive candidate tables/fields" [shape=box];
|
||||
"Recursively scan workspace for metadata files" [shape=box];
|
||||
"Parse each file into canonical (table, field, type) records" [shape=box];
|
||||
"Check candidate tables exist" [shape=box];
|
||||
"Check candidate fields exist in their table" [shape=box];
|
||||
"Fuzzy match missing fields against parsed metadata" [shape=box];
|
||||
"Detect join keys & time columns" [shape=box];
|
||||
"All checks pass?" [shape=diamond];
|
||||
"Build Field Mapping" [shape=box];
|
||||
"Emit Validation Result (VALIDATED)" [shape=box];
|
||||
"Emit Validation Result (NEED_USER_CONFIRMATION)" [shape=box];
|
||||
"接收候选表/字段" [shape=box];
|
||||
"递归扫描工作区中的元数据文件" [shape=box];
|
||||
"将每个文件解析为规范化的 (table, field, type) 记录" [shape=box];
|
||||
"检查候选表是否存在" [shape=box];
|
||||
"检查候选字段是否在其所属表中存在" [shape=box];
|
||||
"对缺失字段做模糊匹配" [shape=box];
|
||||
"检测 join key 与时间列" [shape=box];
|
||||
"所有检查都通过?" [shape=diamond];
|
||||
"构建 Field Mapping" [shape=box];
|
||||
"输出 Validation Result (VALIDATED)" [shape=box];
|
||||
"输出 Validation Result (NEED_USER_CONFIRMATION)" [shape=box];
|
||||
|
||||
"Receive candidate tables/fields" -> "Recursively scan workspace for metadata files";
|
||||
"Recursively scan workspace for metadata files" -> "Parse each file into canonical (table, field, type) records";
|
||||
"Parse each file into canonical (table, field, type) records" -> "Check candidate tables exist";
|
||||
"Check candidate tables exist" -> "Check candidate fields exist in their table";
|
||||
"Check candidate fields exist in their table" -> "Fuzzy match missing fields against parsed metadata";
|
||||
"Fuzzy match missing fields against parsed metadata" -> "Detect join keys & time columns";
|
||||
"Detect join keys & time columns" -> "All checks pass?";
|
||||
"All checks pass?" -> "Build Field Mapping" [label="yes"];
|
||||
"All checks pass?" -> "Emit Validation Result (NEED_USER_CONFIRMATION)" [label="no"];
|
||||
"Build Field Mapping" -> "Emit Validation Result (VALIDATED)";
|
||||
"接收候选表/字段" -> "递归扫描工作区中的元数据文件";
|
||||
"递归扫描工作区中的元数据文件" -> "将每个文件解析为规范化的 (table, field, type) 记录";
|
||||
"将每个文件解析为规范化的 (table, field, type) 记录" -> "检查候选表是否存在";
|
||||
"检查候选表是否存在" -> "检查候选字段是否在其所属表中存在";
|
||||
"检查候选字段是否在其所属表中存在" -> "对缺失字段做模糊匹配";
|
||||
"对缺失字段做模糊匹配" -> "检测 join key 与时间列";
|
||||
"检测 join key 与时间列" -> "所有检查都通过?";
|
||||
"所有检查都通过?" -> "构建 Field Mapping" [label="yes"];
|
||||
"所有检查都通过?" -> "输出 Validation Result (NEED_USER_CONFIRMATION)" [label="no"];
|
||||
"构建 Field Mapping" -> "输出 Validation Result (VALIDATED)";
|
||||
}
|
||||
```
|
||||
|
||||
## Metadata Discovery
|
||||
## 元数据发现(Metadata Discovery)
|
||||
|
||||
- Recursively scan the **current working directory** (and any subdirectories) — do not assume a `metadata/` folder exists.
|
||||
- Filenames often resemble table names (e.g. `loan_order.json`, `customer_info.md`, `dim_product.csv`) but **never trust the filename alone** — open the file and confirm the table identifier inside.
|
||||
- Accepted formats (auto-detect by content, not extension):
|
||||
- 递归扫描**当前工作目录**(及其所有子目录) — 不要假设存在 `metadata/` 目录
|
||||
- 文件名常常形似表名(例如 `loan_order.json`、`customer_info.md`、`dim_product.csv`),但**绝不要只信任文件名** — 必须打开文件,确认里面的表标识符
|
||||
- 支持的格式(按内容自动识别,不按扩展名):
|
||||
- JSON / JSONL
|
||||
- YAML
|
||||
- Markdown tables / headings
|
||||
- CSV (with header)
|
||||
- Excel (`.xlsx`, `.xls`) — use `openpyxl` or `pandas.read_excel`
|
||||
- DDL / `CREATE TABLE` statements
|
||||
- Free-form data dictionaries (parse key-value lines or `field | type | comment` tables)
|
||||
- Convert every file into the canonical record shape before validating:
|
||||
- Markdown 表格 / 标题
|
||||
- CSV(带表头)
|
||||
- Excel(`.xlsx`、`.xls`) — 用 `openpyxl` 或 `pandas.read_excel`
|
||||
- DDL / `CREATE TABLE` 语句
|
||||
- 自由形式的数据字典(解析键值行或 `field | type | comment` 表格)
|
||||
- 在校验之前,先把每个文件转成规范化的记录格式:
|
||||
|
||||
```yaml
|
||||
tables:
|
||||
- table: <physical_table_name>
|
||||
source_file: <relative_path>
|
||||
- table: <物理表名>
|
||||
source_file: <相对路径>
|
||||
fields:
|
||||
- name: <field_name>
|
||||
- name: <字段名>
|
||||
type: <string|int|long|double|decimal|date|timestamp|boolean|...>
|
||||
comment: <optional>
|
||||
comment: <可选>
|
||||
```
|
||||
|
||||
## Validation Rules
|
||||
## 校验规则(Validation Rules)
|
||||
|
||||
Run **all** of the following. Any failure ⇒ status `NEED_USER_CONFIRMATION`.
|
||||
**全部**以下规则都要执行。任意一条失败 ⇒ 状态为 `NEED_USER_CONFIRMATION`。
|
||||
|
||||
1. **Table existence** — every candidate table must appear in the parsed metadata.
|
||||
2. **Field existence** — every candidate field must appear in its candidate table.
|
||||
3. **Field ownership** — a field may exist but belong to a different table; never silently swap.
|
||||
4. **Type sanity** — aggregation targets (sum / avg / count) must be numeric; time filters must be date/timestamp/string-encoded-date.
|
||||
5. **Necessary fields** — flag if common fields are missing for the business logic (stat metric, time, join key, dimension).
|
||||
6. **Fuzzy match** — for every missing field, score candidates by name similarity (`customer_id` ↔ `cust_id`, `cust_no`, `customer_no`) and Levenshtein distance; surface top-N.
|
||||
7. **Join detection** — across the validated tables, look for shared `*_id`, `*_no`, `*_code` columns. If multiple plausible join keys exist, ask.
|
||||
8. **Time column disambiguation** — list every time-like field per table (`create_time`, `apply_time`, `txn_date`, `update_time`, `dt`); ask which one defines the time grain.
|
||||
9. **Coverage** — verify that dimensions, metrics, filters, and sort keys are all present.
|
||||
1. **表存在性** — 每个候选表必须出现在解析后的元数据中
|
||||
2. **字段存在性** — 每个候选字段必须出现在其候选所属表中
|
||||
3. **字段归属** — 字段可能存在但属于另一张表;绝不能悄悄替换
|
||||
4. **类型合理性** — 聚合目标(sum / avg / count)必须为数值型;时间过滤字段必须为 date/timestamp/字符串型日期
|
||||
5. **必要字段** — 如果业务逻辑中常见字段缺失(统计指标、时间、join key、维度),要标记出来
|
||||
6. **模糊匹配** — 对每个缺失字段,按名称相似度打分(`customer_id` ↔ `cust_id`、`cust_no`、`customer_no`)和 Levenshtein 距离,给出 Top-N
|
||||
7. **join 检测** — 在已校验的表之间,查找共享的 `*_id`、`*_no`、`*_code` 列。如果存在多个可能的 join key,必须询问
|
||||
8. **时间列消歧** — 列出每张表中所有时间类字段(`create_time`、`apply_time`、`txn_date`、`update_time`、`dt`);询问哪一个决定时间粒度
|
||||
9. **覆盖度** — 验证维度、指标、过滤列、排序键是否都存在
|
||||
|
||||
## Forbidden Behaviors
|
||||
## 禁止行为(Forbidden Behaviors)
|
||||
|
||||
| Rationalization | Reality |
|
||||
| 自我说服 | 现实 |
|
||||
|---|---|
|
||||
| "The filename is `loan_order.json`, must be that table" | Filename is a hint. Parse the file to confirm. |
|
||||
| "Field exists somewhere, so the join is fine" | Wrong table ownership invalidates the join. Always check ownership. |
|
||||
| "The user probably means `cust_no`, just use it" | Surface as a candidate and ASK. Never auto-substitute. |
|
||||
| "I'll guess the time field" | Multiple time fields ⇒ ASK. The grain defines the entire query. |
|
||||
| "This metadata is enough, no need to scan further" | Always scan recursively. One file can describe many tables. |
|
||||
| "文件名是 `loan_order.json`,肯定就是这张表" | 文件名只是提示,必须解析文件内容确认。 |
|
||||
| "字段存在于某处,所以 join 没问题" | 错误的表归属会让 join 失效。永远要校验归属。 |
|
||||
| "用户八成指的是 `cust_no`,直接用吧" | 必须作为候选暴露并询问,绝不能自动替换。 |
|
||||
| "时间字段我猜一个就行" | 多个时间字段时必须询问。粒度决定整条查询。 |
|
||||
| "这份元数据够了,不必再扫" | 永远要递归扫描。一个文件可能描述多张表。 |
|
||||
|
||||
## Output Contract
|
||||
## 输出契约(Output Contract)
|
||||
|
||||
The output is a single YAML document. Downstream skills consume it as-is.
|
||||
输出是一份单一的 YAML 文档,下游 skill 原样消费。
|
||||
|
||||
```yaml
|
||||
validation_result:
|
||||
@@ -137,26 +139,27 @@ validation_result:
|
||||
- <field>
|
||||
- <field>
|
||||
metadata_sources:
|
||||
- <relative_path_to_metadata_file>
|
||||
- <相对路径指向元数据文件>
|
||||
pending_questions:
|
||||
- "<human-readable question, ideally with options A/B/C>"
|
||||
- "<面向用户的问题,最好给出 A/B/C 选项>"
|
||||
field_mapping:
|
||||
<business_field_name>:
|
||||
table: <physical_table>
|
||||
column: <physical_field>
|
||||
type: <data_type>
|
||||
table: <物理表>
|
||||
column: <物理字段>
|
||||
type: <数据类型>
|
||||
status: <VALIDATED | NEED_USER_CONFIRMATION>
|
||||
```
|
||||
|
||||
`field_mapping` is the **single source of truth** for downstream `logic-planner` and `sql-context-builder`. If a business field has no mapping, the request cannot proceed — add it to `pending_questions`.
|
||||
`field_mapping` 是下游 `logic-planner` 和 `sql-context-builder` 的**唯一权威来源**。如果某个业务字段没有映射,该需求无法继续 — 把它加入 `pending_questions`。
|
||||
|
||||
## Completion Criteria
|
||||
## 完成判定(Completion Criteria)
|
||||
|
||||
Status `VALIDATED` is only allowed when **all** of the following hold:
|
||||
- Every candidate table is present in `validated_tables`
|
||||
- Every candidate field has a `field_mapping` entry
|
||||
- All join keys are decided (or trivially obvious with `high` confidence)
|
||||
- The time column for the query grain is decided
|
||||
- `pending_questions` is empty
|
||||
仅当**以下全部**成立时,才允许状态 `VALIDATED`:
|
||||
|
||||
Anything else ⇒ `NEED_USER_CONFIRMATION` and stop. Do not proceed to `logic-planner`.
|
||||
- 每个候选表都出现在 `validated_tables` 中
|
||||
- 每个候选字段都有 `field_mapping` 条目
|
||||
- 所有 join key 都已确定(或显然到可以用 `high` 置信度推断)
|
||||
- 查询粒度对应的时间列已确定
|
||||
- `pending_questions` 为空
|
||||
|
||||
其余情况 ⇒ `NEED_USER_CONFIRMATION` 并停止。**不得**继续进入 `logic-planner`。
|
||||
|
||||
Reference in New Issue
Block a user