update, translate skill.md

This commit is contained in:
tao.chen
2026-06-22 10:08:52 +08:00
parent 71a23d639c
commit f9b03384ab
6 changed files with 565 additions and 541 deletions
+82 -79
View File
@@ -1,97 +1,99 @@
---
name: logic-planner
description: Use when turning a validated data requirement (candidate tables/fields/joins already confirmed) into a step-by-step execution plan that a SQL generator can implement mechanically. Triggers include "拆解逻辑", "执行计划", "logic plan", "SQL 编排", "怎么算", "指标拆解", "join 顺序". Consumes the Validation Result from metadata-validator and produces a Logic Plan in a fixed step taxonomy (Source / Filter / Join / Transform / Aggregate / Output). Does NOT write SQL.
description: 当需要把一份已校验的数据需求(候选表/字段/join 关联已确认)拆解为 SQL 生成器可机械执行的步骤计划时使用。触发词包括"拆解逻辑""执行计划""logic plan""SQL 编排""怎么算""指标拆解""join 顺序"。消费 metadata-validator 的 Validation Result,产出固定步骤分类法(Source / Filter / Join / Transform / Aggregate / Output)的 Logic Plan。**不写 SQL**。
---
# Logic Planner
# Logic Planner(逻辑规划器)
## Overview
## 概述(Overview)
Decompose a validated data requirement into a deterministic pipeline of steps in a fixed taxonomy. The Logic Plan is the *only* input `sql-context-builder` needs to assemble a SQL Context — no further business reasoning is permitted downstream.
将一份已校验的数据需求分解为固定分类法下的确定性步骤流水线。Logic Plan 是 `sql-context-builder` 拼装 SQL Context **唯一**的输入 — 下游不允许再做任何业务推理。
Core principle: **decompose, don't translate.** This skill never writes SQL syntax. It only describes *what* must happen at each step in business/logical terms. The translation to SQL is a separate concern.
核心原则:**只拆分,不翻译。** 本 skill 绝不写 SQL 语法,只在业务/逻辑层面描述每一步要发生什么。SQL 翻译是另一回事。
## When to Use
## 何时使用(When to Use)
Use when:
- `metadata-validator` returned status `VALIDATED` and you need to design how the query will run
- The user asks "how should I compute this", "what's the logic", "step-by-step plan" for a data requirement
- You need to disambiguate grain, metric formulas, deduplication, windowing, or ranking before SQL
**使用场景:**
Do NOT use when:
- `metadata-validator` returned `NEED_USER_CONFIRMATION` — go back and resolve open questions first
- The user has already provided a written execution plan and just needs SQL — go straight to `sql-context-builder`
- `metadata-validator` 返回状态 `VALIDATED`,需要设计查询如何执行
- 用户询问"这个应该怎么算"、"逻辑是啥"、"一步步怎么走"等数据需求问题
- 写 SQL 之前需要先消除粒度、指标公式、去重、窗口、排名的歧义
## Inputs
**不要使用场景:**
1. **Validation Result** from `metadata-validator` (status `VALIDATED`)
2. The original business requirement (often embedded in `requirements-analysis` output)
3. `field_mapping` (mandatory) — every business field must be mapped to a physical `(table, column, type)` before planning starts
- `metadata-validator` 返回 `NEED_USER_CONFIRMATION` — 先回去解决未决问题
- 用户已经提供了书面执行计划,只是要 SQL — 直接进入 `sql-context-builder`
## Step Taxonomy
## 输入(Inputs)
Every Logic Plan is a list of steps. Each step has exactly one `kind` from this set. Reuse the same kind for repeated operations (e.g. multiple joins, multiple filters).
1. 来自 `metadata-validator`**Validation Result**(状态为 `VALIDATED`)
2. 原始业务需求(通常嵌在 `requirements-analysis` 的输出中)
3. `field_mapping`(必备) — 每个业务字段在规划开始前必须先映射到物理 `(table, column, type)`
| Kind | Purpose | Required fields |
## 步骤分类法(Step Taxonomy)
每个 Logic Plan 都是一组步骤。每个步骤有且仅有一个 `kind`,且只能从下表选取。对重复操作(如多次 join、多次 filter)复用同一种 kind。
| Kind | 用途 | 必填字段 |
|---|---|---|
| `source` | Read from a table or subquery | `alias`, `table`, `columns[]` |
| `filter` | Apply predicates to a named alias | `alias`, `predicates[]` |
| `join` | Combine two aliases on keys | `left`, `right`, `keys[]`, `type` (inner/left/right/full/semi/anti) |
| `transform` | Derive new columns (case-when, arithmetic, type cast) | `alias`, `expressions[]` |
| `aggregate` | Group and reduce | `alias`, `group_by[]`, `metrics[]` (each: `expr`, `alias`, `agg`) |
| `dedupe` | DISTINCT or row_number-based dedup | `alias`, `keys[]`, `strategy` |
| `window` | Ranking / running totals / lag-lead | `alias`, `partition_by[]`, `order_by[]`, `window_fn` |
| `output` | Final projection and ordering | `columns[]`, `order_by[]`, `limit?` |
| `source` | 从表或子查询读取 | `alias`, `table`, `columns[]` |
| `filter` | 在指定 alias 上应用谓词 | `alias`, `predicates[]` |
| `join` | 在 key 上合并两个 alias | `left`, `right`, `keys[]`, `type` (inner/left/right/full/semi/anti) |
| `transform` | 派生新列(case-when、算术、类型转换) | `alias`, `expressions[]` |
| `aggregate` | 分组聚合 | `alias`, `group_by[]`, `metrics[]` (每个含 `expr`, `alias`, `agg`) |
| `dedupe` | DISTINCT 或基于 row_number 的去重 | `alias`, `keys[]`, `strategy` |
| `window` | 排名 / 累计 / lag-lead | `alias`, `partition_by[]`, `order_by[]`, `window_fn` |
| `output` | 最终投影和排序 | `columns[]`, `order_by[]`, `limit?` |
Do not invent new kinds. If a transformation does not fit, surface it as `pending_question` and stop.
不要发明新 kind。如果某个变换无法归入上述分类,作为 `pending_question` 暴露并停止。
## Workflow
## 工作流(Workflow)
```dot
digraph logic_planner {
"Read Validation Result + field_mapping" [shape=box];
"Identify fact vs dimension tables & choose primary source" [shape=box];
"List filters (time range, status, business predicates)" [shape=box];
"Plan joins (alias A ON key = alias B.key)" [shape=box];
"Plan transforms (derived columns, type casts)" [shape=box];
"Plan dedupe / window if needed" [shape=box];
"Plan aggregate (grain + metrics)" [shape=box];
"Plan output projection + order" [shape=box];
"Any ambiguity?" [shape=diamond];
"Emit Logic Plan" [shape=box];
"Emit Logic Plan + pending_questions" [shape=box];
"读取 Validation Result + field_mapping" [shape=box];
"区分事实表 / 维度表,选定主源表" [shape=box];
"列出过滤条件(时间区间、状态、业务谓词)" [shape=box];
"规划 join(alias A ON key = alias B.key)" [shape=box];
"规划 transform(派生列、类型转换)" [shape=box];
"如需去重 / 窗口,一并规划" [shape=box];
"规划 aggregate(粒度 + 指标)" [shape=box];
"规划 output 投影 + 排序" [shape=box];
"还有歧义?" [shape=diamond];
"输出 Logic Plan" [shape=box];
"输出 Logic Plan + pending_questions" [shape=box];
"Read Validation Result + field_mapping" -> "Identify fact vs dimension tables & choose primary source";
"Identify fact vs dimension tables & choose primary source" -> "List filters (time range, status, business predicates)";
"List filters (time range, status, business predicates)" -> "Plan joins (alias A ON key = alias B.key)";
"Plan joins (alias A ON key = alias B.key)" -> "Plan transforms (derived columns, type casts)";
"Plan transforms (derived columns, type casts)" -> "Plan dedupe / window if needed";
"Plan dedupe / window if needed" -> "Plan aggregate (grain + group_by)";
"Plan aggregate (grain + group_by)" -> "Plan output projection + order";
"Plan output projection + order" -> "Any ambiguity?";
"Any ambiguity?" -> "Emit Logic Plan" [label="no"];
"Any ambiguity?" -> "Emit Logic Plan + pending_questions" [label="yes"];
"读取 Validation Result + field_mapping" -> "区分事实表 / 维度表,选定主源表";
"区分事实表 / 维度表,选定主源表" -> "列出过滤条件(时间区间、状态、业务谓词)";
"列出过滤条件(时间区间、状态、业务谓词)" -> "规划 join(alias A ON key = alias B.key)";
"规划 join(alias A ON key = alias B.key)" -> "规划 transform(派生列、类型转换)";
"规划 transform(派生列、类型转换)" -> "如需去重 / 窗口,一并规划";
"如需去重 / 窗口,一并规划" -> "规划 aggregate(粒度 + group_by)";
"规划 aggregate(粒度 + group_by)" -> "规划 output 投影 + 排序";
"规划 output 投影 + 排序" -> "还有歧义?";
"还有歧义?" -> "输出 Logic Plan" [label="no"];
"还有歧义?" -> "输出 Logic Plan + pending_questions" [label="yes"];
}
```
## Required Reasoning (do not skip)
## 必须给出的推理(Required Reasoning,不可跳过)
Before emitting the plan, you MUST explicitly state and record:
在输出 plan 之前,你必须显式陈述并记录:
1. **Grain**what does one output row represent? (e.g. one customer, one customer×month, one order)
2. **Metric formula**for every metric, write the exact expression: numerator and denominator defined.
3. **Time grain**confirm the time field and granularity (day, month, hour). State the window inclusively.
4. **Join cardinality**for every join, predict 1:1 / 1:N / N:N. If N:N, mark it and ask the user whether to dedupe before joining.
5. **Dedup position**if a fact table has row multiplicity (e.g. one customer with many addresses), decide *before* the join which side gets deduped and on which key.
6. **Null strategy**for nullable join keys or measures, decide `INNER` vs `LEFT + COALESCE`.
1. **粒度(Grain)**输出一行代表什么?(例如:一个客户、一个客户×月、一笔订单)
2. **指标公式(Metric formula)**对每个指标写出精确表达式:分子与分母都要定义
3. **时间粒度(Time grain)**确认时间字段与粒度(天 / 月 / 小时),用闭/开区间说明窗口
4. **Join 基数(Join cardinality)**对每个 join,预测 1:1 / 1:N / N:N。如果是 N:N,必须标记并询问用户是否在 join 前去重
5. **去重位置(Dedup position)**如果事实表存在行重复(例如一个客户多个地址),在 join 之前决定哪一侧、在哪个 key 上先去重
6. **空值策略(Null strategy)**对可空 join key 或度量,决定用 `INNER` 还是 `LEFT + COALESCE`
If any of the above is unclear, add a `pending_question` and stop. Do not guess.
以上任意一项不清楚时,加入 `pending_question` 并停止。绝不臆测。
## Output Contract
## 输出契约(Output Contract)
```yaml
logic_plan:
grain: "<one sentence describing what one output row represents>"
grain: "<一句话描述输出一行代表什么>"
fact_table: <table_name>
dimension_tables:
- <table_name>
@@ -111,7 +113,7 @@ logic_plan:
right: { alias: c, table: customer_info }
keys: ["o.customer_id = c.customer_id"]
type: left
cardinality_assumption: 1:N # if known; else state "unknown"
cardinality_assumption: 1:N # 已知时填;否则写 "unknown"
- kind: transform
alias: o
expressions:
@@ -132,27 +134,28 @@ logic_plan:
order_by: ["total_loan DESC"]
limit: 100
pending_questions:
- "<ambiguity not yet resolved, blocking SQL generation>"
- "<尚未解决的歧义,会阻塞 SQL 生成>"
status: <PLANNED | NEED_USER_CONFIRMATION>
```
## Forbidden Behaviors
## 禁止行为(Forbidden Behaviors)
| Rationalization | Reality |
| 自我说服 | 现实 |
|---|---|
| "I'll write the SELECT list directly, the user can read SQL" | Logic Plan uses business terms; SQL is downstream. |
| "Aggregation is obvious, skip the formula" | Explicit numerator/denominator avoids silent metric drift. |
| "1:N vs N:N doesn't matter for SELECT" | It matters: it determines dedup and inflation risk. |
| "Time grain is the user's problem" | Pick a grain. Wrong grain ⇒ wrong numbers. |
| "I'll add a HAVING clause at SQL time" | Logic Plan must list all filters, including post-aggregation. |
| "我直接写 SELECT 列表,用户能读懂 SQL" | Logic Plan 用业务术语,SQL 是下游的事。 |
| "聚合太显然,公式就省了" | 显式给出分子/分母才能避免静默的指标漂移。 |
| "1:N 还是 N:N 对 SELECT 来说无所谓" | 它决定了是否要去重以及是否有膨胀风险,影响很大。 |
| "时间粒度是用户的问题" | 必须由规划器选定一个粒度,选错了数字就全错。 |
| "HAVING 我到 SQL 那一步再加" | 所有过滤(含聚合后)都必须在 Logic Plan 中列出。 |
## Completion Criteria
## 完成判定(Completion Criteria)
Status `PLANNED` requires:
- Grain is one sentence, not a paragraph
- Every `field_mapping` entry is referenced by exactly one step
- Every join has a cardinality assumption
- Every metric has an explicit `agg` and an `alias`
- `pending_questions` is empty
要输出状态 `PLANNED`,必须满足:
Otherwise: `NEED_USER_CONFIRMATION` and stop. Do not proceed to `sql-context-builder`.
- 粒度用一句话描述,不是一段话
- 每个 `field_mapping` 条目都被恰好一个 step 引用
- 每个 join 都有 cardinality 假设
- 每个指标都有显式 `agg``alias`
- `pending_questions` 为空
否则:状态为 `NEED_USER_CONFIRMATION` 并停止。**不得**进入 `sql-context-builder`
+93 -90
View File
@@ -1,116 +1,118 @@
---
name: metadata-validator
description: Use when validating that a data requirement's candidate tables, fields, joins, and time columns actually exist in the project's metadata. Triggers include "validate metadata", "check if table exists", "field mapping", "join key validation", "确认表/字段", "校验 schema", "这个字段在哪张表", "口径对得上吗". Reads metadata files recursively across the workspace (JSON / Markdown / CSV / Excel / YAML / DDL / data dictionary) and produces a standardized Validation Result and Field Mapping.
description: 当需要校验某个数据需求中候选的表、字段、join 关联和时间字段是否真实存在于工作区元数据中时使用。触发词包括"validate metadata""check if table exists""field mapping""join key validation""确认表/字段""校验 schema""这个字段在哪张表""口径对得上吗"。递归扫描工作区中的元数据文件(JSON / Markdown / CSV / Excel / YAML / DDL / 数据字典),并产出标准化的 Validation Result Field Mapping
---
# Metadata Validator
# Metadata Validator(元数据校验器)
## Overview
## 概述(Overview)
Bridge between the requirement's *expected* tables/fields and the *actual* schema living in workspace metadata. Output is consumed verbatim by `logic-planner` and `sql-context-builder` — they must NOT re-validate.
本 skill 在需求的"期望表/字段"和工作区元数据中的"实际 schema"之间架起桥梁。输出会被 `logic-planner` `sql-context-builder` 原样消费,二者**不得**重复校验。
Core principle: **never guess a table name, field name, or join relationship.** If a candidate is not provably present in metadata, surface it as a `pending_question` and stop at status `NEED_USER_CONFIRMATION`.
核心原则:**绝不臆测表名、字段名或 join 关系。** 如果某候选在元数据中无法证实存在,将其作为 `pending_question` 暴露,状态停在 `NEED_USER_CONFIRMATION`
## When to Use
## 何时使用(When to Use)
Use when:
- `requirements-analysis` has produced candidate tables/fields and you need to confirm they exist in real metadata
- The user asks "does this table/field exist", "which file describes table X", "map business field to physical column"
- You need to detect join keys, time columns, or aggregation-suitable numeric columns
- The user is about to write SQL and you must guard against hallucinated column names
**使用场景:**
Do NOT use when:
- The user has already supplied exact table+column names and confirmed them — go straight to `logic-planner`
- The task is code generation without any data warehouse context
- `requirements-analysis` 已经产出候选表/字段,需要确认它们在真实元数据中存在
- 用户询问"这张表/这个字段是否存在"、"表 X 的元数据在哪份文件里"、"业务字段映射到哪个物理列"
- 需要检测 join key、时间列、可用于聚合的数值列
- 即将写 SQL,必须防止出现幻觉字段名
## Inputs
**不要使用场景:**
1. From `requirements-analysis`:
- `candidate_tables`: list of table names mentioned or implied
- `candidate_fields`: list of business field names with their proposed table
- `business_logic`: short summary of what is being computed
2. Workspace metadata: any file under the working directory matching metadata conventions (see below)
- 用户已经提供并确认了精确的表名+列名 — 直接进入 `logic-planner`
- 任务是无数据仓库上下文的纯代码生成
## Workflow
## 输入(Inputs)
1. 来自 `requirements-analysis` 的:
- `candidate_tables`: 用户提到或隐含的表名列表
- `candidate_fields`: 业务字段名及其建议所在表
- `business_logic`: 计算逻辑的简短描述
2. 工作区元数据:当前工作目录下任何符合元数据约定的文件(见下文)
## 工作流(Workflow)
```dot
digraph metadata_validator {
"Receive candidate tables/fields" [shape=box];
"Recursively scan workspace for metadata files" [shape=box];
"Parse each file into canonical (table, field, type) records" [shape=box];
"Check candidate tables exist" [shape=box];
"Check candidate fields exist in their table" [shape=box];
"Fuzzy match missing fields against parsed metadata" [shape=box];
"Detect join keys & time columns" [shape=box];
"All checks pass?" [shape=diamond];
"Build Field Mapping" [shape=box];
"Emit Validation Result (VALIDATED)" [shape=box];
"Emit Validation Result (NEED_USER_CONFIRMATION)" [shape=box];
"接收候选表/字段" [shape=box];
"递归扫描工作区中的元数据文件" [shape=box];
"将每个文件解析为规范化的 (table, field, type) 记录" [shape=box];
"检查候选表是否存在" [shape=box];
"检查候选字段是否在其所属表中存在" [shape=box];
"对缺失字段做模糊匹配" [shape=box];
"检测 join key 与时间列" [shape=box];
"所有检查都通过?" [shape=diamond];
"构建 Field Mapping" [shape=box];
"输出 Validation Result (VALIDATED)" [shape=box];
"输出 Validation Result (NEED_USER_CONFIRMATION)" [shape=box];
"Receive candidate tables/fields" -> "Recursively scan workspace for metadata files";
"Recursively scan workspace for metadata files" -> "Parse each file into canonical (table, field, type) records";
"Parse each file into canonical (table, field, type) records" -> "Check candidate tables exist";
"Check candidate tables exist" -> "Check candidate fields exist in their table";
"Check candidate fields exist in their table" -> "Fuzzy match missing fields against parsed metadata";
"Fuzzy match missing fields against parsed metadata" -> "Detect join keys & time columns";
"Detect join keys & time columns" -> "All checks pass?";
"All checks pass?" -> "Build Field Mapping" [label="yes"];
"All checks pass?" -> "Emit Validation Result (NEED_USER_CONFIRMATION)" [label="no"];
"Build Field Mapping" -> "Emit Validation Result (VALIDATED)";
"接收候选表/字段" -> "递归扫描工作区中的元数据文件";
"递归扫描工作区中的元数据文件" -> "将每个文件解析为规范化的 (table, field, type) 记录";
"将每个文件解析为规范化的 (table, field, type) 记录" -> "检查候选表是否存在";
"检查候选表是否存在" -> "检查候选字段是否在其所属表中存在";
"检查候选字段是否在其所属表中存在" -> "对缺失字段做模糊匹配";
"对缺失字段做模糊匹配" -> "检测 join key 与时间列";
"检测 join key 与时间列" -> "所有检查都通过?";
"所有检查都通过?" -> "构建 Field Mapping" [label="yes"];
"所有检查都通过?" -> "输出 Validation Result (NEED_USER_CONFIRMATION)" [label="no"];
"构建 Field Mapping" -> "输出 Validation Result (VALIDATED)";
}
```
## Metadata Discovery
## 元数据发现(Metadata Discovery)
- Recursively scan the **current working directory** (and any subdirectories) — do not assume a `metadata/` folder exists.
- Filenames often resemble table names (e.g. `loan_order.json`, `customer_info.md`, `dim_product.csv`) but **never trust the filename alone**open the file and confirm the table identifier inside.
- Accepted formats (auto-detect by content, not extension):
- 递归扫描**当前工作目录**(及其所有子目录) — 不要假设存在 `metadata/` 目录
- 文件名常常形似表名(例如 `loan_order.json``customer_info.md``dim_product.csv`),但**绝不要只信任文件名** — 必须打开文件,确认里面的表标识符
- 支持的格式(按内容自动识别,不按扩展名):
- JSON / JSONL
- YAML
- Markdown tables / headings
- CSV (with header)
- Excel (`.xlsx`, `.xls`) — use `openpyxl` or `pandas.read_excel`
- DDL / `CREATE TABLE` statements
- Free-form data dictionaries (parse key-value lines or `field | type | comment` tables)
- Convert every file into the canonical record shape before validating:
- Markdown 表格 / 标题
- CSV(带表头)
- Excel(`.xlsx``.xls`) — `openpyxl` `pandas.read_excel`
- DDL / `CREATE TABLE` 语句
- 自由形式的数据字典(解析键值行或 `field | type | comment` 表格)
- 在校验之前,先把每个文件转成规范化的记录格式:
```yaml
tables:
- table: <physical_table_name>
source_file: <relative_path>
- table: <物理表名>
source_file: <相对路径>
fields:
- name: <field_name>
- name: <字段名>
type: <string|int|long|double|decimal|date|timestamp|boolean|...>
comment: <optional>
comment: <可选>
```
## Validation Rules
## 校验规则(Validation Rules)
Run **all** of the following. Any failure ⇒ status `NEED_USER_CONFIRMATION`.
**全部**以下规则都要执行。任意一条失败 ⇒ 状态为 `NEED_USER_CONFIRMATION`
1. **Table existence** — every candidate table must appear in the parsed metadata.
2. **Field existence** — every candidate field must appear in its candidate table.
3. **Field ownership** — a field may exist but belong to a different table; never silently swap.
4. **Type sanity** — aggregation targets (sum / avg / count) must be numeric; time filters must be date/timestamp/string-encoded-date.
5. **Necessary fields** — flag if common fields are missing for the business logic (stat metric, time, join key, dimension).
6. **Fuzzy match** — for every missing field, score candidates by name similarity (`customer_id` ↔ `cust_id`, `cust_no`, `customer_no`) and Levenshtein distance; surface top-N.
7. **Join detection** — across the validated tables, look for shared `*_id`, `*_no`, `*_code` columns. If multiple plausible join keys exist, ask.
8. **Time column disambiguation** — list every time-like field per table (`create_time`, `apply_time`, `txn_date`, `update_time`, `dt`); ask which one defines the time grain.
9. **Coverage** — verify that dimensions, metrics, filters, and sort keys are all present.
1. **表存在性** — 每个候选表必须出现在解析后的元数据中
2. **字段存在性** — 每个候选字段必须出现在其候选所属表中
3. **字段归属** — 字段可能存在但属于另一张表;绝不能悄悄替换
4. **类型合理性** — 聚合目标(sum / avg / count)必须为数值型;时间过滤字段必须为 date/timestamp/字符串型日期
5. **必要字段** — 如果业务逻辑中常见字段缺失(统计指标、时间、join key、维度),要标记出来
6. **模糊匹配** — 对每个缺失字段,按名称相似度打分(`customer_id` ↔ `cust_id``cust_no``customer_no`) Levenshtein 距离,给出 Top-N
7. **join 检测** — 在已校验的表之间,查找共享的 `*_id``*_no``*_code` 列。如果存在多个可能的 join key,必须询问
8. **时间列消歧** — 列出每张表中所有时间类字段(`create_time``apply_time``txn_date``update_time``dt`);询问哪一个决定时间粒度
9. **覆盖度** — 验证维度、指标、过滤列、排序键是否都存在
## Forbidden Behaviors
## 禁止行为(Forbidden Behaviors)
| Rationalization | Reality |
| 自我说服 | 现实 |
|---|---|
| "The filename is `loan_order.json`, must be that table" | Filename is a hint. Parse the file to confirm. |
| "Field exists somewhere, so the join is fine" | Wrong table ownership invalidates the join. Always check ownership. |
| "The user probably means `cust_no`, just use it" | Surface as a candidate and ASK. Never auto-substitute. |
| "I'll guess the time field" | Multiple time fields ⇒ ASK. The grain defines the entire query. |
| "This metadata is enough, no need to scan further" | Always scan recursively. One file can describe many tables. |
| "文件名是 `loan_order.json`,肯定就是这张表" | 文件名只是提示,必须解析文件内容确认。 |
| "字段存在于某处,所以 join 没问题" | 错误的表归属会让 join 失效。永远要校验归属。 |
| "用户八成指的是 `cust_no`,直接用吧" | 必须作为候选暴露并询问,绝不能自动替换。 |
| "时间字段我猜一个就行" | 多个时间字段时必须询问。粒度决定整条查询。 |
| "这份元数据够了,不必再扫" | 永远要递归扫描。一个文件可能描述多张表。 |
## Output Contract
## 输出契约(Output Contract)
The output is a single YAML document. Downstream skills consume it as-is.
输出是一份单一的 YAML 文档,下游 skill 原样消费。
```yaml
validation_result:
@@ -137,26 +139,27 @@ validation_result:
- <field>
- <field>
metadata_sources:
- <relative_path_to_metadata_file>
- <相对路径指向元数据文件>
pending_questions:
- "<human-readable question, ideally with options A/B/C>"
- "<面向用户的问题,最好给出 A/B/C 选项>"
field_mapping:
<business_field_name>:
table: <physical_table>
column: <physical_field>
type: <data_type>
table: <物理表>
column: <物理字段>
type: <数据类型>
status: <VALIDATED | NEED_USER_CONFIRMATION>
```
`field_mapping` is the **single source of truth** for downstream `logic-planner` and `sql-context-builder`. If a business field has no mapping, the request cannot proceed — add it to `pending_questions`.
`field_mapping` 是下游 `logic-planner` `sql-context-builder` 的**唯一权威来源**。如果某个业务字段没有映射,该需求无法继续 — 把它加入 `pending_questions`
## Completion Criteria
## 完成判定(Completion Criteria)
Status `VALIDATED` is only allowed when **all** of the following hold:
- Every candidate table is present in `validated_tables`
- Every candidate field has a `field_mapping` entry
- All join keys are decided (or trivially obvious with `high` confidence)
- The time column for the query grain is decided
- `pending_questions` is empty
仅当**以下全部**成立时,才允许状态 `VALIDATED`:
Anything else ⇒ `NEED_USER_CONFIRMATION` and stop. Do not proceed to `logic-planner`.
- 每个候选表都出现在 `validated_tables` 中
- 每个候选字段都有 `field_mapping` 条目
- 所有 join key 都已确定(或显然到可以用 `high` 置信度推断)
- 查询粒度对应的时间列已确定
- `pending_questions` 为空
其余情况 ⇒ `NEED_USER_CONFIRMATION` 并停止。**不得**继续进入 `logic-planner`。
+154 -151
View File
@@ -1,103 +1,103 @@
---
name: pyspark-sql-guardrails
description: Use when writing or reviewing PySpark scripts, Spark SQL strings, spark.sql calls, SQL loaded from config files, ETL jobs, warehouse maintenance scripts, or data pipeline code where SQL must be limited to SELECT and INSERT only.
description: 在编写或评审 PySpark 脚本、Spark SQL 字符串、`spark.sql` 调用、从配置文件加载的 SQL、ETL 作业、数仓维护脚本,或任何必须把 SQL 限制为 SELECT INSERT 的数据流水线代码时使用。
---
# PySpark SQL Guardrails
# PySpark SQL Guardrails(PySpark SQL 防护栏)
## Validation Gate (Read This First — Non-Negotiable)
## 校验关口(先读这一段 — 不可妥协)
**Every SQL string this skill produces is UNTRUSTED until it has been run through the bundled validator. The validator runs in one command, immediately after the SQL is generated, and BEFORE the SQL is shown to the user. There is no path that skips this step.**
**本 skill 产出的每一条 SQL 字符串,在跑过自带的校验器之前都是不可信的。** 校验器只需要一条命令,在 SQL 生成之后、**展示给用户之前**立即执行。不存在绕过此步骤的路径。
The whole flow is two commands and one condition:
整条流程就是两条命令 + 一个条件:
```bash
# Step 1 — Generate the SQL (do this in your head / draft)
# Step 2 — Run the validator on it (do this IMMEDIATELY, in the same turn)
echo "<the SQL you just wrote>" | python3 .claude/skills/pyspark-sql-guardrails/scripts/validate_sql.py
# 第 1 步 — 生成 SQL(脑中起草 / 写出来)
# 第 2 步 — 立即把 SQL 喂给校验器(在同一轮里跑)
echo "<the SQL you just wrote>" | python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py
# Step 3 — Read the output:
# PASS - first keyword: select, body length: N → you may now show the SQL
# FAIL - <reason> → STOP. Fix the SQL. Re-run.
# 第 3 步 — 读输出:
# PASS - first keyword: select, body length: N → 可以把 SQL 给用户看了
# FAIL - <reason> → 停下,修 SQL,再跑一次
```
**If you have not pasted the `PASS -` line into the conversation yet, you are not allowed to paste the SQL.** The validator is the gate. The SQL sits on the other side. Do not pass the gate empty-handed.
**如果你还没把 `PASS -` 这一行贴到对话里,就不允许贴 SQL** 校验器就是关口,SQL 站在关口的另一边,绝不能空着手闯关。
This rule is enforced by the skill, not by an external system. Honoring it is a precondition for any output that contains a SQL string.
这条规则由本 skill 自己强制,而非外部系统。遵守它是任何含 SQL 字符串的输出的前提。
---
## Overview
## 概述(Overview)
PySpark SQL in this environment is allowlist-only: generated or executed SQL may be `SELECT` or `INSERT` only. Treat every SQL string, formatted SQL template, config-loaded SQL, notebook cell, and `spark.sql(...)` call as unsafe until validated.
本环境下的 PySpark SQL 只接受白名单:生成或执行的 SQL 只能是 `SELECT` `INSERT`。在通过校验之前,所有 SQL 字符串、格式化后的 SQL 模板、从配置加载的 SQLnotebook 单元格,以及 `spark.sql(...)` 调用一律视为不安全。
Core principle: **never execute SQL by checking that it is “not obviously dangerous”; execute only after proving it is one allowed statement whose first real keyword is SELECT or INSERT.**
核心原则:**绝不能靠"看上去不像危险"来执行 SQL;只有在证明它就是一条以 SELECT INSERT 开头的、合法的语句后,才能执行。**
## Mandatory Validation (Hard Rule)
## 强制校验(Hard Rule)
**Every SQL string MUST pass through `assert_select_or_insert()` before it can reach `spark.sql(...)` — no exceptions.** This is non-negotiable and applies to:
**每条 SQL 字符串在到达 `spark.sql(...)` 之前都必须通过 `assert_select_or_insert()` — 不许有例外。** 这条规则不可妥协,适用于:
- SQL you just generated (literal, f-string, `.format(...)`, `+` concatenation)
- SQL loaded from YAML / JSON / INI / env / CLI / config files
- SQL passed in via parameters, function arguments, or notebook variables
- SQL pasted in from a chat message
- SQL produced by a templating engine (Jinja, string.Template, etc.)
- 你刚生成的 SQL(字面量、f-string`.format(...)``+` 拼接)
- YAML / JSON / INI / 环境变量 / 命令行 / 配置文件加载的 SQL
- 通过参数、函数参数、notebook 变量传入的 SQL
- 从聊天消息里粘贴进来的 SQL
- 由模板引擎(Jinjastring.Template 等)产出的 SQL
The only acceptable call shapes:
唯一可接受的调用形式:
```python
# Preferred: wrapper that validates + executes atomically
# 推荐:封装函数,校验+执行一步到位
safe_spark_sql(spark, sql)
# Or: validate first, then execute explicitly
assert_select_or_insert(sql) # raises on violation
# 或者:先校验,再显式执行
assert_select_or_insert(sql) # 违规即抛错
spark.sql(sql)
```
**Forbidden call shapes (any of these ⇒ pipeline failure):**
**禁止的调用形式(出现即视为流水线失败):**
```python
spark.sql(sql) # direct, no validation
spark.sql(f"SELECT ... {user_input} ...") # f-string into spark.sql
spark.sql(config["sql"]) # config-driven without validation
spark.sql(open("queries/xxx.sql").read()) # file-loaded without validation
spark.sql(sql) # 直接调用,未校验
spark.sql(f"SELECT ... {user_input} ...") # f-string 直接进 spark.sql
spark.sql(config["sql"]) # 配置驱动的 SQL 未校验
spark.sql(open("queries/xxx.sql").read()) # 从文件加载的 SQL 未校验
```
If `assert_select_or_insert()` raises, the pipeline stops. Do not weaken the rule, do not strip forbidden keywords with regex, do not "rewrite" the SQL into a safe-looking form. Reject and re-author.
如果 `assert_select_or_insert()` 抛错,流水线立即停止。**不要**削弱规则,**不要**用正则去剥除禁用关键字,**不要**把 SQL"改写"成看似安全的样子。要么拒绝,要么重新生成。
## Required Rule
## 必守规则(Required Rule)
Allowed:
**允许:**
- `SELECT ...`
- `WITH ... SELECT ...`
- `INSERT INTO ... SELECT ...`
- `INSERT OVERWRITE ... SELECT ...` only if the user explicitly allows overwrite for this job; otherwise ask before using it.
- `INSERT OVERWRITE ... SELECT ...` — 仅当用户明确允许本次任务使用 overwrite,否则必须先问
Forbidden, even for maintenance or metadata refresh:
- `DELETE`, `TRUNCATE`, `DROP`, `ALTER`, `CREATE`, `REPLACE`, `MERGE`, `UPDATE`
- `MSCK REPAIR`, `REFRESH`, `ANALYZE`, `CACHE`, `UNCACHE`, `VACUUM`, `OPTIMIZE`
- Multiple statements separated by semicolons
- Arbitrary SQL from YAML/JSON/env/CLI passed directly to `spark.sql`
**禁止**(即便是维护或元数据刷新):
- `DELETE``TRUNCATE``DROP``ALTER``CREATE``REPLACE``MERGE``UPDATE`
- `MSCK REPAIR``REFRESH``ANALYZE``CACHE``UNCACHE``VACUUM``OPTIMIZE`
- 用分号分隔的多条语句
- YAML/JSON/env/CLI 来的、未经任何处理直接喂给 `spark.sql` 的 SQL
**Execution gate:** `assert_select_or_insert(sql)` (or `safe_spark_sql(spark, sql)`) MUST be called for **every** `spark.sql` invocation. There is no fast path. See *Mandatory Validation (Hard Rule)* above.
**执行关口:** 每次 `spark.sql` 调用都必须先调 `assert_select_or_insert(sql)`(或 `safe_spark_sql(spark, sql)`)。没有快速通道。详见上面的 *强制校验(Hard Rule)*
## Quick Reference
## 速查表(Quick Reference)
| Situation | Do |
| 场景 | 做法 |
|---|---|
| Need rows | `safe_spark_sql(spark, "SELECT ...")` |
| Need to write rows | `safe_spark_sql(spark, "INSERT INTO table SELECT ...")` |
| Need cleanup/dedup/delete | Use SELECT to derive clean data, then `safe_spark_sql` with INSERT into an approved target/staging table; do not delete old data |
| Need schema/table/partition maintenance | Stop and ask; do not emit DDL or metadata SQL |
| SQL comes from config | `assert_select_or_insert(sql)` first, **then** `spark.sql(sql)` |
| User asks “quick fix” | Safety rule still applies — run `assert_select_or_insert` first |
| Generated SQL from `sql-context-builder` | Always run `assert_select_or_insert` before executing |
| 想要取数 | `safe_spark_sql(spark, "SELECT ...")` |
| 想要写数 | `safe_spark_sql(spark, "INSERT INTO table SELECT ...")` |
| 想要清理/去重/删除 | 用 SELECT 派生干净数据,再用 `safe_spark_sql` + INSERT 写入已批准的目标/staging 表;**不要**直接删除旧数据 |
| 想要做 schema/表/分区维护 | 停下来询问用户;**不要**输出 DDL 或元数据 SQL |
| SQL 来自配置文件 | 先 `assert_select_or_insert(sql)`,**再** `spark.sql(sql)` |
| 用户要求"快速修一下" | 安全规则照旧 — 先跑 `assert_select_or_insert` |
| SQL 由 `sql-context-builder` 生成 | 执行前必须跑 `assert_select_or_insert` |
## Safe Pattern
## 安全模板(Safe Pattern)
**MUST:** every `spark.sql` call site uses `safe_spark_sql` (or calls `assert_select_or_insert` immediately before `spark.sql`). The validator is the only thing standing between a generated string and the cluster.
**必须:** 每个 `spark.sql` 调用点都走 `safe_spark_sql`(或在 `spark.sql` 之前立即调 `assert_select_or_insert`)。在校验器面前,一条生成的 SQL 和集群之间只隔这一道闸。
Use validation at the boundary where SQL enters execution:
在 SQL 进入执行的那条边界做校验:
```python
import re
@@ -115,7 +115,7 @@ def assert_select_or_insert(sql: str) -> str:
if not text:
raise ValueError("SQL is empty")
# Allow one optional trailing semicolon, but reject stacked statements.
# 允许一个可选的尾随分号,但拒绝多条堆叠的语句
body = text[:-1].strip() if text.endswith(";") else text
if ";" in body:
raise ValueError("Multiple SQL statements are not allowed")
@@ -138,42 +138,42 @@ def assert_select_or_insert(sql: str) -> str:
def safe_spark_sql(spark, sql: str):
return spark.sql(assert_select_or_insert(sql))
# Good
# 正确
safe_spark_sql(spark, """
INSERT INTO analytics.daily_customer_snapshot
SELECT * FROM staging.daily_customer_snapshot
""")
# Bad: raises before Spark sees it
# 错误:在 Spark 看到 SQL 之前就抛错
safe_spark_sql(spark, "ALTER TABLE analytics.daily_customer_snapshot DROP PARTITION (dt='2026-06-10')")
```
If the project already has a SQL parser such as `sqlglot`, prefer AST validation over regex. Still keep the same allowlist: one statement, top-level SELECT or INSERT, no forbidden DDL/DML/maintenance commands anywhere.
如果项目里已经有 `sqlglot` 之类的 SQL 解析器,优先用 AST 校验而非正则。但白名单要保持一致:只能有一条语句、顶层是 SELECT INSERT、不能出现任何禁用的 DDL/DML/维护命令。
## Local Verification Scaffold (Run After Every Generation)
## 本地验证脚手架(每次生成 SQL 后都要跑)
The validator is useless if it is never executed. After **every** SQL string is produced (whether by you, a sub-agent, a template, or a tool), the next action is to run it through `assert_select_or_insert` in Python and only then show the result to the user.
校验器如果从来不被运行,就是废物。每当产出一条 SQL 字符串(无论由你、子 agent、模板或工具产出),下一步就是在 Python 里用 `assert_select_or_insert` 跑一遍,然后才把结果给用户看。
This skill ships its own validator and test set inside the skill directory so the contract is self-contained and versioned with the skill itself. **Do not redefine the regex inline at call sites. Always import from the bundled file.**
skill 把自己的校验器和测试集内置在 skill 目录里,使契约自包含且与 skill 一起版本化。**不要**在调用点重新定义正则,永远从内置文件 import。
```
.claude/skills/pyspark-sql-guardrails/
├── SKILL.md ← this file
skills/pyspark-sql-guardrails/
├── SKILL.md ← 本文件
└── scripts/
├── sql_guard.py ← canonical validator (assert_select_or_insert + safe_spark_sql)
├── validate_sql.py ← one-line CLI: pipe SQL in, get PASS/FAIL out
└── test_sql_guard.py ← 18-case standard test set
├── sql_guard.py ← 权威校验器(assert_select_or_insert + safe_spark_sql)
├── validate_sql.py ← 一行 CLI:管道喂 SQL,输出 PASS/FAIL
└── test_sql_guard.py ← 18 用例的标准测试集
```
### Step 1 — Use the bundled validator (no copy-pasting, no rewriting)
### 第 1 步 — 用内置校验器(不复制、不重写)
The skill ships a one-line CLI at `validate_sql.py`. Use it. Do not write a different invocation, do not paste inline regex, do not call `assert_select_or_insert` directly from a sub-agent unless the bundled CLI is also broken. The CLI is a thin wrapper around `assert_select_or_insert`; the function it calls is the source of truth.
skill 自带一个一行 CLI:`validate_sql.py`。用它。不要写不同的调用方式,不要贴内联正则,不要让子 agent 直接调 `assert_select_or_insert`,除非这个 CLI 也坏了。CLI 是 `assert_select_or_insert` 的薄壳,它调用的函数才是真正的权威。
**Two ways to feed SQL into the validator:**
**两种把 SQL 喂给校验器的方法:**
```bash
# Way A (preferred for multi-line): pipe via stdin
cat <<'EOF' | python3 .claude/skills/pyspark-sql-guardrails/scripts/validate_sql.py
# 方法 A(推荐,适合多行):走 stdin
cat <<'EOF' | python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py
SELECT
c.city AS city,
SUM(o.loan_amount) AS total_loan
@@ -183,19 +183,19 @@ WHERE o.status = 'SUCCESS'
GROUP BY c.city
EOF
# Way B (single-line only): pass as first argument
python3 .claude/skills/pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dual"
# 方法 B(仅适合单行):作为第一个参数传入
python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dual"
```
**Exit codes:**
**退出码:**
- `0``PASS - first keyword: <verb>, body length: <N>`
- `1``FAIL - <reason>` (the reason names the rule that was violated)
- `2`usage error (no SQL provided)
- `1``FAIL - <reason>`(reason 会指明被违反的规则)
- `2`用法错误(没有提供 SQL)
The bundled `sql_guard.py` and `validate_sql.py` together:
内置的 `sql_guard.py` `validate_sql.py` 一并展示:
```python
# sql_guard.py (bundled)
# sql_guard.py (内置)
import re
_FORBIDDEN_SQL = re.compile(
@@ -227,7 +227,7 @@ def safe_spark_sql(spark, sql: str):
return spark.sql(assert_select_or_insert(sql))
# validate_sql.py (bundled, abridged)
# validate_sql.py (内置,精简版)
import sys, os
SKILL_DIR = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, SKILL_DIR)
@@ -244,120 +244,123 @@ def main():
return 0
```
### Step 2 — Required call pattern after every SQL generation
### 第 2 步 — 每次生成 SQL 后的必用调用模式
The same command, every time, with the SQL you just produced. No variation, no shortcut, no "I'll run it later":
同一条命令,每次都用,SQL 改成你刚生成的那一条。不许变体,不许走捷径,不许说"待会儿再跑":
```bash
echo "<THE_SQL_YOU_JUST_GENERATED>" | python3 .claude/skills/pyspark-sql-guardrails/scripts/validate_sql.py
echo "<THE_SQL_YOU_JUST_GENERATED>" | python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py
```
**Show the validator's output to the user before showing the SQL.** The expected reply shape:
**先把校验器的输出贴给用户,再贴 SQL** 预期的回复样式:
```text
PASS - first keyword: select, body length: 420
```
If you see `FAIL - <reason>` instead, the SQL is not allowed past the gate. Surface the failure to the user verbatim, fix the SQL, re-run the validator. Loop until `PASS`.
如果出现的是 `FAIL - <reason>`,SQL 不允许通过关口。把失败原样报给用户,修 SQL,再跑校验器。循环到 `PASS` 为止。
### Step 3 — Standard test set (run on any change to the validator)
### 第 3 步 — 标准测试集(每次改校验器时跑)
The skill bundles `test_sql_guard.py` with 6 EXPECT_OK + 12 EXPECT_RAISE cases. Run it from the skill directory any time the validator changes, and as part of CI:
skill 自带 `test_sql_guard.py`,含 6 个 EXPECT_OK + 12 EXPECT_RAISE 用例。每次校验器改动,以及在 CI 中,都要在 skill 目录下跑一遍:
```bash
# From project root
python3 .claude/skills/pyspark-sql-guardrails/scripts/test_sql_guard.py
# 在项目根目录
python3 skills/pyspark-sql-guardrails/scripts/test_sql_guard.py
# Or from the scripts/ directory
cd .claude/skills/pyspark-sql-guardrails/scripts
# 或在 scripts/ 目录下
cd skills/pyspark-sql-guardrails/scripts
python3 test_sql_guard.py
```
Expected output: 6 `[OK] EXPECT_OK` lines, 12 `[OK] EXPECT_RAISE: raised as expected` lines, terminated with `ALL TESTS PASSED`. Any deviation is a regression in the validator — fix the validator, not the tests.
预期输出:6 行 `[OK] EXPECT_OK`、12 行 `[OK] EXPECT_RAISE: raised as expected`,以 `ALL TESTS PASSED` 收尾。出现任何偏差都说明校验器有回归 — 修校验器,不是修测试。
### Step 4 — Failure protocol
### 第 4 步 — 失败处理协议
When the validator raises (in CLI form, the `FAIL - <reason>` line):
当校验器抛错(在 CLI 形式下就是 `FAIL - <reason>` 这一行):
1. **Stop the pipeline.** Do not continue to the next stage.
2. **Read the reason.** It names the rule that was violated (empty / multi-statement / wrong verb / forbidden keyword / WITH without SELECT).
3. **Do not patch the validator to accept the SQL.** The validator is the source of truth; the SQL is wrong.
4. **Do not strip forbidden keywords with regex** to "clean up" the SQL. Reject and re-author the SQL itself.
5. **Re-run the validator** with the corrected SQL. Loop until `PASS`.
1. **停下流水线**,不要继续到下一阶段
2. ** reason**,它会指明被违反的规则(空 / 多语句 / 错的 verb / 禁用关键字 / WITH 后没 SELECT)
3. **不要**为了放过这条 SQL 而去改校验器。校验器是权威,SQL 是错的
4. **不要**用正则剥除禁用关键字来"清理"SQL。要拒绝,要么重新生成 SQL
5. **重跑校验器**用修好的 SQL,循环到 `PASS`
### Forbidden Behaviors (validator execution)
### 禁止行为(校验器执行相关)
| Rationalization | Reality |
| 自我说服 | 现实 |
|---|---|
| "I'll just `spark.sql(sql)` and trust it" | That is what the guardrail is preventing. Run the validator. |
| "The validator output is in the previous turn" | Validators are stateful only via the function call. Re-run it. |
| "It's a one-liner, eyeballing is enough" | Eyeballing is exactly the failure mode this guardrail exists to prevent. |
| "I already ran it on a similar SQL" | Each SQL string is a new value. Re-run on the actual string being shipped. |
| "I'll inline a regex instead of using `validate_sql.py`" | The skill ships one canonical CLI. Inline copies drift and miss updates. |
| "The SQL is in a code block, not yet 'executed'" | Showing the SQL to the user IS the execution. The gate is before the code block, not after. |
| "我就 `spark.sql(sql)`,信它" | 这正是防护栏要拦的。跑校验器。 |
| "校验器的输出在上一轮里" | 校验器只有通过函数调用才"有状态",再跑一遍。 |
| "就一行,肉眼看看就够了" | 肉眼判断正是这个防护栏要防的失败模式。 |
| "我已经对一条类似的 SQL 跑过了" | 每条 SQL 字符串都是新值,要针对实际要交付的字符串重跑。 |
| "我直接内联一段正则,不用 `validate_sql.py`" | skill 只给一个权威 CLI。内联副本会漂移、错过更新。 |
| "SQL 在代码块里,还没'执行'" | 把 SQL 展示给用户就已经是执行了。关口在代码块之前,不在之后。 |
## Common Rationalizations
## 常见自我说服(Common Rationalizations)
| Excuse | Reality |
| 借口 | 现实 |
|---|---|
| “It is just metadata refresh.” | `REFRESH`, `MSCK`, and `ANALYZE` are outside SELECT/INSERT. Ask first. |
| MERGE is non-destructive.” | Policy allows SELECT/INSERT only. `MERGE` is forbidden. |
| DELETE+INSERT is the existing pattern.” | Existing unsafe patterns do not override the rule. |
| “The config is trusted.” | Config SQL still needs allowlist validation before `spark.sql`. |
| “I can strip forbidden words with regex.” | Do not rewrite unsafe SQL into safe-looking SQL. Reject it. |
| “We need cleanup before insert.” | Build cleaned data with SELECT and write with INSERT; do not mutate/delete existing data. |
| **“The SQL is hard-coded, no need to validate.”** | **Hard-coded strings still must pass through `assert_select_or_insert`. The function checks the *string*, not its provenance. A literal in source code is no safer than a config value.** |
| **“I already eyeballed it, the verb is SELECT.”** | **Eyeballing is exactly the failure mode this guardrail prevents. Trust the function, not the eye. If `assert_select_or_insert` was not called, the SQL is untrusted by definition.** |
| **“I'll wrap it later / in a follow-up PR.”** | **No. Add the wrapper at the call site *now*. A naked `spark.sql(...)` in committed code is a regression, not a TODO.** |
| "只是元数据刷新而已" | `REFRESH``MSCK``ANALYZE` 都不在 SELECT/INSERT 范围,先问。 |
| "MERGE 是非破坏性的" | 策略只允许 SELECT/INSERT,`MERGE` 一律禁止。 |
| "DELETE+INSERT 是已有模式" | 已有不安全模式不能凌驾于规则之上。 |
| "配置是可信的" | 配置里的 SQL 在 `spark.sql` 前一样要过白名单。 |
| "我可以用正则剥除禁用词" | 不要把不安全的 SQL 改写成看似安全的样子,直接拒绝。 |
| "我们要在 insert 之前做清理" | 用 SELECT 派生干净数据,再用 INSERT 写入;**不要**变更/删除已有数据。 |
| **"SQL 是硬编码的,不用校验。"** | **硬编码字符串一样要走 `assert_select_or_insert`。该函数检查的是字符串本身,不是它的出处。源码里的字面量并不比配置里的值更安全。** |
| **"我肉眼已经看过了,verb SELECT。"** | **肉眼判断正是这个防护栏要防的失败模式。信函数,别信眼睛。`assert_select_or_insert` 没跑过,SQL 在定义上就是不可信的。** |
| **"我到下一轮/下一个 PR 再包。"** | **不行,现在就在调用点加上 wrapper。已提交代码里裸 `spark.sql(...)` 是回归,不是 TODO** |
## Red Flags
## 红旗信号(Red Flags)
Stop and add validation or ask the user when you see:
- `spark.sql(config["sql"])`, YAML SQL, env SQL, CLI SQL, or f-string SQL from external input
- Any SQL verb other than SELECT/WITH/INSERT
- `INSERT OVERWRITE` without explicit approval for overwrite semantics
- Semicolon-separated SQL batches
- Partition repair, schema migration, stale partition cleanup, compaction, vacuum, or stats collection requests
- Requests phrased as “quick cleanup”, “just refresh metadata”, “drop old partitions”, or “reuse DELETE+INSERT”
- **A direct `spark.sql(sql)` call with no preceding `assert_select_or_insert(sql)` (or no use of `safe_spark_sql`) — even if the SQL string is a hard-coded literal**
- **A pull request, diff, or code review that introduces a new `spark.sql(...)` site without the wrapper**
看到下面任一项就要停下,加校验或先问用户:
## Common Mistakes
- `spark.sql(config["sql"])`、YAML SQL、env SQL、CLI SQL,或来自外部输入的 f-string SQL
- 任何不是 SELECT/WITH/INSERT 的 SQL verb
- `INSERT OVERWRITE` 没拿到针对 overwrite 语义的明确批准
- 用分号分批的 SQL
- 分区修复、schema 迁移、过期分区清理、合并压实、vacuum、统计信息收集的请求
- 措辞为"快速清理一下"、"就刷一下元数据"、"删掉旧分区"、"复用 DELETE+INSERT" 的请求
- **直接的 `spark.sql(sql)` 调用,前面没有 `assert_select_or_insert(sql)`(也没用 `safe_spark_sql`)— 即便 SQL 字符串是硬编码字面量**
- **某条 PR、diff 或代码评审引入了新的 `spark.sql(...)` 调用点,却没加 wrapper**
- Checking only for `DELETE` and `TRUNCATE`; Spark risk also includes `DROP`, `ALTER`, `MERGE`, `UPDATE`, `MSCK`, `REFRESH`, `ANALYZE`, `VACUUM`, and `OPTIMIZE`.
- Validating generated SQL but not config-loaded SQL.
- Allowing multiple statements because the first statement is SELECT.
- Treating comments as harmless while a later statement contains forbidden SQL.
- Using `CREATE OR REPLACE TEMP VIEW` through SQL; prefer DataFrame `createOrReplaceTempView` if a temp view is needed.
- **Calling `spark.sql(sql)` directly without routing through `assert_select_or_insert` / `safe_spark_sql`. The allowlist is enforced at the call site, not at SQL-generation time. A generated SQL that *looks* safe is not *validated* safe until the function has run on it.**
- **Writing inline `if sql.startswith("SELECT"): spark.sql(sql)` checks. That is not validation. Only `assert_select_or_insert` (or an equivalent AST parser with the same allowlist) counts.**
## 常见错误(Common Mistakes)
- 只检查 `DELETE``TRUNCATE`;Spark 的风险还包括 `DROP``ALTER``MERGE``UPDATE``MSCK``REFRESH``ANALYZE``VACUUM``OPTIMIZE`
- 只校验了生成的 SQL,没校验配置里加载的 SQL
- 因为首条语句是 SELECT,就允许了多语句堆叠
- 把注释当作无害,但后面跟着一句禁用的 SQL
- 在 SQL 里用 `CREATE OR REPLACE TEMP VIEW`;如果需要临时视图,改用 DataFrame 的 `createOrReplaceTempView`
- **直接调 `spark.sql(sql)`,不走 `assert_select_or_insert` / `safe_spark_sql`。白名单在调用点生效,不在 SQL 生成时生效。一条"看起来"安全的 SQL,在函数跑过它之前都不算"已校验"安全。**
- **写内联的 `if sql.startswith("SELECT"): spark.sql(sql)`。这不是校验,只有 `assert_select_or_insert`(或等价且白名单一致的 AST 解析器)才算。**
---
## Pipeline Position
## 在流水线中的位置(Pipeline Position)
This skill is the **write-time / execution boundary** of the PySpark SQL pipeline:
skill 是 PySpark SQL 流水线的**写入/执行关口**:
```
requirements-analysis → metadata-validator → logic-planner → sql-context-builder → pyspark-sql-guardrails → sql-review
```
**Required upstream skill:** `sql-context-builder`every generated SQL must come with a `sql_context` that froze aliases, join keys, and field sources. SQL without an associated context is a `HIGH` finding under `sql-review` (see check #1: reference integrity).
**强依赖的上游 skill:** `sql-context-builder`每条生成的 SQL 必须带有一份固化了 aliasjoin key 和字段来源的 `sql_context`。没有伴随 context 的 SQL 在 `sql-review` 中会命中 `HIGH` 风险(见 check #1:reference integrity)
**Required downstream skill:** `sql-review`this guardrail validates *what statements are allowed*; `sql-review` validates *whether the allowed statement is correct*. Both must pass before `spark.sql(...)` is called. Do not skip `sql-review` because the guardrail passed.
**强依赖的下游 skill:** `sql-review`本防护栏校验"允许哪些语句";`sql-review` 校验"这条被允许的语句是否正确"。两者都通过之后才能 `spark.sql(...)`。**不要**因为防护栏过了就跳过 `sql-review`
**Execution gate (enforced by this skill):** the only path from a SQL string to the cluster is `assert_select_or_insert(sql)` (or its wrapper `safe_spark_sql(spark, sql)`). A direct `spark.sql(sql)` is a pipeline violation. This rule overrides the pipeline ordering — even if `sql-review` has already passed, the SQL still has to pass through this guardrail at execution time. Guardrail and reviewer are independent checks; one does not satisfy the other.
**执行关口(本 skill 强制):** SQL 字符串通往集群的唯一通道是 `assert_select_or_insert(sql)`(或其封装 `safe_spark_sql(spark, sql)`)。直接的 `spark.sql(sql)` 是流水线违规。这条规则凌驾于流水线顺序 — 即便 `sql-review` 已经通过,执行时仍要过这一道防护栏。防护栏与评审是相互独立的检查,一道过不能代替另一道。
**The guardrail does not check:**
- Whether the column actually exists in the table (that is `sql-review`'s job via the SQL Context)
- Whether the join key is correct (that is `sql-review`'s job)
- Whether aggregation is duplicated or rows will inflate (that is `sql-review`'s job)
- Whether time/partition filters are present (that is `sql-review`'s job)
**本防护栏不检查:**
**The guardrail does check:**
- First real keyword is `SELECT`, `WITH ... SELECT`, or `INSERT ... SELECT`
- No stacked statements (`;`-separated batches)
- No forbidden verbs anywhere in the body
- `INSERT OVERWRITE` is gated on explicit user approval
- 列是否真的存在于表(那是 `sql-review` 通过 SQL Context 校验的事)
- join key 是否正确(那是 `sql-review` 的事)
- 聚合是否被重复、行是否会膨胀(那是 `sql-review` 的事)
- 是否带时间/分区过滤(那是 `sql-review` 的事)
When the guardrail raises, fix the SQL, do not weaken the guardrail. When in doubt, ask the user before generating `INSERT OVERWRITE` or any DDL.
**本防护栏检查:**
- 第一个真实关键字是 `SELECT``WITH ... SELECT``INSERT ... SELECT`
- 没有 `;` 堆叠的多语句
- 体内不出现任何禁用的 verb
- `INSERT OVERWRITE` 必须基于用户的明确批准
校验器抛错时,改 SQL,**不要**削弱校验器。拿不准时,在生成 `INSERT OVERWRITE` 或任何 DDL 之前先问用户。
+69 -66
View File
@@ -1,28 +1,30 @@
---
name: pyspark-sql-pipeline
description: Use when orchestrating the full PySpark SQL pipeline from a business requirement to a reviewed, ready-to-execute SQL string. Triggers include "完整跑一遍取数流程", "生成 SQL 并 review", "end-to-end 取数", "full pipeline", "从头到尾", "数据需求到 SQL". Coordinates requirements-analysis → metadata-validator → logic-planner → sql-context-builder → pyspark-sql-guardrails → sql-review as a single deterministic flow with explicit user-confirmation gates between stages. Does NOT execute SQL — it drives the pipeline to a reviewed SQL string.
description: 当需要把业务需求一路编排到"已被评审过、可执行的 SQL 字符串"时使用。触发词包括"完整跑一遍取数流程""生成 SQL 并 review""end-to-end 取数""full pipeline""从头到尾""数据需求到 SQL"。把 requirements-analysis → metadata-validator → logic-planner → sql-context-builder → pyspark-sql-guardrails → sql-review 串成一条确定性流水线,阶段之间用显式的用户确认关口隔开。**不执行 SQL**只把流水线驱动到一条被评审过的 SQL 字符串。
---
# PySpark SQL Pipeline
# PySpark SQL Pipeline(PySpark SQL 流水线)
## Overview
## 概述(Overview)
End-to-end orchestrator for turning a business data requirement into a reviewed SQL string. Each stage has a fixed input/output contract; between any two stages there is a **user-confirmation gate** when the upstream status is not green.
把一份业务数据需求一路编排到"已评审"SQL 字符串的端到端协调器。每个阶段有固定的输入/输出契约;任意两阶段之间,当上游状态不是绿色时,都设有**用户确认关口**。
Core principle: **never skip a stage, never let a stage quietly proceed past its own `NEED_USER_CONFIRMATION`.** The pipeline is a sequence of hand-offs, not a single agent's internal monologue.
核心原则:**绝不跳阶段,也绝不让任何阶段悄无声息地越过它自己的 `NEED_USER_CONFIRMATION`。** 流水线是一连串交接,不是某个 agent 的内心独白。
## When to Use
## 何时使用(When to Use)
Use when:
- The user gives a data requirement and wants SQL produced end-to-end (e.g. "统计近 30 天各城市贷款总额及客户数")
- The user asks to "run the full pipeline" or "从需求到 SQL 一条龙"
- A non-trivial SQL change touches multiple skills' contracts
**使用场景:**
Do NOT use when:
- The user only needs one stage (e.g. "review this SQL" → use `sql-review` directly)
- The user explicitly says "skip validation" or "just give me SQL fast" — surface the risk and ask once; if they confirm, you may collapse stages but you MUST still apply `pyspark-sql-guardrails` and `sql-review`
- 用户给出一个数据需求,希望端到端生成 SQL(例如"统计近 30 天各城市贷款总额及客户数")
- 用户要求"跑完整流水线"或"从需求到 SQL 一条龙"
- 一个非平凡的 SQL 改动会同时触碰多个 skill 的契约
## The Pipeline
**不要使用场景:**
- 用户只需要其中一个阶段(例如"评审这条 SQL" → 直接用 `sql-review`)
- 用户明确说"跳过校验"或"快点给我 SQL"— 主动说明风险并确认一次;若用户确认,可以合并阶段,但**仍必须**应用 `pyspark-sql-guardrails``sql-review`
## 流水线(The Pipeline)
```dot
digraph pipeline {
@@ -30,92 +32,93 @@ digraph pipeline {
"2. metadata-validator" [shape=box];
"3. logic-planner" [shape=box];
"4. sql-context-builder" [shape=box];
"5. SQL generation" [shape=box];
"5. SQL 生成" [shape=box];
"6. pyspark-sql-guardrails" [shape=box];
"7. sql-review" [shape=box];
"1. requirements-analysis" -> "2. metadata-validator" [label="status=READY_FOR_VALIDATION"];
"2. metadata-validator" -> "3. logic-planner" [label="status=VALIDATED"];
"3. logic-planner" -> "4. sql-context-builder" [label="status=PLANNED"];
"4. sql-context-builder" -> "5. SQL generation" [label="status=READY"];
"5. SQL generation" -> "6. pyspark-sql-guardrails" [label="SQL string"];
"4. sql-context-builder" -> "5. SQL 生成" [label="status=READY"];
"5. SQL 生成" -> "6. pyspark-sql-guardrails" [label="SQL 字符串"];
"6. pyspark-sql-guardrails" -> "7. sql-review" [label="guardrail pass"];
}
```
Every arrow is a **contract boundary**. Do not let a stage start before its predecessor has produced the expected status.
每条箭头都是**契约边界**。前序阶段没产出预期状态时,绝不允许下游阶段启动。
## Stage Contracts
## 阶段契约(Stage Contracts)
| # | Stage | Skill | Input | Output Status (green) | Output Status (stop) |
| # | 阶段 | Skill | 输入 | 绿色输出状态 | 停止状态 |
|---|---|---|---|---|---|
| 1 | Requirements | `requirements-analysis` | user message | `READY_FOR_VALIDATION` | `NEED_USER_CONFIRMATION` |
| 2 | Metadata | `metadata-validator` | requirements_output + workspace | `VALIDATED` | `NEED_USER_CONFIRMATION` |
| 3 | Logic | `logic-planner` | validation_result | `PLANNED` | `NEED_USER_CONFIRMATION` |
| 4 | Context | `sql-context-builder` | logic_plan + field_mapping | `READY` | `NEED_USER_CONFIRMATION` |
| 5 | Generate | (no skill — see "SQL Generation" below) | sql_context | SQL string | n/a |
| 6 | Guardrail | `pyspark-sql-guardrails` | SQL string | `pass` | `fail` (raise) |
| 7 | Review | `sql-review` | SQL string + sql_context | `verdict: PASS` | `verdict: FAIL` |
| 1 | 需求 | `requirements-analysis` | 用户消息 | `READY_FOR_VALIDATION` | `NEED_USER_CONFIRMATION` |
| 2 | 元数据 | `metadata-validator` | requirements_output + 工作区 | `VALIDATED` | `NEED_USER_CONFIRMATION` |
| 3 | 逻辑 | `logic-planner` | validation_result | `PLANNED` | `NEED_USER_CONFIRMATION` |
| 4 | 上下文 | `sql-context-builder` | logic_plan + field_mapping | `READY` | `NEED_USER_CONFIRMATION` |
| 5 | 生成 | (无独立 skill — 见下文"SQL 生成") | sql_context | SQL 字符串 | n/a |
| 6 | 防护栏 | `pyspark-sql-guardrails` | SQL 字符串 | `pass` | `fail`(抛错) |
| 7 | 评审 | `sql-review` | SQL 字符串 + sql_context | `verdict: PASS` | `verdict: FAIL` |
## Gate Rules (READ THESE CAREFULLY)
## 关口规则(Gate Rules,务必细读)
A stage that returns a stop status **must not** be followed by the next stage. Instead:
返回停止状态的阶段**不能**被下一阶段跟随。要做的:
1. Surface the `pending_questions` from that stage to the user.
2. Stop the pipeline.
3. After the user answers, re-run from the same stage (not from the beginning) using the user's answers as additional input.
4. Once the stage produces its green status, advance to the next stage.
1. 把该阶段的 `pending_questions` 暴露给用户
2. 停下流水线
3. 等用户回答后,从**同一个阶段**重跑(不是从头跑),把用户的新回答作为额外输入
4. 一旦该阶段产出绿色状态,推进到下一阶段
Stages are idempotent. Re-running a stage with the same inputs must produce the same output; with new inputs, it must produce a new output that supersedes the old one. Carry forward all prior `field_mapping`, `validation_result`, `logic_plan`, and `sql_context` as the durable artifacts of the pipeline.
阶段是幂等的。同一份输入重跑必须产出同样的输出;新输入则必须产出能覆盖旧输出的新输出。所有前置的 `field_mapping``validation_result``logic_plan``sql_context` 必须作为流水线的持久化产物一路带下来。
## SQL Generation (Stage 5)
## SQL 生成(阶段 5)
There is no dedicated skill for generating SQL. The pipeline performs this step directly, with these constraints:
本步没有独立的 skill。流水线直接执行,约束如下:
- Use the `sql_context` as the **only** source of field names, aliases, and join keys.
- Compose the SQL from the `logic_plan.steps` in order: `source``filter``join``transform``dedupe``window``aggregate``output`.
- Every `<alias>.<column>` in the output must appear in `sql_context.fields`.
- The generated SQL must be runnable through `pyspark-sql-guardrails.safe_spark_sql` without further edits.
- `sql_context` 是字段名、alias、join key 的**唯一**来源
- `logic_plan.steps` 的顺序组装 SQL:`source``filter``join``transform``dedupe``window``aggregate``output`
- 输出中每个 `<alias>.<column>` 都必须出现在 `sql_context.fields`
- 生成的 SQL 必须能在 `pyspark-sql-guardrails.safe_spark_sql` 中不修改地跑过
## User-Confirmation Messages
## 用户确认消息(User-Confirmation Messages)
Each gate should produce a single, structured message in this shape:
每个关口应产出一条结构化消息,格式如下:
```markdown
## Pipeline paused at: <stage name>
## Pipeline paused at: <阶段名>
**Status:** NEED_USER_CONFIRMATION
**Open questions:**
1. <question 1>
2. <question 2>
1. <问题 1>
2. <问题 2>
**Pending options:** A / B / C / other
**To resume:** answer the questions above, then say "继续" (continue).
**To resume:** 回答以上问题后,说"继续"。
```
Do not present a stage's output as final when its status is a stop status. Do not bury the `pending_questions` inside a longer narrative.
不要把某个阶段的输出当作最终结果,如果它的状态是停止状态。不要把 `pending_questions` 埋在冗长叙述里。
## Failure Handling
## 失败处理(Failure Handling)
- **Guardrail failure (`pyspark-sql-guardrails`)** — do not retry the same SQL. Return to Stage 5 and regenerate, this time respecting the guardrail.
- **Review verdict FAIL with HIGH findings** — return to Stage 5 (or earlier if a finding is about context, in which case return to Stage 4). Do not "patch" the SQL to silence a finding; address the root cause.
- **Review verdict FAIL with only MEDIUM/LOW** — surface the findings to the user. They decide whether to ship or fix.
- **Inconsistency between stages** (e.g. SQL references a column not in `sql_context`) — treat as a Stage 5 bug; the SQL was generated without honoring the context. Regenerate, do not amend.
- **防护栏失败(`pyspark-sql-guardrails`)** — 不要用同一条 SQL 重试。回到阶段 5 重新生成,这一次遵守防护栏
- **评审结论为 FAIL 且有 HIGH 风险** — 回到阶段 5(或更早,如果是 context 类的发现,就回到阶段 4)。**不要**"打个补丁"让发现项闭嘴;要解决根因
- **评审结论为 FAIL 但只有 MEDIUM/LOW** — 把发现项暴露给用户,由用户决定上线还是改
- **阶段间不一致**(例如 SQL 引用了 `sql_context` 里没有的列) — 视为阶段 5 的 bug;SQL 在生成时没遵守 context。**重新生成,不要修补**。
## Forbidden Behaviors
## 禁止行为(Forbidden Behaviors)
| Rationalization | Reality |
| 自我说服 | 现实 |
|---|---|
| "User asked for SQL fast, skip the gates" | Gates exist because skipping them produces wrong numbers. Ask the user once; if they confirm, you may collapse only adjacent non-validation stages. Never skip guardrail or review. |
| "Just one stage failed, no need to ask" | Every stop status is a question to the user. Always ask. |
| "I'll inline the validation into SQL generation" | Stages are separate so they can be re-run independently. Inlining couples them. |
| "Review found MEDIUM, ship it" | Surface the findings. The user — not the agent — decides. |
| "Field mapping is obvious, skip the validator" | Field mapping is the validator's job. Bypassing it is the #1 cause of hallucinated column names. |
| "Same status, just continue" | Status names are fixed strings. Re-check the actual status text, do not pattern-match on memory. |
| "用户要 SQL 快点,关口跳过" | 关口存在的原因就是跳过它们会让数字变错。先问一次;如果用户确认,可以只合并相邻的非校验阶段。**永远不能**跳过防护栏或评审。 |
| "只一个阶段失败,不必问" | 每一个停止状态都是要问用户的问题。永远要问。 |
| "我把校验内联到 SQL 生成里" | 阶段分开是为了能独立重跑,内联会把它们耦合死。 |
| "评审只发现 MEDIUM,直接上线" | 把发现项暴露给用户,决定权在用户,不在 agent。 |
| "字段映射太显然,跳过校验器" | 字段映射就是校验器的工作。绕过它就是幻觉字段名的头号原因。 |
| "状态一样,直接继续" | 状态名是固定字符串。重读实际状态文本,不要凭记忆做模式匹配。 |
## Completion Criteria
## 完成判定(Completion Criteria)
The pipeline is complete only when:
- All 7 stages have run in order
- `sql-review.verdict` is `PASS` (or the user has explicitly accepted remaining MEDIUM/LOW findings)
- The final SQL string, the `sql_context`, the `logic_plan`, the `validation_result`, and the `requirements_output` are all available to the user in the same response
流水线完成需要**全部**满足:
If the user later changes the requirement, restart at Stage 1 — do not patch downstream artifacts.
- 7 个阶段按序都跑过
- `sql-review.verdict``PASS`(或用户已显式接受剩余的 MEDIUM/LOW 发现)
- 同一份回复里同时向用户交付:最终 SQL 字符串、`sql_context``logic_plan``validation_result``requirements_output`
如果用户后续修改了需求,从阶段 1 重新开始 — **不要**打补丁修下游产物。
+91 -82
View File
@@ -1,99 +1,107 @@
---
name: sql-context-builder
description: Use when assembling the single trusted SQL context that the SQL generator and reviewer will consume. Triggers include "build SQL context", "固化 join key", "table aliases", "列出来自哪张表", "准备写 SQL", "context 已经齐了". Consumes a validated Logic Plan + Field Mapping and produces a normalized SQL Context with stable aliases, frozen join keys, and explicit fact/dimension tagging. The SQL generator MUST NOT re-derive any of this.
description: 当需要组装 SQL 生成器与评审器共同消费的、唯一可信的 SQL Context 时使用。触发词包括"build SQL context""固化 join key""table aliases""列出来自哪张表""准备写 SQL""context 已经齐了"。消费已校验的 Logic Plan Field Mapping,产出带有稳定 alias、固化 join key、显式 fact/dimension 标记的标准化 SQL Context。SQL 生成器**不得**再重新推导其中任何一项。
---
# SQL Context Builder
# SQL Context Builder(SQL 上下文构建器)
## Overview
## 概述(Overview)
Freeze every decision that the SQL generator would otherwise re-invent. Aliases, join keys, field sources, types, role tags (fact vs dimension vs filter), and time columns are all pinned here exactly once. The output is the **single trusted input** for `sql-review` and for any SQL-emitting step.
把 SQL 生成器本来要重复发明的所有决策一次性固化下来:alias、join key、字段来源、类型、角色标签(fact / dimension / filter)、时间列,都在这里只此一次地钉死。输出是 `sql-review` 及任何 SQL 输出步骤的**唯一可信输入**。
Core principle: **anything not in the SQL Context does not exist for downstream skills.** If a downstream skill needs something new, the request must go back through `metadata-validator` and `logic-planner` first.
核心原则:**任何不在 SQL Context 中的字段,下游 skill 视同不存在。** 如果下游 skill 还需要新内容,请求必须先回退到 `metadata-validator` `logic-planner`
## When to Use
## 何时使用(When to Use)
Use when:
- `logic-planner` returned status `PLANNED`
- The user is about to write Spark SQL and you need a normalized context (aliases, join keys, column sources)
- Multiple SQL drafts need to be generated against the *same* logic — context guarantees they stay consistent
**使用场景:**
Do NOT use when:
- `logic-plan.status` is `NEED_USER_CONFIRMATION` — go back to `logic-planner`
- The SQL is trivial (one table, no join) — you can still build the context, but skip if the user explicitly opts out
- The task is to translate the plan into SQL — that is a separate concern from this skill
- `logic-planner` 返回状态 `PLANNED`
- 用户即将写 Spark SQL,需要一份规范化的 context(alias、join key、字段来源)
- 同一份逻辑要产出多版 SQL 草稿 — context 保证它们保持一致
## Inputs
**不要使用场景:**
1. `logic_plan` from `logic-planner` (status `PLANNED`)
2. `field_mapping` from `metadata-validator` (status `VALIDATED`)
3. Original `validation_result` for table-level metadata (types, sources)
- `logic-plan.status``NEED_USER_CONFIRMATION` — 回到 `logic-planner`
- SQL 极简单(单表无 join) — 仍然建议构建 context,除非用户明确跳过
- 任务是把 plan 翻译成 SQL — 那是另一回事,不是本 skill 的职责
## Workflow
## 输入(Inputs)
1. 来自 `logic-planner``logic_plan`(状态 `PLANNED`)
2. 来自 `metadata-validator``field_mapping`(状态 `VALIDATED`)
3. 原始 `validation_result`(用于表级元数据:类型、来源)
## 工作流(Workflow)
```dot
digraph sql_context_builder {
"Read logic_plan + field_mapping + validation_result" [shape=box];
"Assign stable alias to every table" [shape=box];
"Freeze join keys (left_alias.field = right_alias.field)" [shape=box];
"Resolve every field to (alias, column, type, source_file)" [shape=box];
"Tag each field: dimension | metric | filter | time | join_key" [shape=box];
"Mark fact vs dimension tables" [shape=box];
"Verify Logic Plan fields all exist in context" [shape=box];
"All required fields present?" [shape=diamond];
"Emit SQL Context" [shape=box];
"Emit SQL Context + missing_fields" [shape=box];
"读取 logic_plan + field_mapping + validation_result" [shape=box];
"为每张表分配稳定 alias" [shape=box];
"固化 join key(left_alias.field = right_alias.field)" [shape=box];
"把每个字段解析为 (alias, column, type, source_file)" [shape=box];
"为每个字段打标签:dimension | metric | filter | time | join_key" [shape=box];
"标记事实表 / 维度表" [shape=box];
"校验 Logic Plan 的字段在 context 中全部存在" [shape=box];
"所有必需字段都齐全?" [shape=diamond];
"输出 SQL Context" [shape=box];
"输出 SQL Context + missing_fields" [shape=box];
"Read logic_plan + field_mapping + validation_result" -> "Assign stable alias to every table";
"Assign stable alias to every table" -> "Freeze join keys (left_alias.field = right_alias.field)";
"Freeze join keys (left_alias.field = right_alias.field)" -> "Resolve every field to (alias, column, type, source_file)";
"Resolve every field to (alias, column, type, source_file)" -> "Tag each field: dimension | metric | filter | time | join_key";
"Tag each field: dimension | metric | filter | time | join_key" -> "Mark fact vs dimension tables";
"Mark fact vs dimension tables" -> "Verify Logic Plan fields all exist in context";
"Verify Logic Plan fields all exist in context" -> "All required fields present?";
"All required fields present?" -> "Emit SQL Context" [label="yes"];
"All required fields present?" -> "Emit SQL Context + missing_fields" [label="no"];
"读取 logic_plan + field_mapping + validation_result" -> "为每张表分配稳定 alias";
"为每张表分配稳定 alias" -> "固化 join key(left_alias.field = right_alias.field)";
"固化 join key(left_alias.field = right_alias.field)" -> "把每个字段解析为 (alias, column, type, source_file)";
"把每个字段解析为 (alias, column, type, source_file)" -> "为每个字段打标签:dimension | metric | filter | time | join_key";
"为每个字段打标签:dimension | metric | filter | time | join_key" -> "标记事实表 / 维度表";
"标记事实表 / 维度表" -> "校验 Logic Plan 的字段在 context 中全部存在";
"校验 Logic Plan 的字段在 context 中全部存在" -> "所有必需字段都齐全?";
"所有必需字段都齐全?" -> "输出 SQL Context" [label="yes"];
"所有必需字段都齐全?" -> "输出 SQL Context + missing_fields" [label="no"];
}
```
## Rules
## 规则(Rules)
### Alias Assignment
- One alias per table, never reused.
- Fact table (the one with the primary metric) gets a short alias first (`o`, `f`, `t1`).
- Dimension tables get meaningful aliases (`c` for customer, `p` for product, `ch` for channel).
- Aliases are **frozen** here. The SQL generator and reviewer must not invent new ones.
### Alias 分配
### Join Key Freezing
- For every `logic_plan.steps[kind=join]`, freeze the exact predicate:
- 一张表一个 alias,绝不重用
- 事实表(承载主指标的表)优先拿到短 alias(`o``f``t1`)
- 维度表用有意义的 alias(`c` 代表 customer,`p` 代表 product,`ch` 代表 channel)
- alias **在此处固化**,SQL 生成器与评审器不得再发明新 alias
### Join Key 固化
- 对每条 `logic_plan.steps[kind=join]`,固化精确的谓词:
- `left_alias.col1 = right_alias.col2`
- Include join type (inner / left / right / full / semi / anti).
- Include cardinality assumption carried over from the Logic Plan.
- If two tables could join on multiple keys, freeze the chosen one here. Other candidates are out of scope for SQL.
- 包含 join 类型(inner / left / right / full / semi / anti)
- 沿用 Logic Plan 中的 cardinality 假设
- 如果两张表有多种可能的 join key,在此处固化为所选的那一个,其他候选不再纳入 SQL 范畴
### Field Resolution
For every field used in the Logic Plan, record:
- `alias` (which table it lives on)
- `column` (physical name)
- `type` (from `validation_result`)
- `source_file` (which metadata file it came from — for traceability)
- `role`: one of `dimension` | `metric` | `filter` | `time` | `join_key` | `derived`
### 字段解析(Field Resolution)
A field may have multiple roles (e.g. a `customer_id` that is both `join_key` and `dimension`); list all.
对 Logic Plan 用到的每个字段,记录:
### Fact vs Dimension Tagging
- The table containing the primary event/measure is `fact`.
- Tables that enrich (customer, product, channel, city) are `dimension`.
- One fact per query. If the Logic Plan implies multiple facts, raise a `pending_question` — that is a planning issue, not a context issue.
- `alias`(字段所在表)
- `column`(物理列名)
- `type`(取自 `validation_result`)
- `source_file`(元数据文件来源 — 用于追溯)
- `role`:取自 `dimension` | `metric` | `filter` | `time` | `join_key` | `derived` 之一
### Coverage Check
- Every step in `logic_plan.steps` must reference fields that exist in this context.
- Every `field_mapping` entry must appear in this context.
- Every `group_by` and `order_by` field must be tagged.
- Every metric in `aggregate` must have a `metric` tag.
- Every predicate in `filter` must reference a `filter` or `time` tag.
一个字段可以有多个 role(例如 `customer_id` 同时是 `join_key``dimension`),全部列出。
## Output Contract
### Fact vs Dimension 标记
- 承载主要事件/度量的表为 `fact`
- 起丰富作用的表(customer、product、channel、city)为 `dimension`
- 一条 query 只能有一个 fact。如果 Logic Plan 隐含多张事实表,作为 `pending_question` 抛出 — 那是规划问题,不是 context 问题
### 覆盖度校验
- `logic_plan.steps` 中的每一步引用的字段都必须存在于本 context
- 每条 `field_mapping` 都必须出现在本 context
- 每个 `group_by``order_by` 字段都必须有标签
- `aggregate` 中的每个指标都必须有 `metric` 标签
- `filter` 中的每条谓词都必须引用 `filter``time` 标签的字段
## 输出契约(Output Contract)
```yaml
sql_context:
@@ -149,21 +157,22 @@ sql_context:
status: <READY | NEED_USER_CONFIRMATION>
```
## Forbidden Behaviors
## 禁止行为(Forbidden Behaviors)
| Rationalization | Reality |
| 自我说服 | 现实 |
|---|---|
| "I'll let the SQL generator pick aliases" | Aliases are frozen here. Re-aliasing downstream breaks `sql-review`. |
| "The join key is obvious, skip the freeze" | The whole point of this skill is to freeze join keys. |
| "Field type doesn't matter, it's all strings in SQL" | Type drives aggregations, casts, and partition-pruning decisions. |
| "Derived columns can be invented in SQL" | Every derived column must already appear in the Logic Plan. |
| "Multiple facts? Just join them all" | Multi-fact queries are a Logic Plan issue. Surface and stop. |
| "alias 让 SQL 生成器自己选" | alias 就在此处固化。下游重新起 alias 会破坏 `sql-review` |
| "join key 太显然,跳过固化" | 整个 skill 的意义就在于此 — 把 join key 钉死。 |
| "字段类型无所谓,SQL 里都是字符串" | 类型决定了聚合、cast、分区裁剪的策略。 |
| "派生列到 SQL 阶段再发明就行" | 每个派生列都必须在 Logic Plan 里就出现。 |
| "多张事实表?全 join 上就行" | 多事实是 Logic Plan 的问题,要暴露并停止。 |
## Completion Criteria
## 完成判定(Completion Criteria)
`status: READY` only when:
- Every Logic Plan step has matching entries in this context
- `missing_fields` is empty
- Every metric has a defined `agg` and `expr`
- Every join has a frozen key and a type
- The time grain is set
仅当**以下全部**成立时,`status: READY`:
- Logic Plan 的每一步在本 context 中都有对应条目
- `missing_fields` 为空
- 每个指标都有 `agg``expr`
- 每个 join 都有固化的 key 和 type
- 时间粒度已设定
+76 -73
View File
@@ -1,98 +1,100 @@
---
name: sql-review
description: Use when performing a static review of a generated PySpark SQL string before it is executed. Triggers include "review this SQL", "SQL 评审", "静态审查", "risk level", "check join cardinality", "数据膨胀", "重复聚合", "分区裁剪". Consumes a SQL string plus the SQL Context that produced it, and emits a risk-graded review (HIGH / MEDIUM / LOW) with concrete fixes. Does NOT modify the SQL — it reports findings.
description: 对执行前生成好的 PySpark SQL 字符串做静态评审时使用。触发词包括"review this SQL""SQL 评审""静态审查""risk level""check join cardinality""数据膨胀""重复聚合""分区裁剪"。消费 SQL 字符串及其产出的 SQL Context,产出按风险分级的评审结果(HIGH / MEDIUM / LOW)以及具体修复建议。**不修改 SQL**只报告发现。
---
# SQL Review
# SQL Review(SQL 评审)
## Overview
## 概述(Overview)
Static, pre-execution review of a generated PySpark SQL string. The reviewer does **not** rewrite SQL — it emits findings, severity, and suggested fixes. The author of the SQL decides what to act on.
对一条生成好的 PySpark SQL 字符串做执行前的静态评审。评审者**不**改写 SQL — 它只输出发现项、严重程度和建议修复。由 SQL 的作者决定采纳哪些。
Core principle: **review against the SQL Context, not against vibes.** Every field, alias, and join key referenced in the SQL must exist in the SQL Context that produced it. If it does not, that is itself a `HIGH` finding (the SQL bypassed the contract).
核心原则:**对照 SQL Context 评审,而不是凭直觉。** SQL 中引用的每一个字段、alias、join key 都必须存在于产出它的 SQL Context。如果不存在,这就是一项 `HIGH` 风险(SQL 绕过了契约)。
## When to Use
## 何时使用(When to Use)
Use when:
- A SQL string has been generated from a `sql_context` and is about to be executed
- The user pastes a SQL string and asks for a review, optimization, or risk assessment
- A code change adds a new `spark.sql(...)` call to a PySpark script
**使用场景:**
Do NOT use when:
- The SQL is not Spark SQL (e.g. plain Postgres, HiveQL outside Spark) — at minimum replace Spark-specific checks with the relevant engine's equivalents, but flag it
- The SQL is a one-off interactive query that won't be reused — the user can opt out
- 一条 SQL 字符串已经从 `sql_context` 生成出来,即将执行
- 用户粘贴一条 SQL 字符串,要求评审、优化或风险评估
- 代码改动为 PySpark 脚本新增了 `spark.sql(...)` 调用
## Inputs
**不要使用场景:**
1. The **SQL string** to review
2. The **SQL Context** that the SQL was generated from (or, if absent, an explicit acknowledgement that the SQL is "untrusted / ad-hoc")
- 这条 SQL 不是 Spark SQL(例如纯 Postgres、HiveQL 在 Spark 之外)— 至少要把 Spark 特有的检查替换成对应引擎的等价检查,并标注出来
- 是一次性、不会被复用的交互查询 — 用户可以主动选择跳过
If the SQL Context is missing, raise a single `HIGH` finding: *"SQL has no associated SQL Context; downstream skills cannot guarantee field/join correctness."* Then continue with structural checks.
## 输入(Inputs)
## Severity Model
1. 要评审的 **SQL 字符串**
2. 产出这条 SQL 的 **SQL Context**(如果缺失,需要显式声明"该 SQL 是不可信 / 临时拼凑的")
| Level | Meaning | Action |
如果 SQL Context 缺失,抛出一项 `HIGH` 风险:*"SQL 没有对应的 SQL Context;下游 skill 无法保证字段 / join 的正确性。"* 然后继续做结构性检查。
## 严重程度模型(Severity Model)
| 等级 | 含义 | 处置 |
|---|---|---|
| `HIGH` | Will produce wrong numbers, fail to run, or violate policy | Block execution. Must fix. |
| `MEDIUM` | Likely correct, but inefficient, fragile, or risky at scale | Fix before production. OK in dev. |
| `LOW` | Style / best-practice nit | Optional fix. |
| `HIGH` | 会导致数字错误、执行失败或违反策略 | 阻断执行,必须修。 |
| `MEDIUM` | 大概率对,但低效、脆弱或大规模场景有风险 | 上线前修,开发环境可放过。 |
| `LOW` | 风格 / 最佳实践小瑕疵 | 可选修。 |
Emit findings in this order: `HIGH``MEDIUM``LOW`. A single `HIGH` finding makes the overall review verdict `FAIL`.
按以下顺序输出发现项:`HIGH``MEDIUM``LOW`。任一 `HIGH` 让整体评审结论为 `FAIL`
## Workflow
## 工作流(Workflow)
```dot
digraph sql_review {
"Parse SQL (logical, not just regex)" [shape=box];
"Cross-check aliases, columns, join keys against SQL Context" [shape=box];
"Detect multi-statement or forbidden verbs (guardrail)" [shape=box];
"Analyze join cardinality & inflation risk" [shape=box];
"Detect repeated aggregation on same measure" [shape=box];
"Detect full table scan & missing partition filter" [shape=box];
"Verify GROUP BY matches SELECT" [shape=box];
"Check null handling on join keys & measures" [shape=box];
"Check field types vs operations" [shape=box];
"Aggregate findings into verdict" [shape=box];
"解析 SQL(逻辑层面,不只是正则)" [shape=box];
"对照 SQL Context 交叉校验 alias、字段、join key" [shape=box];
"检测多语句 / 禁用 verb(防护栏)" [shape=box];
"分析 join 基数与膨胀风险" [shape=box];
"检测同一度量的重复聚合" [shape=box];
"检测全表扫描与缺失分区过滤" [shape=box];
"校验 GROUP BY SELECT 一致" [shape=box];
"检查 join key 与度量的空值处理" [shape=box];
"检查字段类型与运算的匹配" [shape=box];
"汇总成评审结论" [shape=box];
"Parse SQL (logical, not just regex)" -> "Cross-check aliases, columns, join keys against SQL Context";
"Cross-check aliases, columns, join keys against SQL Context" -> "Detect multi-statement or forbidden verbs (guardrail)";
"Detect multi-statement or forbidden verbs (guardrail)" -> "Analyze join cardinality & inflation risk";
"Analyze join cardinality & inflation risk" -> "Detect repeated aggregation on same measure";
"Detect repeated aggregation on same measure" -> "Detect full table scan & missing partition filter";
"Detect full table scan & missing partition filter" -> "Verify GROUP BY matches SELECT";
"Verify GROUP BY matches SELECT" -> "Check null handling on join keys & measures";
"Check null handling on join keys & measures" -> "Check field types vs operations";
"Check field types vs operations" -> "Aggregate findings into verdict";
"解析 SQL(逻辑层面,不只是正则)" -> "对照 SQL Context 交叉校验 alias、字段、join key";
"对照 SQL Context 交叉校验 alias、字段、join key" -> "检测多语句 / 禁用 verb(防护栏)";
"检测多语句 / 禁用 verb(防护栏)" -> "分析 join 基数与膨胀风险";
"分析 join 基数与膨胀风险" -> "检测同一度量的重复聚合";
"检测同一度量的重复聚合" -> "检测全表扫描与缺失分区过滤";
"检测全表扫描与缺失分区过滤" -> "校验 GROUP BY SELECT 一致";
"校验 GROUP BY SELECT 一致" -> "检查 join key 与度量的空值处理";
"检查 join key 与度量的空值处理" -> "检查字段类型与运算的匹配";
"检查字段类型与运算的匹配" -> "汇总成评审结论";
}
```
## Checks (run all)
## 检查项(Checks,全部执行)
1. **Reference integrity**every `<alias>.<column>` in the SQL must exist in the SQL Context. Mismatches are `HIGH`.
2. **Join key match**every `ON` predicate must match a frozen join from the SQL Context. Invented joins are `HIGH`.
3. **Forbidden verbs / multi-statement** — defer to `pyspark-sql-guardrails`. Any DDL/DML/maintenance verb, or any `;`-stacked statement, is `HIGH`.
4. **Join cardinality risk** — for every join, if the SQL Context marks cardinality as `1:N` or `N:N` and there is no dedup, raise `MEDIUM` (1:N) or `HIGH` (N:N).
5. **Repeated aggregation** — if the same measure is aggregated twice (e.g. once in a subquery, once outside) without intentional re-aggregation, raise `HIGH`.
6. **Partition / time filter** — for any fact table with a known time field, missing `WHERE` on that field is `HIGH` (full scan risk).
7. **GROUP BY completeness** — every selected non-aggregated column must be in `GROUP BY`. Mismatch is `HIGH`.
8. **Null handling** — nullable join keys with `INNER JOIN` silently drop rows (`MEDIUM`); nullable measures aggregated without `COALESCE` can produce `NULL` that breaks downstream `WHERE ... > 0` (`MEDIUM`).
9. **Type mismatch** — string columns used in numeric aggregation without cast (`MEDIUM`); date string compared to `TIMESTAMP` literal (`MEDIUM`).
10. **Spark best practice**`COUNT(*)` vs `COUNT(1)` is a style nit (`LOW`); `SELECT *` in production is `MEDIUM`; missing `DISTRIBUTE BY` / `CLUSTER BY` for skewed joins is `LOW` to `MEDIUM`.
1. **引用完整性(Reference integrity)**SQL 中的每个 `<alias>.<column>` 都必须存在于 SQL Context。不匹配即 `HIGH`
2. **Join key 匹配**每个 `ON` 谓词都必须对得上 SQL Context 中已固化的 join。凭空发明的 join 算 `HIGH`
3. **禁用 verb / 多语句** — 交给 `pyspark-sql-guardrails`。任何 DDL/DML/维护 verb,或任何 `;` 堆叠的语句,均为 `HIGH`
4. **Join 基数风险** — 对每个 join,如果 SQL Context 标记基数为 `1:N` `N:N` 且没有去重,分别抛 `MEDIUM`(1:N) `HIGH`(N:N)
5. **重复聚合** — 同一度量被聚合两次(例如子查询里一次、外层再一次)且非有意重新聚合,抛 `HIGH`
6. **分区 / 时间过滤** — 对任何有已知时间字段的事实表,缺少对该字段的 `WHERE` 过滤即为 `HIGH`(全表扫描风险)
7. **GROUP BY 完整性** — SELECT 中每个未聚合的列都必须出现在 `GROUP BY` 中。不匹配为 `HIGH`
8. **空值处理** — 可空 join key 上做 `INNER JOIN` 会静默丢行(`MEDIUM`);可空度量未用 `COALESCE` 聚合,可能产出 `NULL` 进而破坏下游 `WHERE ... > 0`(`MEDIUM`)
9. **类型不匹配** — 字符串列未做 cast 就参与数值聚合(`MEDIUM`);日期字符串和 `TIMESTAMP` 字面量比较(`MEDIUM`)
10. **Spark 最佳实践**`COUNT(*)` vs `COUNT(1)` 属于风格(`LOW`);生产环境用 `SELECT *` `MEDIUM`;数据倾斜 join 缺 `DISTRIBUTE BY` / `CLUSTER BY` `LOW` `MEDIUM`
## Output Contract
## 输出契约(Output Contract)
```yaml
sql_review:
verdict: <PASS | FAIL>
highest_severity: <HIGH | MEDIUM | LOW>
summary: "<one-sentence summary>"
summary: "<一句话总结>"
findings:
- id: F1
severity: HIGH
check: reference_integrity
location: "<line/column or SQL fragment>"
message: "<what is wrong>"
fix: "<concrete suggested change>"
location: "<行/列号或 SQL 片段>"
message: "<错在哪>"
fix: "<具体的建议修改>"
- id: F2
severity: MEDIUM
check: join_cardinality
@@ -104,21 +106,22 @@ sql_review:
- "Reviewed by: sql-review v1"
```
## Forbidden Behaviors
## 禁止行为(Forbidden Behaviors)
| Rationalization | Reality |
| 自我说服 | 现实 |
|---|---|
| "The SQL looks fine, no need to cross-check the context" | A column can look fine and still not exist in the source table. Cross-check is mandatory. |
| "I'll fix the SQL while reviewing" | The reviewer reports. The author fixes. Separation of concerns. |
| "No SQL Context was provided, so I'll skip field checks" | Missing context is itself a `HIGH` finding. Never silently skip. |
| "It's just a dev query, skip partition check" | The check is cheap. Skipping it produces silent full scans. |
| "1:N joins are normal, no risk" | 1:N is fine *if* the grain is preserved. The reviewer must verify, not assume. |
| "Regex is enough to check GROUP BY" | GROUP BY membership is structural. Use a parser, not regex. |
| "SQL 看起来没问题,不用对一遍 context" | 字段可能"看起来对"但源表里根本没有,必须做交叉校验。 |
| "我顺手在评审时把 SQL 修了" | 评审只报告,修是作者的事。职责分离。 |
| "没给 SQL Context,那字段检查就跳过" | 缺 context 本身就是 `HIGH`,绝不能悄悄跳过。 |
| "只是开发查询,分区检查跳过" | 这一项成本极低,跳过就会留下静默的全表扫描。 |
| "1:N join 是常态,没风险" | 在保留粒度的前提下 1:N 是 OK 的,但评审必须验证,不能假设。 |
| "用正则就够了,能查 GROUP BY" | GROUP BY 成员关系是结构性的,要用解析器,不要用正则。 |
## Completion Criteria
## 完成判定(Completion Criteria)
A review is complete when:
- All 10 checks have been evaluated
- Every `HIGH` finding has a `fix`
- `verdict` is `PASS` iff `highest_severity` is `MEDIUM` or `LOW`
- The output YAML can be pasted into a PR comment without edits
评审完成的条件:
- 全部 10 项检查都已评估
- 每一条 `HIGH` 风险都有 `fix`
- 当且仅当 `highest_severity``MEDIUM``LOW` 时,`verdict` 才为 `PASS`
- 输出的 YAML 可直接贴到 PR 评论,无需修改