init
This commit is contained in:
tao.chen
@@ -0,0 +1,89 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## What this repo is
|
||||
|
||||
`skill-factory` is a collection of Claude Code **skills** that together implement a deterministic pipeline for turning a Chinese-language business data requirement into a reviewed PySpark SQL string. Each subdirectory at the repository root is a self-contained skill:
|
||||
|
||||
```
|
||||
requirements-analysis → metadata-validator → logic-planner → sql-context-builder → (SQL gen) → pyspark-sql-guardrails → sql-review
|
||||
```
|
||||
|
||||
`pyspark-sql-pipeline` is the orchestrator that drives the full flow with explicit user-confirmation gates between stages.
|
||||
|
||||
Every stage has a fixed **input/output contract** with one of two statuses: a *green* status that lets the pipeline advance, or `NEED_USER_CONFIRMATION` which **must stop** the pipeline and surface `pending_questions` to the user. Stage hand-offs are durable artifacts (`requirements_output`, `validation_result`, `field_mapping`, `logic_plan`, `sql_context`) that must be carried forward — never re-derived downstream.
|
||||
|
||||
Status names are fixed strings (`READY_FOR_VALIDATION`, `VALIDATED`, `PLANNED`, `READY`, `PASS`, `FAIL`); re-check the actual status text, do not pattern-match on memory.
|
||||
|
||||
## Skill file format
|
||||
|
||||
Every skill is a single `SKILL.md` with YAML frontmatter (`name`, `description`) describing when to invoke the skill. New skills must follow this format; do not introduce other conventions. The first paragraph of `description` is the trigger surface — keep it concrete (input contract + trigger phrases).
|
||||
|
||||
## Hard rule: the validation gate
|
||||
|
||||
`pyspark-sql-guardrails/SKILL.md` enforces a non-negotiable gate: **every generated SQL string must pass through `assert_select_or_insert()` before it is shown to the user**. The pipeline ordering and the gate are independent — even if `sql-review` already passed, the SQL still has to pass through this guardrail.
|
||||
|
||||
The forbidden-behaviors table in `pyspark-sql-guardrails/SKILL.md` enumerates every rationalization for skipping the validator; treat the list as closed.
|
||||
|
||||
## Commands
|
||||
|
||||
### Validate a SQL string (the gate)
|
||||
|
||||
```bash
|
||||
# Pipe multi-line SQL via stdin (preferred)
|
||||
cat <<'EOF' | python3 pyspark-sql-guardrails/scripts/validate_sql.py
|
||||
SELECT 1 FROM dual
|
||||
EOF
|
||||
|
||||
# Or single-line as first argument
|
||||
python3 pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dual"
|
||||
```
|
||||
|
||||
Exit codes:
|
||||
- `0` → `PASS - first keyword: <verb>, body length: <N>` — you may now show the SQL
|
||||
- `1` → `FAIL - <reason>` — stop, fix the SQL, re-run
|
||||
- `2` → usage error (no SQL provided)
|
||||
|
||||
### Run the validator's standard test set
|
||||
|
||||
```bash
|
||||
# From project root
|
||||
python3 pyspark-sql-guardrails/scripts/test_sql_guard.py
|
||||
|
||||
# Or from the scripts directory
|
||||
cd pyspark-sql-guardrails/scripts && python3 test_sql_guard.py
|
||||
```
|
||||
|
||||
Expected output: 6 `[OK] EXPECT_OK` lines, 12 `[OK] EXPECT_RAISE: raised as expected` lines, terminated with `ALL TESTS PASSED`. Run this on **any** change to `sql_guard.py` / `validate_sql.py`.
|
||||
|
||||
The canonical validator lives in `pyspark-sql-guardrails/scripts/sql_guard.py` and exports `assert_select_or_insert(sql)` and `safe_spark_sql(spark, sql)`. Do not redefine the regex inline at call sites; import from this module.
|
||||
|
||||
### Validating metadata files
|
||||
|
||||
`metadata-validator` recursively scans the current working directory for metadata files (JSON / YAML / Markdown / CSV / Excel / DDL / data dictionaries). There is no CLI — the skill is invoked conversationally. Filenames hint at table identity but **never trust the filename**; open the file and confirm the table identifier inside.
|
||||
|
||||
### Evals
|
||||
|
||||
`requirements-analysis/evals/evals.json` contains 3 prompt/expected-output pairs for the requirements-analysis skill. There is no test runner — the file documents expected behavior for manual or harness-driven eval.
|
||||
|
||||
## Pipeline architecture
|
||||
|
||||
Each skill in the pipeline has a single responsibility and refuses to do work that belongs to its neighbors:
|
||||
|
||||
- **requirements-analysis** — clarify ambiguous business terms (time window, user definition, conversion semantics) into a DRD. Never writes SQL. Outputs `requirements_output` with status `READY_FOR_VALIDATION` only when all 7 completion items are checked.
|
||||
- **metadata-validator** — confirm candidate tables/fields/joins actually exist in workspace metadata. Outputs `validation_result` with `field_mapping` as the single source of truth for downstream stages. Never guesses column names — surface as `pending_question` instead.
|
||||
- **logic-planner** — decompose into a fixed step taxonomy (`source` / `filter` / `join` / `transform` / `dedupe` / `window` / `aggregate` / `output`). No SQL syntax, business terms only. Required reasoning: grain, metric formula, time grain, join cardinality, dedup position, null strategy.
|
||||
- **sql-context-builder** — freeze aliases, join keys, field sources, types, and role tags exactly once. Anything not in the SQL Context does not exist for downstream skills.
|
||||
- **(SQL generation)** — no dedicated skill. The pipeline composes SQL from `logic_plan.steps` in taxonomy order, using only `sql_context.fields`. Must be runnable through `safe_spark_sql` without edits.
|
||||
- **pyspark-sql-guardrails** — allowlist validator. Allowed: `SELECT`, `WITH ... SELECT`, `INSERT [INTO|OVERWRITE] ... SELECT`. Forbidden: `DELETE`, `TRUNCATE`, `DROP`, `ALTER`, `CREATE`, `REPLACE`, `MERGE`, `UPDATE`, `MSCK`, `REFRESH`, `ANALYZE`, `CACHE`, `UNCACHE`, `VACUUM`, `OPTIMIZE`, stacked statements, arbitrary SQL from config/YAML/env passed directly to `spark.sql`. `INSERT OVERWRITE` requires explicit user approval.
|
||||
- **sql-review** — static review against the SQL Context (not vibes). 10 checks; severity `HIGH`/`MEDIUM`/`LOW`; verdict `FAIL` if any `HIGH`. Reports findings — does not modify SQL.
|
||||
|
||||
The two non-pipeline skills (`logic-planner` taxonomy details, `sql-review` check list) are deliberately stricter than typical code review because they are the only line of defense against hallucinated column names, 1:N row inflation, and silent full scans on partitioned fact tables.
|
||||
|
||||
## Working with this repo
|
||||
|
||||
- **Adding a new skill** — create `<skill-name>/SKILL.md` with the YAML frontmatter and follow the same input/output/status contract used by existing skills. If the skill emits SQL, it must be downstream of `pyspark-sql-guardrails` (or use it as a library).
|
||||
- **Modifying `sql_guard.py`** — run `test_sql_guard.py` after every change. The 18-case test set is the contract; do not weaken it to make a SQL string pass.
|
||||
- **Pipeline status changes** — never collapse gates without user confirmation. The pipeline's value comes from each stage being a separate, re-runnable hand-off; inlining them couples them.
|
||||
- **Triggers in `description:` frontmatter** — these are what Claude Code matches against user prompts. Be specific (input contract + concrete trigger phrases). Vague triggers cause skill over-invocation.
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
# 使用 Node.js 22 版本的轻量级 Alpine 镜像
|
||||
FROM node:22-alpine
|
||||
|
||||
# 设置工作目录
|
||||
WORKDIR /workspace
|
||||
|
||||
# 安装 opencode-ai 并清理 npm 缓存以最小化镜像体积
|
||||
RUN npm install -g opencode-ai && \
|
||||
npm cache clean --force
|
||||
|
||||
# 验证安装并确认版本
|
||||
RUN opencode --version
|
||||
|
||||
# 使用 ENTRYPOINT 让容器像原生 CLI 工具一样工作
|
||||
ENTRYPOINT ["opencode"]
|
||||
@@ -0,0 +1,158 @@
|
||||
---
|
||||
name: logic-planner
|
||||
description: Use when turning a validated data requirement (candidate tables/fields/joins already confirmed) into a step-by-step execution plan that a SQL generator can implement mechanically. Triggers include "拆解逻辑", "执行计划", "logic plan", "SQL 编排", "怎么算", "指标拆解", "join 顺序". Consumes the Validation Result from metadata-validator and produces a Logic Plan in a fixed step taxonomy (Source / Filter / Join / Transform / Aggregate / Output). Does NOT write SQL.
|
||||
---
|
||||
|
||||
# Logic Planner
|
||||
|
||||
## Overview
|
||||
|
||||
Decompose a validated data requirement into a deterministic pipeline of steps in a fixed taxonomy. The Logic Plan is the *only* input `sql-context-builder` needs to assemble a SQL Context — no further business reasoning is permitted downstream.
|
||||
|
||||
Core principle: **decompose, don't translate.** This skill never writes SQL syntax. It only describes *what* must happen at each step in business/logical terms. The translation to SQL is a separate concern.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use when:
|
||||
- `metadata-validator` returned status `VALIDATED` and you need to design how the query will run
|
||||
- The user asks "how should I compute this", "what's the logic", "step-by-step plan" for a data requirement
|
||||
- You need to disambiguate grain, metric formulas, deduplication, windowing, or ranking before SQL
|
||||
|
||||
Do NOT use when:
|
||||
- `metadata-validator` returned `NEED_USER_CONFIRMATION` — go back and resolve open questions first
|
||||
- The user has already provided a written execution plan and just needs SQL — go straight to `sql-context-builder`
|
||||
|
||||
## Inputs
|
||||
|
||||
1. **Validation Result** from `metadata-validator` (status `VALIDATED`)
|
||||
2. The original business requirement (often embedded in `requirements-analysis` output)
|
||||
3. `field_mapping` (mandatory) — every business field must be mapped to a physical `(table, column, type)` before planning starts
|
||||
|
||||
## Step Taxonomy
|
||||
|
||||
Every Logic Plan is a list of steps. Each step has exactly one `kind` from this set. Reuse the same kind for repeated operations (e.g. multiple joins, multiple filters).
|
||||
|
||||
| Kind | Purpose | Required fields |
|
||||
|---|---|---|
|
||||
| `source` | Read from a table or subquery | `alias`, `table`, `columns[]` |
|
||||
| `filter` | Apply predicates to a named alias | `alias`, `predicates[]` |
|
||||
| `join` | Combine two aliases on keys | `left`, `right`, `keys[]`, `type` (inner/left/right/full/semi/anti) |
|
||||
| `transform` | Derive new columns (case-when, arithmetic, type cast) | `alias`, `expressions[]` |
|
||||
| `aggregate` | Group and reduce | `alias`, `group_by[]`, `metrics[]` (each: `expr`, `alias`, `agg`) |
|
||||
| `dedupe` | DISTINCT or row_number-based dedup | `alias`, `keys[]`, `strategy` |
|
||||
| `window` | Ranking / running totals / lag-lead | `alias`, `partition_by[]`, `order_by[]`, `window_fn` |
|
||||
| `output` | Final projection and ordering | `columns[]`, `order_by[]`, `limit?` |
|
||||
|
||||
Do not invent new kinds. If a transformation does not fit, surface it as `pending_question` and stop.
|
||||
|
||||
## Workflow
|
||||
|
||||
```dot
|
||||
digraph logic_planner {
|
||||
"Read Validation Result + field_mapping" [shape=box];
|
||||
"Identify fact vs dimension tables & choose primary source" [shape=box];
|
||||
"List filters (time range, status, business predicates)" [shape=box];
|
||||
"Plan joins (alias A ON key = alias B.key)" [shape=box];
|
||||
"Plan transforms (derived columns, type casts)" [shape=box];
|
||||
"Plan dedupe / window if needed" [shape=box];
|
||||
"Plan aggregate (grain + metrics)" [shape=box];
|
||||
"Plan output projection + order" [shape=box];
|
||||
"Any ambiguity?" [shape=diamond];
|
||||
"Emit Logic Plan" [shape=box];
|
||||
"Emit Logic Plan + pending_questions" [shape=box];
|
||||
|
||||
"Read Validation Result + field_mapping" -> "Identify fact vs dimension tables & choose primary source";
|
||||
"Identify fact vs dimension tables & choose primary source" -> "List filters (time range, status, business predicates)";
|
||||
"List filters (time range, status, business predicates)" -> "Plan joins (alias A ON key = alias B.key)";
|
||||
"Plan joins (alias A ON key = alias B.key)" -> "Plan transforms (derived columns, type casts)";
|
||||
"Plan transforms (derived columns, type casts)" -> "Plan dedupe / window if needed";
|
||||
"Plan dedupe / window if needed" -> "Plan aggregate (grain + group_by)";
|
||||
"Plan aggregate (grain + group_by)" -> "Plan output projection + order";
|
||||
"Plan output projection + order" -> "Any ambiguity?";
|
||||
"Any ambiguity?" -> "Emit Logic Plan" [label="no"];
|
||||
"Any ambiguity?" -> "Emit Logic Plan + pending_questions" [label="yes"];
|
||||
}
|
||||
```
|
||||
|
||||
## Required Reasoning (do not skip)
|
||||
|
||||
Before emitting the plan, you MUST explicitly state and record:
|
||||
|
||||
1. **Grain** — what does one output row represent? (e.g. one customer, one customer×month, one order)
|
||||
2. **Metric formula** — for every metric, write the exact expression: numerator and denominator defined.
|
||||
3. **Time grain** — confirm the time field and granularity (day, month, hour). State the window inclusively.
|
||||
4. **Join cardinality** — for every join, predict 1:1 / 1:N / N:N. If N:N, mark it and ask the user whether to dedupe before joining.
|
||||
5. **Dedup position** — if a fact table has row multiplicity (e.g. one customer with many addresses), decide *before* the join which side gets deduped and on which key.
|
||||
6. **Null strategy** — for nullable join keys or measures, decide `INNER` vs `LEFT + COALESCE`.
|
||||
|
||||
If any of the above is unclear, add a `pending_question` and stop. Do not guess.
|
||||
|
||||
## Output Contract
|
||||
|
||||
```yaml
|
||||
logic_plan:
|
||||
grain: "<one sentence describing what one output row represents>"
|
||||
fact_table: <table_name>
|
||||
dimension_tables:
|
||||
- <table_name>
|
||||
steps:
|
||||
- kind: source
|
||||
alias: o
|
||||
table: loan_order
|
||||
columns: [customer_id, loan_amount, apply_time, status]
|
||||
- kind: filter
|
||||
alias: o
|
||||
predicates:
|
||||
- "o.apply_time >= '<window_start>'"
|
||||
- "o.apply_time < '<window_end>'"
|
||||
- "o.status = 'SUCCESS'"
|
||||
- kind: join
|
||||
left: { alias: o }
|
||||
right: { alias: c, table: customer_info }
|
||||
keys: ["o.customer_id = c.customer_id"]
|
||||
type: left
|
||||
cardinality_assumption: 1:N # if known; else state "unknown"
|
||||
- kind: transform
|
||||
alias: o
|
||||
expressions:
|
||||
- expr: "CASE WHEN o.loan_amount >= 100000 THEN 'large' ELSE 'small' END"
|
||||
alias: loan_bucket
|
||||
- kind: aggregate
|
||||
alias: o
|
||||
group_by: ["c.city"]
|
||||
metrics:
|
||||
- expr: "SUM(o.loan_amount)"
|
||||
alias: total_loan
|
||||
agg: sum
|
||||
- expr: "COUNT(DISTINCT o.customer_id)"
|
||||
alias: customer_cnt
|
||||
agg: count_distinct
|
||||
- kind: output
|
||||
columns: ["c.city AS city", "total_loan", "customer_cnt"]
|
||||
order_by: ["total_loan DESC"]
|
||||
limit: 100
|
||||
pending_questions:
|
||||
- "<ambiguity not yet resolved, blocking SQL generation>"
|
||||
status: <PLANNED | NEED_USER_CONFIRMATION>
|
||||
```
|
||||
|
||||
## Forbidden Behaviors
|
||||
|
||||
| Rationalization | Reality |
|
||||
|---|---|
|
||||
| "I'll write the SELECT list directly, the user can read SQL" | Logic Plan uses business terms; SQL is downstream. |
|
||||
| "Aggregation is obvious, skip the formula" | Explicit numerator/denominator avoids silent metric drift. |
|
||||
| "1:N vs N:N doesn't matter for SELECT" | It matters: it determines dedup and inflation risk. |
|
||||
| "Time grain is the user's problem" | Pick a grain. Wrong grain ⇒ wrong numbers. |
|
||||
| "I'll add a HAVING clause at SQL time" | Logic Plan must list all filters, including post-aggregation. |
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
Status `PLANNED` requires:
|
||||
- Grain is one sentence, not a paragraph
|
||||
- Every `field_mapping` entry is referenced by exactly one step
|
||||
- Every join has a cardinality assumption
|
||||
- Every metric has an explicit `agg` and an `alias`
|
||||
- `pending_questions` is empty
|
||||
|
||||
Otherwise: `NEED_USER_CONFIRMATION` and stop. Do not proceed to `sql-context-builder`.
|
||||
@@ -0,0 +1,162 @@
|
||||
---
|
||||
name: metadata-validator
|
||||
description: Use when validating that a data requirement's candidate tables, fields, joins, and time columns actually exist in the project's metadata. Triggers include "validate metadata", "check if table exists", "field mapping", "join key validation", "确认表/字段", "校验 schema", "这个字段在哪张表", "口径对得上吗". Reads metadata files recursively across the workspace (JSON / Markdown / CSV / Excel / YAML / DDL / data dictionary) and produces a standardized Validation Result and Field Mapping.
|
||||
---
|
||||
|
||||
# Metadata Validator
|
||||
|
||||
## Overview
|
||||
|
||||
Bridge between the requirement's *expected* tables/fields and the *actual* schema living in workspace metadata. Output is consumed verbatim by `logic-planner` and `sql-context-builder` — they must NOT re-validate.
|
||||
|
||||
Core principle: **never guess a table name, field name, or join relationship.** If a candidate is not provably present in metadata, surface it as a `pending_question` and stop at status `NEED_USER_CONFIRMATION`.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use when:
|
||||
- `requirements-analysis` has produced candidate tables/fields and you need to confirm they exist in real metadata
|
||||
- The user asks "does this table/field exist", "which file describes table X", "map business field to physical column"
|
||||
- You need to detect join keys, time columns, or aggregation-suitable numeric columns
|
||||
- The user is about to write SQL and you must guard against hallucinated column names
|
||||
|
||||
Do NOT use when:
|
||||
- The user has already supplied exact table+column names and confirmed them — go straight to `logic-planner`
|
||||
- The task is code generation without any data warehouse context
|
||||
|
||||
## Inputs
|
||||
|
||||
1. From `requirements-analysis`:
|
||||
- `candidate_tables`: list of table names mentioned or implied
|
||||
- `candidate_fields`: list of business field names with their proposed table
|
||||
- `business_logic`: short summary of what is being computed
|
||||
2. Workspace metadata: any file under the working directory matching metadata conventions (see below)
|
||||
|
||||
## Workflow
|
||||
|
||||
```dot
|
||||
digraph metadata_validator {
|
||||
"Receive candidate tables/fields" [shape=box];
|
||||
"Recursively scan workspace for metadata files" [shape=box];
|
||||
"Parse each file into canonical (table, field, type) records" [shape=box];
|
||||
"Check candidate tables exist" [shape=box];
|
||||
"Check candidate fields exist in their table" [shape=box];
|
||||
"Fuzzy match missing fields against parsed metadata" [shape=box];
|
||||
"Detect join keys & time columns" [shape=box];
|
||||
"All checks pass?" [shape=diamond];
|
||||
"Build Field Mapping" [shape=box];
|
||||
"Emit Validation Result (VALIDATED)" [shape=box];
|
||||
"Emit Validation Result (NEED_USER_CONFIRMATION)" [shape=box];
|
||||
|
||||
"Receive candidate tables/fields" -> "Recursively scan workspace for metadata files";
|
||||
"Recursively scan workspace for metadata files" -> "Parse each file into canonical (table, field, type) records";
|
||||
"Parse each file into canonical (table, field, type) records" -> "Check candidate tables exist";
|
||||
"Check candidate tables exist" -> "Check candidate fields exist in their table";
|
||||
"Check candidate fields exist in their table" -> "Fuzzy match missing fields against parsed metadata";
|
||||
"Fuzzy match missing fields against parsed metadata" -> "Detect join keys & time columns";
|
||||
"Detect join keys & time columns" -> "All checks pass?";
|
||||
"All checks pass?" -> "Build Field Mapping" [label="yes"];
|
||||
"All checks pass?" -> "Emit Validation Result (NEED_USER_CONFIRMATION)" [label="no"];
|
||||
"Build Field Mapping" -> "Emit Validation Result (VALIDATED)";
|
||||
}
|
||||
```
|
||||
|
||||
## Metadata Discovery
|
||||
|
||||
- Recursively scan the **current working directory** (and any subdirectories) — do not assume a `metadata/` folder exists.
|
||||
- Filenames often resemble table names (e.g. `loan_order.json`, `customer_info.md`, `dim_product.csv`) but **never trust the filename alone** — open the file and confirm the table identifier inside.
|
||||
- Accepted formats (auto-detect by content, not extension):
|
||||
- JSON / JSONL
|
||||
- YAML
|
||||
- Markdown tables / headings
|
||||
- CSV (with header)
|
||||
- Excel (`.xlsx`, `.xls`) — use `openpyxl` or `pandas.read_excel`
|
||||
- DDL / `CREATE TABLE` statements
|
||||
- Free-form data dictionaries (parse key-value lines or `field | type | comment` tables)
|
||||
- Convert every file into the canonical record shape before validating:
|
||||
|
||||
```yaml
|
||||
tables:
|
||||
- table: <physical_table_name>
|
||||
source_file: <relative_path>
|
||||
fields:
|
||||
- name: <field_name>
|
||||
type: <string|int|long|double|decimal|date|timestamp|boolean|...>
|
||||
comment: <optional>
|
||||
```
|
||||
|
||||
## Validation Rules
|
||||
|
||||
Run **all** of the following. Any failure ⇒ status `NEED_USER_CONFIRMATION`.
|
||||
|
||||
1. **Table existence** — every candidate table must appear in the parsed metadata.
|
||||
2. **Field existence** — every candidate field must appear in its candidate table.
|
||||
3. **Field ownership** — a field may exist but belong to a different table; never silently swap.
|
||||
4. **Type sanity** — aggregation targets (sum / avg / count) must be numeric; time filters must be date/timestamp/string-encoded-date.
|
||||
5. **Necessary fields** — flag if common fields are missing for the business logic (stat metric, time, join key, dimension).
|
||||
6. **Fuzzy match** — for every missing field, score candidates by name similarity (`customer_id` ↔ `cust_id`, `cust_no`, `customer_no`) and Levenshtein distance; surface top-N.
|
||||
7. **Join detection** — across the validated tables, look for shared `*_id`, `*_no`, `*_code` columns. If multiple plausible join keys exist, ask.
|
||||
8. **Time column disambiguation** — list every time-like field per table (`create_time`, `apply_time`, `txn_date`, `update_time`, `dt`); ask which one defines the time grain.
|
||||
9. **Coverage** — verify that dimensions, metrics, filters, and sort keys are all present.
|
||||
|
||||
## Forbidden Behaviors
|
||||
|
||||
| Rationalization | Reality |
|
||||
|---|---|
|
||||
| "The filename is `loan_order.json`, must be that table" | Filename is a hint. Parse the file to confirm. |
|
||||
| "Field exists somewhere, so the join is fine" | Wrong table ownership invalidates the join. Always check ownership. |
|
||||
| "The user probably means `cust_no`, just use it" | Surface as a candidate and ASK. Never auto-substitute. |
|
||||
| "I'll guess the time field" | Multiple time fields ⇒ ASK. The grain defines the entire query. |
|
||||
| "This metadata is enough, no need to scan further" | Always scan recursively. One file can describe many tables. |
|
||||
|
||||
## Output Contract
|
||||
|
||||
The output is a single YAML document. Downstream skills consume it as-is.
|
||||
|
||||
```yaml
|
||||
validation_result:
|
||||
validated_tables:
|
||||
- <table_name>
|
||||
validated_fields:
|
||||
- <table_name>.<field_name>
|
||||
missing_tables: []
|
||||
missing_fields:
|
||||
- <field_name_not_found_anywhere>
|
||||
candidate_fields:
|
||||
<missing_field_name>:
|
||||
- <candidate_1>
|
||||
- <candidate_2>
|
||||
candidate_tables:
|
||||
<missing_table_name>:
|
||||
- <candidate_1>
|
||||
join_candidates:
|
||||
- left: <table_a>.<field>
|
||||
right: <table_b>.<field>
|
||||
confidence: <high|medium|low>
|
||||
time_field_candidates:
|
||||
<table_name>:
|
||||
- <field>
|
||||
- <field>
|
||||
metadata_sources:
|
||||
- <relative_path_to_metadata_file>
|
||||
pending_questions:
|
||||
- "<human-readable question, ideally with options A/B/C>"
|
||||
field_mapping:
|
||||
<business_field_name>:
|
||||
table: <physical_table>
|
||||
column: <physical_field>
|
||||
type: <data_type>
|
||||
status: <VALIDATED | NEED_USER_CONFIRMATION>
|
||||
```
|
||||
|
||||
`field_mapping` is the **single source of truth** for downstream `logic-planner` and `sql-context-builder`. If a business field has no mapping, the request cannot proceed — add it to `pending_questions`.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
Status `VALIDATED` is only allowed when **all** of the following hold:
|
||||
- Every candidate table is present in `validated_tables`
|
||||
- Every candidate field has a `field_mapping` entry
|
||||
- All join keys are decided (or trivially obvious with `high` confidence)
|
||||
- The time column for the query grain is decided
|
||||
- `pending_questions` is empty
|
||||
|
||||
Anything else ⇒ `NEED_USER_CONFIRMATION` and stop. Do not proceed to `logic-planner`.
|
||||
@@ -0,0 +1,363 @@
|
||||
---
|
||||
name: pyspark-sql-guardrails
|
||||
description: Use when writing or reviewing PySpark scripts, Spark SQL strings, spark.sql calls, SQL loaded from config files, ETL jobs, warehouse maintenance scripts, or data pipeline code where SQL must be limited to SELECT and INSERT only.
|
||||
---
|
||||
|
||||
# PySpark SQL Guardrails
|
||||
|
||||
## Validation Gate (Read This First — Non-Negotiable)
|
||||
|
||||
**Every SQL string this skill produces is UNTRUSTED until it has been run through the bundled validator. The validator runs in one command, immediately after the SQL is generated, and BEFORE the SQL is shown to the user. There is no path that skips this step.**
|
||||
|
||||
The whole flow is two commands and one condition:
|
||||
|
||||
```bash
|
||||
# Step 1 — Generate the SQL (do this in your head / draft)
|
||||
# Step 2 — Run the validator on it (do this IMMEDIATELY, in the same turn)
|
||||
echo "<the SQL you just wrote>" | python3 .claude/skills/pyspark-sql-guardrails/scripts/validate_sql.py
|
||||
|
||||
# Step 3 — Read the output:
|
||||
# PASS - first keyword: select, body length: N → you may now show the SQL
|
||||
# FAIL - <reason> → STOP. Fix the SQL. Re-run.
|
||||
```
|
||||
|
||||
**If you have not pasted the `PASS -` line into the conversation yet, you are not allowed to paste the SQL.** The validator is the gate. The SQL sits on the other side. Do not pass the gate empty-handed.
|
||||
|
||||
This rule is enforced by the skill, not by an external system. Honoring it is a precondition for any output that contains a SQL string.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
PySpark SQL in this environment is allowlist-only: generated or executed SQL may be `SELECT` or `INSERT` only. Treat every SQL string, formatted SQL template, config-loaded SQL, notebook cell, and `spark.sql(...)` call as unsafe until validated.
|
||||
|
||||
Core principle: **never execute SQL by checking that it is “not obviously dangerous”; execute only after proving it is one allowed statement whose first real keyword is SELECT or INSERT.**
|
||||
|
||||
## Mandatory Validation (Hard Rule)
|
||||
|
||||
**Every SQL string MUST pass through `assert_select_or_insert()` before it can reach `spark.sql(...)` — no exceptions.** This is non-negotiable and applies to:
|
||||
|
||||
- SQL you just generated (literal, f-string, `.format(...)`, `+` concatenation)
|
||||
- SQL loaded from YAML / JSON / INI / env / CLI / config files
|
||||
- SQL passed in via parameters, function arguments, or notebook variables
|
||||
- SQL pasted in from a chat message
|
||||
- SQL produced by a templating engine (Jinja, string.Template, etc.)
|
||||
|
||||
The only acceptable call shapes:
|
||||
|
||||
```python
|
||||
# Preferred: wrapper that validates + executes atomically
|
||||
safe_spark_sql(spark, sql)
|
||||
|
||||
# Or: validate first, then execute explicitly
|
||||
assert_select_or_insert(sql) # raises on violation
|
||||
spark.sql(sql)
|
||||
```
|
||||
|
||||
**Forbidden call shapes (any of these ⇒ pipeline failure):**
|
||||
|
||||
```python
|
||||
spark.sql(sql) # direct, no validation
|
||||
spark.sql(f"SELECT ... {user_input} ...") # f-string into spark.sql
|
||||
spark.sql(config["sql"]) # config-driven without validation
|
||||
spark.sql(open("queries/xxx.sql").read()) # file-loaded without validation
|
||||
```
|
||||
|
||||
If `assert_select_or_insert()` raises, the pipeline stops. Do not weaken the rule, do not strip forbidden keywords with regex, do not "rewrite" the SQL into a safe-looking form. Reject and re-author.
|
||||
|
||||
## Required Rule
|
||||
|
||||
Allowed:
|
||||
- `SELECT ...`
|
||||
- `WITH ... SELECT ...`
|
||||
- `INSERT INTO ... SELECT ...`
|
||||
- `INSERT OVERWRITE ... SELECT ...` only if the user explicitly allows overwrite for this job; otherwise ask before using it.
|
||||
|
||||
Forbidden, even for maintenance or metadata refresh:
|
||||
- `DELETE`, `TRUNCATE`, `DROP`, `ALTER`, `CREATE`, `REPLACE`, `MERGE`, `UPDATE`
|
||||
- `MSCK REPAIR`, `REFRESH`, `ANALYZE`, `CACHE`, `UNCACHE`, `VACUUM`, `OPTIMIZE`
|
||||
- Multiple statements separated by semicolons
|
||||
- Arbitrary SQL from YAML/JSON/env/CLI passed directly to `spark.sql`
|
||||
|
||||
**Execution gate:** `assert_select_or_insert(sql)` (or `safe_spark_sql(spark, sql)`) MUST be called for **every** `spark.sql` invocation. There is no fast path. See *Mandatory Validation (Hard Rule)* above.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Situation | Do |
|
||||
|---|---|
|
||||
| Need rows | `safe_spark_sql(spark, "SELECT ...")` |
|
||||
| Need to write rows | `safe_spark_sql(spark, "INSERT INTO table SELECT ...")` |
|
||||
| Need cleanup/dedup/delete | Use SELECT to derive clean data, then `safe_spark_sql` with INSERT into an approved target/staging table; do not delete old data |
|
||||
| Need schema/table/partition maintenance | Stop and ask; do not emit DDL or metadata SQL |
|
||||
| SQL comes from config | `assert_select_or_insert(sql)` first, **then** `spark.sql(sql)` |
|
||||
| User asks “quick fix” | Safety rule still applies — run `assert_select_or_insert` first |
|
||||
| Generated SQL from `sql-context-builder` | Always run `assert_select_or_insert` before executing |
|
||||
|
||||
## Safe Pattern
|
||||
|
||||
**MUST:** every `spark.sql` call site uses `safe_spark_sql` (or calls `assert_select_or_insert` immediately before `spark.sql`). The validator is the only thing standing between a generated string and the cluster.
|
||||
|
||||
Use validation at the boundary where SQL enters execution:
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
_FORBIDDEN_SQL = re.compile(
|
||||
r"\b(delete|truncate|drop|alter|create|replace|merge|update|msck|refresh|analyze|cache|uncache|vacuum|optimize)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
_LEADING_COMMENTS = re.compile(r"\A\s*(?:--[^\n]*\n|/\*.*?\*/\s*)*", re.DOTALL)
|
||||
|
||||
|
||||
def assert_select_or_insert(sql: str) -> str:
|
||||
text = sql.strip()
|
||||
if not text:
|
||||
raise ValueError("SQL is empty")
|
||||
|
||||
# Allow one optional trailing semicolon, but reject stacked statements.
|
||||
body = text[:-1].strip() if text.endswith(";") else text
|
||||
if ";" in body:
|
||||
raise ValueError("Multiple SQL statements are not allowed")
|
||||
|
||||
normalized = _LEADING_COMMENTS.sub("", body).lstrip()
|
||||
first = normalized.split(None, 1)[0].lower() if normalized else ""
|
||||
|
||||
if first not in {"select", "with", "insert"}:
|
||||
raise ValueError(f"Only SELECT and INSERT SQL are allowed, got {first!r}")
|
||||
|
||||
if _FORBIDDEN_SQL.search(normalized):
|
||||
raise ValueError("Forbidden SQL keyword found")
|
||||
|
||||
if first == "with" and not re.search(r"\bselect\b", normalized, re.IGNORECASE):
|
||||
raise ValueError("WITH statements must be SELECT queries")
|
||||
|
||||
return body
|
||||
|
||||
|
||||
def safe_spark_sql(spark, sql: str):
|
||||
return spark.sql(assert_select_or_insert(sql))
|
||||
|
||||
# Good
|
||||
safe_spark_sql(spark, """
|
||||
INSERT INTO analytics.daily_customer_snapshot
|
||||
SELECT * FROM staging.daily_customer_snapshot
|
||||
""")
|
||||
|
||||
# Bad: raises before Spark sees it
|
||||
safe_spark_sql(spark, "ALTER TABLE analytics.daily_customer_snapshot DROP PARTITION (dt='2026-06-10')")
|
||||
```
|
||||
|
||||
If the project already has a SQL parser such as `sqlglot`, prefer AST validation over regex. Still keep the same allowlist: one statement, top-level SELECT or INSERT, no forbidden DDL/DML/maintenance commands anywhere.
|
||||
|
||||
## Local Verification Scaffold (Run After Every Generation)
|
||||
|
||||
The validator is useless if it is never executed. After **every** SQL string is produced (whether by you, a sub-agent, a template, or a tool), the next action is to run it through `assert_select_or_insert` in Python and only then show the result to the user.
|
||||
|
||||
This skill ships its own validator and test set inside the skill directory so the contract is self-contained and versioned with the skill itself. **Do not redefine the regex inline at call sites. Always import from the bundled file.**
|
||||
|
||||
```
|
||||
.claude/skills/pyspark-sql-guardrails/
|
||||
├── SKILL.md ← this file
|
||||
└── scripts/
|
||||
├── sql_guard.py ← canonical validator (assert_select_or_insert + safe_spark_sql)
|
||||
├── validate_sql.py ← one-line CLI: pipe SQL in, get PASS/FAIL out
|
||||
└── test_sql_guard.py ← 18-case standard test set
|
||||
```
|
||||
|
||||
### Step 1 — Use the bundled validator (no copy-pasting, no rewriting)
|
||||
|
||||
The skill ships a one-line CLI at `validate_sql.py`. Use it. Do not write a different invocation, do not paste inline regex, do not call `assert_select_or_insert` directly from a sub-agent unless the bundled CLI is also broken. The CLI is a thin wrapper around `assert_select_or_insert`; the function it calls is the source of truth.
|
||||
|
||||
**Two ways to feed SQL into the validator:**
|
||||
|
||||
```bash
|
||||
# Way A (preferred for multi-line): pipe via stdin
|
||||
cat <<'EOF' | python3 .claude/skills/pyspark-sql-guardrails/scripts/validate_sql.py
|
||||
SELECT
|
||||
c.city AS city,
|
||||
SUM(o.loan_amount) AS total_loan
|
||||
FROM loan_order o
|
||||
INNER JOIN customer_info c ON o.customer_id = c.customer_id
|
||||
WHERE o.status = 'SUCCESS'
|
||||
GROUP BY c.city
|
||||
EOF
|
||||
|
||||
# Way B (single-line only): pass as first argument
|
||||
python3 .claude/skills/pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dual"
|
||||
```
|
||||
|
||||
**Exit codes:**
|
||||
- `0` → `PASS - first keyword: <verb>, body length: <N>`
|
||||
- `1` → `FAIL - <reason>` (the reason names the rule that was violated)
|
||||
- `2` → usage error (no SQL provided)
|
||||
|
||||
The bundled `sql_guard.py` and `validate_sql.py` together:
|
||||
|
||||
```python
|
||||
# sql_guard.py (bundled)
|
||||
import re
|
||||
|
||||
_FORBIDDEN_SQL = re.compile(
|
||||
r"\b(delete|truncate|drop|alter|create|replace|merge|update|msck|refresh|analyze|cache|uncache|vacuum|optimize)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_LEADING_COMMENTS = re.compile(r"\A\s*(?:--[^\n]*\n|/\*.*?\*/\s*)*", re.DOTALL)
|
||||
|
||||
|
||||
def assert_select_or_insert(sql: str) -> str:
|
||||
text = sql.strip()
|
||||
if not text:
|
||||
raise ValueError("SQL is empty")
|
||||
body = text[:-1].strip() if text.endswith(";") else text
|
||||
if ";" in body:
|
||||
raise ValueError("Multiple SQL statements are not allowed")
|
||||
normalized = _LEADING_COMMENTS.sub("", body).lstrip()
|
||||
first = normalized.split(None, 1)[0].lower() if normalized else ""
|
||||
if first not in {"select", "with", "insert"}:
|
||||
raise ValueError(f"Only SELECT and INSERT SQL are allowed, got {first!r}")
|
||||
if _FORBIDDEN_SQL.search(normalized):
|
||||
raise ValueError("Forbidden SQL keyword found")
|
||||
if first == "with" and not re.search(r"\bselect\b", normalized, re.IGNORECASE):
|
||||
raise ValueError("WITH statements must be SELECT queries")
|
||||
return body
|
||||
|
||||
|
||||
def safe_spark_sql(spark, sql: str):
|
||||
return spark.sql(assert_select_or_insert(sql))
|
||||
|
||||
|
||||
# validate_sql.py (bundled, abridged)
|
||||
import sys, os
|
||||
SKILL_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, SKILL_DIR)
|
||||
from sql_guard import assert_select_or_insert
|
||||
|
||||
def main():
|
||||
sql = sys.argv[1] if len(sys.argv) > 1 else sys.stdin.read()
|
||||
try:
|
||||
body = assert_select_or_insert(sql)
|
||||
except ValueError as e:
|
||||
print(f"FAIL - {e}"); return 1
|
||||
first = body.split(None, 1)[0].lower()
|
||||
print(f"PASS - first keyword: {first}, body length: {len(body)}")
|
||||
return 0
|
||||
```
|
||||
|
||||
### Step 2 — Required call pattern after every SQL generation
|
||||
|
||||
The same command, every time, with the SQL you just produced. No variation, no shortcut, no "I'll run it later":
|
||||
|
||||
```bash
|
||||
echo "<THE_SQL_YOU_JUST_GENERATED>" | python3 .claude/skills/pyspark-sql-guardrails/scripts/validate_sql.py
|
||||
```
|
||||
|
||||
**Show the validator's output to the user before showing the SQL.** The expected reply shape:
|
||||
|
||||
```text
|
||||
PASS - first keyword: select, body length: 420
|
||||
```
|
||||
|
||||
If you see `FAIL - <reason>` instead, the SQL is not allowed past the gate. Surface the failure to the user verbatim, fix the SQL, re-run the validator. Loop until `PASS`.
|
||||
|
||||
### Step 3 — Standard test set (run on any change to the validator)
|
||||
|
||||
The skill bundles `test_sql_guard.py` with 6 EXPECT_OK + 12 EXPECT_RAISE cases. Run it from the skill directory any time the validator changes, and as part of CI:
|
||||
|
||||
```bash
|
||||
# From project root
|
||||
python3 .claude/skills/pyspark-sql-guardrails/scripts/test_sql_guard.py
|
||||
|
||||
# Or from the scripts/ directory
|
||||
cd .claude/skills/pyspark-sql-guardrails/scripts
|
||||
python3 test_sql_guard.py
|
||||
```
|
||||
|
||||
Expected output: 6 `[OK] EXPECT_OK` lines, 12 `[OK] EXPECT_RAISE: raised as expected` lines, terminated with `ALL TESTS PASSED`. Any deviation is a regression in the validator — fix the validator, not the tests.
|
||||
|
||||
### Step 4 — Failure protocol
|
||||
|
||||
When the validator raises (in CLI form, the `FAIL - <reason>` line):
|
||||
|
||||
1. **Stop the pipeline.** Do not continue to the next stage.
|
||||
2. **Read the reason.** It names the rule that was violated (empty / multi-statement / wrong verb / forbidden keyword / WITH without SELECT).
|
||||
3. **Do not patch the validator to accept the SQL.** The validator is the source of truth; the SQL is wrong.
|
||||
4. **Do not strip forbidden keywords with regex** to "clean up" the SQL. Reject and re-author the SQL itself.
|
||||
5. **Re-run the validator** with the corrected SQL. Loop until `PASS`.
|
||||
|
||||
### Forbidden Behaviors (validator execution)
|
||||
|
||||
| Rationalization | Reality |
|
||||
|---|---|
|
||||
| "I'll just `spark.sql(sql)` and trust it" | That is what the guardrail is preventing. Run the validator. |
|
||||
| "The validator output is in the previous turn" | Validators are stateful only via the function call. Re-run it. |
|
||||
| "It's a one-liner, eyeballing is enough" | Eyeballing is exactly the failure mode this guardrail exists to prevent. |
|
||||
| "I already ran it on a similar SQL" | Each SQL string is a new value. Re-run on the actual string being shipped. |
|
||||
| "I'll inline a regex instead of using `validate_sql.py`" | The skill ships one canonical CLI. Inline copies drift and miss updates. |
|
||||
| "The SQL is in a code block, not yet 'executed'" | Showing the SQL to the user IS the execution. The gate is before the code block, not after. |
|
||||
|
||||
## Common Rationalizations
|
||||
|
||||
| Excuse | Reality |
|
||||
|---|---|
|
||||
| “It is just metadata refresh.” | `REFRESH`, `MSCK`, and `ANALYZE` are outside SELECT/INSERT. Ask first. |
|
||||
| “MERGE is non-destructive.” | Policy allows SELECT/INSERT only. `MERGE` is forbidden. |
|
||||
| “DELETE+INSERT is the existing pattern.” | Existing unsafe patterns do not override the rule. |
|
||||
| “The config is trusted.” | Config SQL still needs allowlist validation before `spark.sql`. |
|
||||
| “I can strip forbidden words with regex.” | Do not rewrite unsafe SQL into safe-looking SQL. Reject it. |
|
||||
| “We need cleanup before insert.” | Build cleaned data with SELECT and write with INSERT; do not mutate/delete existing data. |
|
||||
| **“The SQL is hard-coded, no need to validate.”** | **Hard-coded strings still must pass through `assert_select_or_insert`. The function checks the *string*, not its provenance. A literal in source code is no safer than a config value.** |
|
||||
| **“I already eyeballed it, the verb is SELECT.”** | **Eyeballing is exactly the failure mode this guardrail prevents. Trust the function, not the eye. If `assert_select_or_insert` was not called, the SQL is untrusted by definition.** |
|
||||
| **“I'll wrap it later / in a follow-up PR.”** | **No. Add the wrapper at the call site *now*. A naked `spark.sql(...)` in committed code is a regression, not a TODO.** |
|
||||
|
||||
## Red Flags
|
||||
|
||||
Stop and add validation or ask the user when you see:
|
||||
- `spark.sql(config["sql"])`, YAML SQL, env SQL, CLI SQL, or f-string SQL from external input
|
||||
- Any SQL verb other than SELECT/WITH/INSERT
|
||||
- `INSERT OVERWRITE` without explicit approval for overwrite semantics
|
||||
- Semicolon-separated SQL batches
|
||||
- Partition repair, schema migration, stale partition cleanup, compaction, vacuum, or stats collection requests
|
||||
- Requests phrased as “quick cleanup”, “just refresh metadata”, “drop old partitions”, or “reuse DELETE+INSERT”
|
||||
- **A direct `spark.sql(sql)` call with no preceding `assert_select_or_insert(sql)` (or no use of `safe_spark_sql`) — even if the SQL string is a hard-coded literal**
|
||||
- **A pull request, diff, or code review that introduces a new `spark.sql(...)` site without the wrapper**
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
- Checking only for `DELETE` and `TRUNCATE`; Spark risk also includes `DROP`, `ALTER`, `MERGE`, `UPDATE`, `MSCK`, `REFRESH`, `ANALYZE`, `VACUUM`, and `OPTIMIZE`.
|
||||
- Validating generated SQL but not config-loaded SQL.
|
||||
- Allowing multiple statements because the first statement is SELECT.
|
||||
- Treating comments as harmless while a later statement contains forbidden SQL.
|
||||
- Using `CREATE OR REPLACE TEMP VIEW` through SQL; prefer DataFrame `createOrReplaceTempView` if a temp view is needed.
|
||||
- **Calling `spark.sql(sql)` directly without routing through `assert_select_or_insert` / `safe_spark_sql`. The allowlist is enforced at the call site, not at SQL-generation time. A generated SQL that *looks* safe is not *validated* safe until the function has run on it.**
|
||||
- **Writing inline `if sql.startswith("SELECT"): spark.sql(sql)` checks. That is not validation. Only `assert_select_or_insert` (or an equivalent AST parser with the same allowlist) counts.**
|
||||
|
||||
---
|
||||
|
||||
## Pipeline Position
|
||||
|
||||
This skill is the **write-time / execution boundary** of the PySpark SQL pipeline:
|
||||
|
||||
```
|
||||
requirements-analysis → metadata-validator → logic-planner → sql-context-builder → pyspark-sql-guardrails → sql-review
|
||||
```
|
||||
|
||||
**Required upstream skill:** `sql-context-builder` — every generated SQL must come with a `sql_context` that froze aliases, join keys, and field sources. SQL without an associated context is a `HIGH` finding under `sql-review` (see check #1: reference integrity).
|
||||
|
||||
**Required downstream skill:** `sql-review` — this guardrail validates *what statements are allowed*; `sql-review` validates *whether the allowed statement is correct*. Both must pass before `spark.sql(...)` is called. Do not skip `sql-review` because the guardrail passed.
|
||||
|
||||
**Execution gate (enforced by this skill):** the only path from a SQL string to the cluster is `assert_select_or_insert(sql)` (or its wrapper `safe_spark_sql(spark, sql)`). A direct `spark.sql(sql)` is a pipeline violation. This rule overrides the pipeline ordering — even if `sql-review` has already passed, the SQL still has to pass through this guardrail at execution time. Guardrail and reviewer are independent checks; one does not satisfy the other.
|
||||
|
||||
**The guardrail does not check:**
|
||||
- Whether the column actually exists in the table (that is `sql-review`'s job via the SQL Context)
|
||||
- Whether the join key is correct (that is `sql-review`'s job)
|
||||
- Whether aggregation is duplicated or rows will inflate (that is `sql-review`'s job)
|
||||
- Whether time/partition filters are present (that is `sql-review`'s job)
|
||||
|
||||
**The guardrail does check:**
|
||||
- First real keyword is `SELECT`, `WITH ... SELECT`, or `INSERT ... SELECT`
|
||||
- No stacked statements (`;`-separated batches)
|
||||
- No forbidden verbs anywhere in the body
|
||||
- `INSERT OVERWRITE` is gated on explicit user approval
|
||||
|
||||
When the guardrail raises, fix the SQL, do not weaken the guardrail. When in doubt, ask the user before generating `INSERT OVERWRITE` or any DDL.
|
||||
Binary file not shown.
@@ -0,0 +1,52 @@
|
||||
"""PySpark SQL Guardrails — validator and safe execution wrapper.
|
||||
|
||||
This module is the canonical implementation referenced by
|
||||
.claude/skills/pyspark-sql-guardrails/SKILL.md. Do not redefine the
|
||||
regex inline at call sites; do not write "lighter" versions. Always
|
||||
import this module and route `spark.sql(...)` through `safe_spark_sql`.
|
||||
|
||||
The validator is pure-Python and has no Spark / I/O dependencies. It
|
||||
is safe to import in unit tests, CI, notebooks, and ad-hoc scripts.
|
||||
"""
|
||||
|
||||
import re
|
||||
|
||||
_FORBIDDEN_SQL = re.compile(
|
||||
r"\b(delete|truncate|drop|alter|create|replace|merge|update|msck|refresh|analyze|cache|uncache|vacuum|optimize)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_LEADING_COMMENTS = re.compile(r"\A\s*(?:--[^\n]*\n|/\*.*?\*/\s*)*", re.DOTALL)
|
||||
|
||||
|
||||
def assert_select_or_insert(sql: str) -> str:
|
||||
"""Return the SQL body if it is a single SELECT/INSERT, else raise ValueError.
|
||||
|
||||
Hard-allowlist. No I/O, no logging side effects, no Spark dependency.
|
||||
Safe to call in unit tests, CI, notebooks, and ad-hoc scripts.
|
||||
"""
|
||||
text = sql.strip()
|
||||
if not text:
|
||||
raise ValueError("SQL is empty")
|
||||
|
||||
body = text[:-1].strip() if text.endswith(";") else text
|
||||
if ";" in body:
|
||||
raise ValueError("Multiple SQL statements are not allowed")
|
||||
|
||||
normalized = _LEADING_COMMENTS.sub("", body).lstrip()
|
||||
first = normalized.split(None, 1)[0].lower() if normalized else ""
|
||||
|
||||
if first not in {"select", "with", "insert"}:
|
||||
raise ValueError(f"Only SELECT and INSERT SQL are allowed, got {first!r}")
|
||||
|
||||
if _FORBIDDEN_SQL.search(normalized):
|
||||
raise ValueError("Forbidden SQL keyword found")
|
||||
|
||||
if first == "with" and not re.search(r"\bselect\b", normalized, re.IGNORECASE):
|
||||
raise ValueError("WITH statements must be SELECT queries")
|
||||
|
||||
return body
|
||||
|
||||
|
||||
def safe_spark_sql(spark, sql: str):
|
||||
"""Validate then execute. The ONLY sanctioned path to spark.sql()."""
|
||||
return spark.sql(assert_select_or_insert(sql))
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Standard test set for sql_guard.assert_select_or_insert.
|
||||
|
||||
Run from the scripts/ subdirectory of the skill:
|
||||
cd .claude/skills/pyspark-sql-guardrails/scripts
|
||||
python3 test_sql_guard.py
|
||||
|
||||
Or from the project root:
|
||||
python3 .claude/skills/pyspark-sql-guardrails/scripts/test_sql_guard.py
|
||||
|
||||
Any change to assert_select_or_insert must be followed by running this
|
||||
set: all EXPECT_RAISE cases must raise, all EXPECT_OK cases must return
|
||||
cleanly. Failures are loud (non-zero exit).
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Allow running this file directly from the scripts/ subdirectory.
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from sql_guard import assert_select_or_insert
|
||||
|
||||
|
||||
def expect_ok(name, sql):
|
||||
try:
|
||||
assert_select_or_insert(sql)
|
||||
print(f" [OK] {name}")
|
||||
except ValueError as e:
|
||||
raise AssertionError(f"{name} should have passed but raised: {e}")
|
||||
|
||||
|
||||
def expect_raise(name, sql):
|
||||
try:
|
||||
assert_select_or_insert(sql)
|
||||
raise AssertionError(f"{name} should have raised but passed")
|
||||
except ValueError:
|
||||
print(f" [OK] {name}: raised as expected")
|
||||
|
||||
|
||||
print("=== EXPECT_OK ===")
|
||||
expect_ok("plain SELECT", "SELECT 1")
|
||||
expect_ok("WITH ... SELECT", "WITH t AS (SELECT 1 AS x) SELECT x FROM t")
|
||||
expect_ok("trailing semicolon", "SELECT 1 FROM dual;")
|
||||
expect_ok("leading -- comment", "-- comment\nSELECT 1")
|
||||
expect_ok("leading /* comment", "/* hi */ SELECT 1")
|
||||
expect_ok("INSERT ... SELECT", "INSERT INTO t SELECT 1")
|
||||
|
||||
print()
|
||||
print("=== EXPECT_RAISE ===")
|
||||
expect_raise("DROP", "DROP TABLE loan_order")
|
||||
expect_raise("UPDATE", "UPDATE loan_order SET status = 1")
|
||||
expect_raise("DELETE", "DELETE FROM loan_order")
|
||||
expect_raise("TRUNCATE", "TRUNCATE TABLE loan_order")
|
||||
expect_raise("ALTER", "ALTER TABLE loan_order ADD COLUMN x INT")
|
||||
expect_raise("MERGE", "MERGE INTO t USING s ON t.id = s.id")
|
||||
expect_raise("MSCK", "MSCK REPAIR TABLE loan_order")
|
||||
expect_raise("REFRESH", "REFRESH TABLE loan_order")
|
||||
expect_raise("VACUUM", "VACUUM TABLE loan_order")
|
||||
expect_raise("stacked ;", "SELECT 1; DROP TABLE loan_order")
|
||||
expect_raise("empty", " ")
|
||||
expect_raise("comment-hidden", "-- ok\nDROP TABLE loan_order")
|
||||
|
||||
print()
|
||||
print("ALL TESTS PASSED")
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env python3
|
||||
"""One-line SQL guard for the pyspark-sql-guardrails skill.
|
||||
|
||||
Usage:
|
||||
# Pipe SQL via stdin (preferred for multi-line SQL)
|
||||
echo "SELECT ..." | python3 validate_sql.py
|
||||
cat query.sql | python3 validate_sql.py
|
||||
|
||||
# Or pass SQL as the first argument (single-line, quoting-friendly)
|
||||
python3 validate_sql.py "SELECT 1"
|
||||
|
||||
Exit code:
|
||||
0 -> PASS (SQL is allowed)
|
||||
1 -> FAIL (SQL violates the allowlist; reason printed to stdout)
|
||||
|
||||
This script is the ONLY sanctioned way to validate a generated SQL
|
||||
string before showing it to the user. It is intentionally short so it
|
||||
can be invoked in a single Bash call from the assistant.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Allow running from anywhere; resolve bundled sql_guard.py.
|
||||
# The skill layout is:
|
||||
# .claude/skills/pyspark-sql-guardrails/
|
||||
# ├── SKILL.md
|
||||
# └── scripts/
|
||||
# ├── sql_guard.py (this script's sibling)
|
||||
# ├── validate_sql.py
|
||||
# └── test_sql_guard.py
|
||||
SCRIPTS_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, SCRIPTS_DIR)
|
||||
|
||||
from sql_guard import assert_select_or_insert # noqa: E402
|
||||
|
||||
|
||||
def read_sql() -> str:
|
||||
if len(sys.argv) > 1:
|
||||
return sys.argv[1]
|
||||
if not sys.stdin.isatty():
|
||||
return sys.stdin.read()
|
||||
print(__doc__, file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
sql = read_sql()
|
||||
try:
|
||||
body = assert_select_or_insert(sql)
|
||||
except ValueError as e:
|
||||
print(f"FAIL - {e}")
|
||||
return 1
|
||||
first = body.split(None, 1)[0].lower()
|
||||
print(f"PASS - first keyword: {first}, body length: {len(body)}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
name: pyspark-sql-pipeline
|
||||
description: Use when orchestrating the full PySpark SQL pipeline from a business requirement to a reviewed, ready-to-execute SQL string. Triggers include "完整跑一遍取数流程", "生成 SQL 并 review", "end-to-end 取数", "full pipeline", "从头到尾", "数据需求到 SQL". Coordinates requirements-analysis → metadata-validator → logic-planner → sql-context-builder → pyspark-sql-guardrails → sql-review as a single deterministic flow with explicit user-confirmation gates between stages. Does NOT execute SQL — it drives the pipeline to a reviewed SQL string.
|
||||
---
|
||||
|
||||
# PySpark SQL Pipeline
|
||||
|
||||
## Overview
|
||||
|
||||
End-to-end orchestrator for turning a business data requirement into a reviewed SQL string. Each stage has a fixed input/output contract; between any two stages there is a **user-confirmation gate** when the upstream status is not green.
|
||||
|
||||
Core principle: **never skip a stage, never let a stage quietly proceed past its own `NEED_USER_CONFIRMATION`.** The pipeline is a sequence of hand-offs, not a single agent's internal monologue.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use when:
|
||||
- The user gives a data requirement and wants SQL produced end-to-end (e.g. "统计近 30 天各城市贷款总额及客户数")
|
||||
- The user asks to "run the full pipeline" or "从需求到 SQL 一条龙"
|
||||
- A non-trivial SQL change touches multiple skills' contracts
|
||||
|
||||
Do NOT use when:
|
||||
- The user only needs one stage (e.g. "review this SQL" → use `sql-review` directly)
|
||||
- The user explicitly says "skip validation" or "just give me SQL fast" — surface the risk and ask once; if they confirm, you may collapse stages but you MUST still apply `pyspark-sql-guardrails` and `sql-review`
|
||||
|
||||
## The Pipeline
|
||||
|
||||
```dot
|
||||
digraph pipeline {
|
||||
"1. requirements-analysis" [shape=box];
|
||||
"2. metadata-validator" [shape=box];
|
||||
"3. logic-planner" [shape=box];
|
||||
"4. sql-context-builder" [shape=box];
|
||||
"5. SQL generation" [shape=box];
|
||||
"6. pyspark-sql-guardrails" [shape=box];
|
||||
"7. sql-review" [shape=box];
|
||||
|
||||
"1. requirements-analysis" -> "2. metadata-validator" [label="status=READY_FOR_VALIDATION"];
|
||||
"2. metadata-validator" -> "3. logic-planner" [label="status=VALIDATED"];
|
||||
"3. logic-planner" -> "4. sql-context-builder" [label="status=PLANNED"];
|
||||
"4. sql-context-builder" -> "5. SQL generation" [label="status=READY"];
|
||||
"5. SQL generation" -> "6. pyspark-sql-guardrails" [label="SQL string"];
|
||||
"6. pyspark-sql-guardrails" -> "7. sql-review" [label="guardrail pass"];
|
||||
}
|
||||
```
|
||||
|
||||
Every arrow is a **contract boundary**. Do not let a stage start before its predecessor has produced the expected status.
|
||||
|
||||
## Stage Contracts
|
||||
|
||||
| # | Stage | Skill | Input | Output Status (green) | Output Status (stop) |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | Requirements | `requirements-analysis` | user message | `READY_FOR_VALIDATION` | `NEED_USER_CONFIRMATION` |
|
||||
| 2 | Metadata | `metadata-validator` | requirements_output + workspace | `VALIDATED` | `NEED_USER_CONFIRMATION` |
|
||||
| 3 | Logic | `logic-planner` | validation_result | `PLANNED` | `NEED_USER_CONFIRMATION` |
|
||||
| 4 | Context | `sql-context-builder` | logic_plan + field_mapping | `READY` | `NEED_USER_CONFIRMATION` |
|
||||
| 5 | Generate | (no skill — see "SQL Generation" below) | sql_context | SQL string | n/a |
|
||||
| 6 | Guardrail | `pyspark-sql-guardrails` | SQL string | `pass` | `fail` (raise) |
|
||||
| 7 | Review | `sql-review` | SQL string + sql_context | `verdict: PASS` | `verdict: FAIL` |
|
||||
|
||||
## Gate Rules (READ THESE CAREFULLY)
|
||||
|
||||
A stage that returns a stop status **must not** be followed by the next stage. Instead:
|
||||
|
||||
1. Surface the `pending_questions` from that stage to the user.
|
||||
2. Stop the pipeline.
|
||||
3. After the user answers, re-run from the same stage (not from the beginning) using the user's answers as additional input.
|
||||
4. Once the stage produces its green status, advance to the next stage.
|
||||
|
||||
Stages are idempotent. Re-running a stage with the same inputs must produce the same output; with new inputs, it must produce a new output that supersedes the old one. Carry forward all prior `field_mapping`, `validation_result`, `logic_plan`, and `sql_context` as the durable artifacts of the pipeline.
|
||||
|
||||
## SQL Generation (Stage 5)
|
||||
|
||||
There is no dedicated skill for generating SQL. The pipeline performs this step directly, with these constraints:
|
||||
|
||||
- Use the `sql_context` as the **only** source of field names, aliases, and join keys.
|
||||
- Compose the SQL from the `logic_plan.steps` in order: `source` → `filter` → `join` → `transform` → `dedupe` → `window` → `aggregate` → `output`.
|
||||
- Every `<alias>.<column>` in the output must appear in `sql_context.fields`.
|
||||
- The generated SQL must be runnable through `pyspark-sql-guardrails.safe_spark_sql` without further edits.
|
||||
|
||||
## User-Confirmation Messages
|
||||
|
||||
Each gate should produce a single, structured message in this shape:
|
||||
|
||||
```markdown
|
||||
## Pipeline paused at: <stage name>
|
||||
**Status:** NEED_USER_CONFIRMATION
|
||||
**Open questions:**
|
||||
1. <question 1>
|
||||
2. <question 2>
|
||||
**Pending options:** A / B / C / other
|
||||
**To resume:** answer the questions above, then say "继续" (continue).
|
||||
```
|
||||
|
||||
Do not present a stage's output as final when its status is a stop status. Do not bury the `pending_questions` inside a longer narrative.
|
||||
|
||||
## Failure Handling
|
||||
|
||||
- **Guardrail failure (`pyspark-sql-guardrails`)** — do not retry the same SQL. Return to Stage 5 and regenerate, this time respecting the guardrail.
|
||||
- **Review verdict FAIL with HIGH findings** — return to Stage 5 (or earlier if a finding is about context, in which case return to Stage 4). Do not "patch" the SQL to silence a finding; address the root cause.
|
||||
- **Review verdict FAIL with only MEDIUM/LOW** — surface the findings to the user. They decide whether to ship or fix.
|
||||
- **Inconsistency between stages** (e.g. SQL references a column not in `sql_context`) — treat as a Stage 5 bug; the SQL was generated without honoring the context. Regenerate, do not amend.
|
||||
|
||||
## Forbidden Behaviors
|
||||
|
||||
| Rationalization | Reality |
|
||||
|---|---|
|
||||
| "User asked for SQL fast, skip the gates" | Gates exist because skipping them produces wrong numbers. Ask the user once; if they confirm, you may collapse only adjacent non-validation stages. Never skip guardrail or review. |
|
||||
| "Just one stage failed, no need to ask" | Every stop status is a question to the user. Always ask. |
|
||||
| "I'll inline the validation into SQL generation" | Stages are separate so they can be re-run independently. Inlining couples them. |
|
||||
| "Review found MEDIUM, ship it" | Surface the findings. The user — not the agent — decides. |
|
||||
| "Field mapping is obvious, skip the validator" | Field mapping is the validator's job. Bypassing it is the #1 cause of hallucinated column names. |
|
||||
| "Same status, just continue" | Status names are fixed strings. Re-check the actual status text, do not pattern-match on memory. |
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The pipeline is complete only when:
|
||||
- All 7 stages have run in order
|
||||
- `sql-review.verdict` is `PASS` (or the user has explicitly accepted remaining MEDIUM/LOW findings)
|
||||
- The final SQL string, the `sql_context`, the `logic_plan`, the `validation_result`, and the `requirements_output` are all available to the user in the same response
|
||||
|
||||
If the user later changes the requirement, restart at Stage 1 — do not patch downstream artifacts.
|
||||
@@ -0,0 +1,273 @@
|
||||
---
|
||||
name: requirements-analysis
|
||||
description: 数据需求解读与拆解 skill。当用户提出"统计/计算/分析/取数/拉个数/做个报表/口径/指标/转化率/留存/活跃/DAU/GMV/漏斗/同环比"等数据类需求，或提供数据分析数据开发需求文档，或贴出表 schema 询问怎么写 SQL/Hive/Spark/取数逻辑时使用。本 skill 不直接写 SQL，而是先把需求拆解成可执行的 DRD（数据需求规格说明书）：识别核心实体、解析表结构、推导字段、识别歧义口径、构建 join 关系，并主动向用户索要缺失信息直到口径完全明确。无论用户是否提供 schema、是否催促"直接给 SQL"，都先走完澄清流程再交付。
|
||||
---
|
||||
|
||||
# 数据需求解读 (Requirements Analysis)
|
||||
|
||||
## 你的角色
|
||||
|
||||
你是数据开发工程师面前的"需求接口人"。用户（通常是产品、运营、数据分析师，甚至业务方）抛过来一句模糊的统计需求，你的任务是：把它翻译成另一位数据开发同学拿到就能直接写 SQL 的规格说明书（DRD）。
|
||||
|
||||
这意味着你的产出**不是 SQL**，而是一份口径明确、字段对齐、join 关系清晰的需求文档。
|
||||
|
||||
## 为什么先澄清再动手
|
||||
|
||||
数据需求最大的坑不在 SQL 写错，而在口径理解错。"近 30 天新用户转化率"这一句话里，"近 30 天"、"新用户"、"转化"三个词每一个都至少有 3 种合理解释。如果不先澄清就动手，最后跑出来的数字可能和业务方期望差几个数量级，返工成本远高于多问几轮。
|
||||
|
||||
所以本 skill 的核心动作是：**发现歧义 → 主动提问 → 等待确认 → 再推进**。宁可多问一轮，不要默认假设。
|
||||
|
||||
## 输入分流
|
||||
|
||||
接到需求后，先判断属于哪种情况：
|
||||
|
||||
### 情况 A：只有业务需求，没有表结构
|
||||
|
||||
例如："统计近 30 天新用户转化率"
|
||||
|
||||
主动索要：
|
||||
- 涉及哪些表（让用户列出表名即可）
|
||||
- 这些表的 schema（字段名 + 类型）
|
||||
- 关键字段的业务含义（特别是状态、时间、金额类）
|
||||
- 指标口径定义（"新用户"、"转化"分别指什么）
|
||||
|
||||
### 情况 B：需求 + 单表/少量表 schema
|
||||
|
||||
理解业务逻辑 → 推导所需字段 → 检查是否缺关键字段 → 向用户确认口径。
|
||||
|
||||
### 情况 C：需求 + 多表 schema
|
||||
|
||||
额外要做：
|
||||
- 区分主表 / 维表 / 事实表
|
||||
- 推导 join 关系（on 哪些字段、内连接还是左连接）
|
||||
- 检查 join 字段在两边是否都存在、类型是否一致
|
||||
|
||||
## 核心动作（每轮对话都要走）
|
||||
|
||||
### 1. 需求拆解
|
||||
|
||||
按这 7 个维度把用户的话拆开，**任何一个不明确就要追问**：
|
||||
|
||||
```text
|
||||
业务目标：为什么要看这个数？
|
||||
统计对象：是用户、订单、商品、还是别的？
|
||||
统计范围：哪些数据进入统计？哪些被过滤？
|
||||
统计时间：从什么时候到什么时候？按自然日还是滚动？
|
||||
统计维度：按什么分组？（渠道 / 城市 / 产品线 / 时间粒度）
|
||||
统计指标：算 count? sum? 比率？分子分母分别是什么？
|
||||
输出形式：一个数？一张表？带哪些列？
|
||||
```
|
||||
|
||||
### 2. 识别核心实体
|
||||
|
||||
从需求里抽出涉及的业务实体（用户、订单、商品、商户、设备、渠道、活动、贷款、客户、账户、交易……），并推测每个实体对应哪张表。
|
||||
|
||||
如果用户没给表名，**列出你的猜测**让用户确认或补充。
|
||||
|
||||
### 3. Schema 解析（拿到表结构后）
|
||||
|
||||
对每张表，要分析清楚：
|
||||
|
||||
| 维度 | 含义 |
|
||||
|------|------|
|
||||
| 表用途 | 这张表记录什么业务事件/状态 |
|
||||
| 主键 | 唯一标识一行的字段 |
|
||||
| 业务主键 | 业务上的唯一键（可能和主键不同） |
|
||||
| 时间字段 | create_time / update_time / 业务时间，用哪个？ |
|
||||
| 状态字段 | 订单状态、用户状态等枚举字段，可取值是什么 |
|
||||
| 维度字段 | 可用于分组的字段 |
|
||||
| 金额字段 | 单位是元还是分？是否含税？ |
|
||||
|
||||
### 4. 推导所需字段
|
||||
|
||||
对照需求拆解，列出每张表要用到的字段。**显式标注哪些字段你假设存在但还没确认**，让用户打勾或补充。
|
||||
|
||||
### 5. 识别缺失字段
|
||||
|
||||
常见缺失：用户 ID、订单 ID、时间字段、状态字段、金额字段、渠道字段、产品字段。如果发现关键字段缺失，**直接指出**，不要绕开它继续推进。
|
||||
|
||||
### 6. 识别需求歧义（重要！）
|
||||
|
||||
这一步是本 skill 的核心价值。**对每一个模糊词，主动列出可能的解释让用户选择**。常见歧义点：
|
||||
|
||||
**时间口径**
|
||||
- "近 30 天" → ① 自然日 [今天-30, 今天)；② T-30 不含今天 [昨天-30, 昨天]；③ 最近 30 个完整日；④ 滚动 30 天
|
||||
- "本月" → 自然月还是结算月？
|
||||
- "凌晨数据归哪天" → 0 点切分还是 4 点切分？
|
||||
|
||||
**用户口径**
|
||||
- "新用户" → 注册用户 / 首次下单用户 / 首次登录用户 / 首次激活用户
|
||||
- "活跃用户" → 登录过 / 操作过 / 下单过
|
||||
- "去重" → 按 user_id 还是按设备 ID 去重？
|
||||
|
||||
**状态/转化口径**
|
||||
- "转化" → 下单 / 支付成功 / 放款 / 激活 / 完成首单
|
||||
- "成功订单" → 状态码具体是哪些？是否包含部分退款？
|
||||
- "支付金额" → 应付 / 实付 / 实收（扣手续费后）
|
||||
|
||||
**关联口径**
|
||||
- "用户的订单" → 创建人 / 收货人 / 实际付款人？
|
||||
- 多表 join → inner / left / 是否要去重防止笛卡尔积？
|
||||
|
||||
输出形式：
|
||||
|
||||
```markdown
|
||||
## 待确认问题
|
||||
|
||||
1. "新用户"指的是？
|
||||
- [ ] A. 在统计窗口内首次注册的用户
|
||||
- [ ] B. 在统计窗口内首次下单的用户
|
||||
- [ ] C. 其他（请说明）
|
||||
|
||||
2. "近 30 天"是？
|
||||
- [ ] A. [今天-30, 今天) 含今天
|
||||
- [ ] B. [昨天-30, 昨天] T-1 口径
|
||||
- [ ] C. 其他
|
||||
```
|
||||
|
||||
### 7. 构建 join 关系
|
||||
|
||||
多表场景下，明确画出关联：
|
||||
|
||||
```
|
||||
user_info.user_id = order_info.user_id (left join, 左表为主)
|
||||
order_info.order_id = pay_info.order_id (inner join)
|
||||
```
|
||||
|
||||
要标明：
|
||||
- 关联字段及类型是否一致
|
||||
- join 类型（inner / left / right / full）
|
||||
- 是否会产生一对多导致重复计算
|
||||
|
||||
### 8. 确认机制
|
||||
|
||||
**严格禁止**：
|
||||
- 自行脑补未说明的业务逻辑
|
||||
- 默认字段含义（哪怕字段名看上去很标准）
|
||||
- 默认时间口径、用户口径、转化口径
|
||||
- 在关键问题没确认前就给出最终 SQL
|
||||
|
||||
每轮回复都要包含三块：
|
||||
- ✅ **已确认**：到目前为止双方对齐的内容
|
||||
- ❓ **待确认**：还需要用户回答的问题（编号列出，方便用户对照回答）
|
||||
- 📋 **当前推导**：基于已知信息你推出的字段清单 / join 关系（标注哪些是假设）
|
||||
|
||||
## 完成判定
|
||||
|
||||
只有以下 7 项**全部**明确，才算需求分析完成、可以交付 DRD：
|
||||
|
||||
- [ ] 业务逻辑明确
|
||||
- [ ] 指标口径明确（分子分母、聚合方式）
|
||||
- [ ] 时间范围明确（窗口定义、时区、是否含端点）
|
||||
- [ ] 维度明确（group by 哪些字段）
|
||||
- [ ] 表来源明确（每个数据来自哪张表）
|
||||
- [ ] join 关系明确（关联键、关联类型）
|
||||
- [ ] 字段明确（每个字段都已被用户确认存在）
|
||||
|
||||
任何一项打问号，都继续走澄清流程，不要交付最终 DRD。
|
||||
|
||||
## 最终交付：DRD 模板
|
||||
|
||||
当 7 项全部确认后，按这个模板输出：
|
||||
|
||||
```markdown
|
||||
# 数据需求规格说明书 (DRD)
|
||||
|
||||
## 一、业务目标
|
||||
（1-2 句话说明这个数据是做什么用的、给谁看、支撑什么决策）
|
||||
|
||||
## 二、统计口径
|
||||
- 统计对象：xxx
|
||||
- 时间范围：xxx（精确到时区、端点）
|
||||
- 过滤条件：xxx
|
||||
- 聚合维度：xxx
|
||||
- 指标定义：
|
||||
- 指标 A = 分子 / 分母，分子定义为 xxx，分母定义为 xxx
|
||||
|
||||
## 三、涉及表
|
||||
|
||||
### 表 1：<table_name>
|
||||
- 用途：xxx
|
||||
- 粒度：一行代表 xxx
|
||||
- 使用字段：
|
||||
- `field_a`: 含义
|
||||
- `field_b`: 含义
|
||||
|
||||
### 表 2：<table_name>
|
||||
（同上）
|
||||
|
||||
## 四、表关联关系
|
||||
```
|
||||
table_a.id = table_b.a_id (left join)
|
||||
```
|
||||
说明：xxx
|
||||
|
||||
## 五、最终字段清单
|
||||
|
||||
| 表名 | 字段名 | 用途 | 备注 |
|
||||
|------|--------|------|------|
|
||||
| table_a | id | 主键 | |
|
||||
| table_a | created_at | 时间过滤 | UTC+8 |
|
||||
|
||||
## 六、实现逻辑（伪代码层面）
|
||||
1. 从 table_a 取 [time_range] 内的数据，过滤 status = xxx
|
||||
2. 按 user_id left join table_b
|
||||
3. group by xxx，聚合 xxx
|
||||
4. 输出列：xxx
|
||||
|
||||
## 七、待开发 SQL 所需信息
|
||||
全部已确认 ✓
|
||||
```
|
||||
|
||||
## 风格提醒
|
||||
|
||||
- 每轮回复都用结构化 markdown，不要长段落散文
|
||||
- 提问要给选项（A/B/C），降低用户回答成本
|
||||
- 字段名、表名用反引号 `code` 包起来
|
||||
- 时间相关问题特别仔细，时区、端点、自然日 vs 滚动这三件事最容易翻车
|
||||
- 如果用户催"直接给 SQL"，温和坚持："为了避免数字跑出来不对，先把这 N 个口径敲定，几分钟就能确认完，然后直接交付准确的 SQL"
|
||||
- 始终记得：你的产出物是 DRD，不是 SQL
|
||||
|
||||
## 例外情况
|
||||
|
||||
如果用户**明确表示**"我已经想清楚口径了，不用再问，按以下定义直接生成 SQL"，并且把所有 7 项完成判定都写明了，那么可以跳过澄清直接进入 DRD/SQL 阶段。但即便这样，也要在回复里复述一遍你理解的口径，让用户最后过一眼。
|
||||
|
||||
---
|
||||
|
||||
## 下游契约（Pipeline Handoff）
|
||||
|
||||
本 skill 是 PySpark SQL 流水线的**第一站**。澄清完成后，必须把结构化结果以标准化形式交付给 `metadata-validator`，由其继续推进。
|
||||
|
||||
**Required next step:** `metadata-validator` — 它会递归扫描工作区元数据文件并校验候选表/字段/关联/时间字段是否真实存在。
|
||||
|
||||
在交付 DRD 之前，必须同时输出以下标准化结构（与第七节 DRD 内容一致）：
|
||||
|
||||
```yaml
|
||||
requirements_output:
|
||||
business_goal: "<一句话业务目标>"
|
||||
business_logic: "<一段话业务逻辑，含统计对象/时间窗口/分组/指标分子分母>"
|
||||
candidate_tables:
|
||||
- <table_name> # 用户提及或根据业务推断的物理表
|
||||
candidate_fields:
|
||||
- name: <business_field_name> # 业务字段名（如 客户号 / 贷款金额）
|
||||
suggested_table: <table_name> # 建议所在表
|
||||
suggested_column: <column_name> # 建议字段名（如未确认可省略）
|
||||
role: <dimension|metric|filter|time|join_key>
|
||||
time_window:
|
||||
field_hint: <table.column> # 用户提到的时间字段
|
||||
window: "<窗口描述，如 近 30 天 / 2026-05-01 ~ 2026-05-31>"
|
||||
join_hints: # 用户已说明的关联（如有）
|
||||
- left_table: <table>
|
||||
left_field: <column>
|
||||
right_table: <table>
|
||||
right_field: <column>
|
||||
join_type: <inner|left|right|full>
|
||||
status: <READY_FOR_VALIDATION | NEED_USER_CONFIRMATION>
|
||||
```
|
||||
|
||||
**契约约束：**
|
||||
|
||||
- `status: READY_FOR_VALIDATION` 只有在第 7 项完成判定全部打勾时才允许输出。
|
||||
- `status: NEED_USER_CONFIRMATION` 时，必须同时在回复里保留第七节的 DRD「待确认问题」清单，`metadata-validator` **不得**在澄清完成前介入。
|
||||
- `candidate_tables` / `candidate_fields` 是 `metadata-validator` 的**唯一**输入；不要让 `metadata-validator` 自行反推业务字段。
|
||||
- 本 skill 不直接进入 SQL 生成。任何“直接给 SQL”的要求都必须先走完澄清 → DRD → `metadata-validator` → `logic-planner` → `sql-context-builder` 这条链路。
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"skill_name": "requirements-analysis",
|
||||
"evals": [
|
||||
{
|
||||
"id": 1,
|
||||
"name": "vague-no-schema",
|
||||
"prompt": "帮我统计下近30天新用户转化率",
|
||||
"expected_output": "应主动索要相关表/schema/字段说明，并对'近30天'、'新用户'、'转化'这三个词分别提出多选式澄清问题。不应直接生成 SQL 或假设口径。",
|
||||
"files": []
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"name": "schema-with-ambiguity",
|
||||
"prompt": "统计上周每个渠道的支付成功订单数和支付金额。表结构如下：\n\norder_info(\n order_id bigint,\n user_id bigint,\n channel_id int,\n create_time timestamp,\n pay_time timestamp,\n status int, -- 1待支付 2已支付 3已发货 4已完成 5已退款 6部分退款\n amount decimal(18,2), -- 应付金额\n paid_amount decimal(18,2) -- 实付金额\n)\n\nchannel_dim(\n channel_id int,\n channel_name varchar\n)",
|
||||
"expected_output": "应识别出'支付成功'状态码歧义（2/3/4/6 是否都算）、'支付金额'是 amount 还是 paid_amount、'上周'是自然周还是滚动 7 天、退款订单是否计入。应提议 left join channel_dim。不应直接生成 SQL。",
|
||||
"files": []
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"name": "multi-table-loan",
|
||||
"prompt": "我要算下贷款产品的首贷转化漏斗，从注册到首次申请到首次放款的转化率，按月看趋势。表：\n\nuser_register(user_id bigint, register_time timestamp, channel varchar)\nloan_apply(apply_id bigint, user_id bigint, apply_time timestamp, product_id int, apply_amount decimal)\nloan_disburse(disburse_id bigint, apply_id bigint, disburse_time timestamp, disburse_amount decimal, status int)\nproduct_dim(product_id int, product_name varchar, product_type varchar)",
|
||||
"expected_output": "应识别多表 join 关系（user → apply 按 user_id, apply → disburse 按 apply_id），追问首贷定义（按用户首次还是按产品首次）、放款 status 含义、'按月看'是注册月还是事件发生月、是否限定产品类型、漏斗分母是否包含没注册成功的用户。",
|
||||
"files": []
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
---
|
||||
name: sql-context-builder
|
||||
description: Use when assembling the single trusted SQL context that the SQL generator and reviewer will consume. Triggers include "build SQL context", "固化 join key", "table aliases", "列出来自哪张表", "准备写 SQL", "context 已经齐了". Consumes a validated Logic Plan + Field Mapping and produces a normalized SQL Context with stable aliases, frozen join keys, and explicit fact/dimension tagging. The SQL generator MUST NOT re-derive any of this.
|
||||
---
|
||||
|
||||
# SQL Context Builder
|
||||
|
||||
## Overview
|
||||
|
||||
Freeze every decision that the SQL generator would otherwise re-invent. Aliases, join keys, field sources, types, role tags (fact vs dimension vs filter), and time columns are all pinned here exactly once. The output is the **single trusted input** for `sql-review` and for any SQL-emitting step.
|
||||
|
||||
Core principle: **anything not in the SQL Context does not exist for downstream skills.** If a downstream skill needs something new, the request must go back through `metadata-validator` and `logic-planner` first.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use when:
|
||||
- `logic-planner` returned status `PLANNED`
|
||||
- The user is about to write Spark SQL and you need a normalized context (aliases, join keys, column sources)
|
||||
- Multiple SQL drafts need to be generated against the *same* logic — context guarantees they stay consistent
|
||||
|
||||
Do NOT use when:
|
||||
- `logic-plan.status` is `NEED_USER_CONFIRMATION` — go back to `logic-planner`
|
||||
- The SQL is trivial (one table, no join) — you can still build the context, but skip if the user explicitly opts out
|
||||
- The task is to translate the plan into SQL — that is a separate concern from this skill
|
||||
|
||||
## Inputs
|
||||
|
||||
1. `logic_plan` from `logic-planner` (status `PLANNED`)
|
||||
2. `field_mapping` from `metadata-validator` (status `VALIDATED`)
|
||||
3. Original `validation_result` for table-level metadata (types, sources)
|
||||
|
||||
## Workflow
|
||||
|
||||
```dot
|
||||
digraph sql_context_builder {
|
||||
"Read logic_plan + field_mapping + validation_result" [shape=box];
|
||||
"Assign stable alias to every table" [shape=box];
|
||||
"Freeze join keys (left_alias.field = right_alias.field)" [shape=box];
|
||||
"Resolve every field to (alias, column, type, source_file)" [shape=box];
|
||||
"Tag each field: dimension | metric | filter | time | join_key" [shape=box];
|
||||
"Mark fact vs dimension tables" [shape=box];
|
||||
"Verify Logic Plan fields all exist in context" [shape=box];
|
||||
"All required fields present?" [shape=diamond];
|
||||
"Emit SQL Context" [shape=box];
|
||||
"Emit SQL Context + missing_fields" [shape=box];
|
||||
|
||||
"Read logic_plan + field_mapping + validation_result" -> "Assign stable alias to every table";
|
||||
"Assign stable alias to every table" -> "Freeze join keys (left_alias.field = right_alias.field)";
|
||||
"Freeze join keys (left_alias.field = right_alias.field)" -> "Resolve every field to (alias, column, type, source_file)";
|
||||
"Resolve every field to (alias, column, type, source_file)" -> "Tag each field: dimension | metric | filter | time | join_key";
|
||||
"Tag each field: dimension | metric | filter | time | join_key" -> "Mark fact vs dimension tables";
|
||||
"Mark fact vs dimension tables" -> "Verify Logic Plan fields all exist in context";
|
||||
"Verify Logic Plan fields all exist in context" -> "All required fields present?";
|
||||
"All required fields present?" -> "Emit SQL Context" [label="yes"];
|
||||
"All required fields present?" -> "Emit SQL Context + missing_fields" [label="no"];
|
||||
}
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
### Alias Assignment
|
||||
- One alias per table, never reused.
|
||||
- Fact table (the one with the primary metric) gets a short alias first (`o`, `f`, `t1`).
|
||||
- Dimension tables get meaningful aliases (`c` for customer, `p` for product, `ch` for channel).
|
||||
- Aliases are **frozen** here. The SQL generator and reviewer must not invent new ones.
|
||||
|
||||
### Join Key Freezing
|
||||
- For every `logic_plan.steps[kind=join]`, freeze the exact predicate:
|
||||
- `left_alias.col1 = right_alias.col2`
|
||||
- Include join type (inner / left / right / full / semi / anti).
|
||||
- Include cardinality assumption carried over from the Logic Plan.
|
||||
- If two tables could join on multiple keys, freeze the chosen one here. Other candidates are out of scope for SQL.
|
||||
|
||||
### Field Resolution
|
||||
For every field used in the Logic Plan, record:
|
||||
- `alias` (which table it lives on)
|
||||
- `column` (physical name)
|
||||
- `type` (from `validation_result`)
|
||||
- `source_file` (which metadata file it came from — for traceability)
|
||||
- `role`: one of `dimension` | `metric` | `filter` | `time` | `join_key` | `derived`
|
||||
|
||||
A field may have multiple roles (e.g. a `customer_id` that is both `join_key` and `dimension`); list all.
|
||||
|
||||
### Fact vs Dimension Tagging
|
||||
- The table containing the primary event/measure is `fact`.
|
||||
- Tables that enrich (customer, product, channel, city) are `dimension`.
|
||||
- One fact per query. If the Logic Plan implies multiple facts, raise a `pending_question` — that is a planning issue, not a context issue.
|
||||
|
||||
### Coverage Check
|
||||
- Every step in `logic_plan.steps` must reference fields that exist in this context.
|
||||
- Every `field_mapping` entry must appear in this context.
|
||||
- Every `group_by` and `order_by` field must be tagged.
|
||||
- Every metric in `aggregate` must have a `metric` tag.
|
||||
- Every predicate in `filter` must reference a `filter` or `time` tag.
|
||||
|
||||
## Output Contract
|
||||
|
||||
```yaml
|
||||
sql_context:
|
||||
aliases:
|
||||
- alias: o
|
||||
table: loan_order
|
||||
role: fact
|
||||
source_file: metadata/loan_order.json
|
||||
- alias: c
|
||||
table: customer_info
|
||||
role: dimension
|
||||
source_file: metadata/customer_info.md
|
||||
joins:
|
||||
- left: { alias: o, column: customer_id }
|
||||
right: { alias: c, column: customer_id }
|
||||
type: left
|
||||
cardinality_assumption: 1:N
|
||||
fields:
|
||||
- alias: o
|
||||
column: customer_id
|
||||
type: string
|
||||
role: [join_key]
|
||||
source_file: metadata/loan_order.json
|
||||
- alias: o
|
||||
column: loan_amount
|
||||
type: decimal(18,2)
|
||||
role: [metric]
|
||||
source_file: metadata/loan_order.json
|
||||
- alias: o
|
||||
column: apply_time
|
||||
type: timestamp
|
||||
role: [time, filter]
|
||||
source_file: metadata/loan_order.json
|
||||
- alias: c
|
||||
column: city
|
||||
type: string
|
||||
role: [dimension]
|
||||
source_file: metadata/customer_info.md
|
||||
metrics:
|
||||
- alias: total_loan
|
||||
expr: "SUM(o.loan_amount)"
|
||||
agg: sum
|
||||
- alias: customer_cnt
|
||||
expr: "COUNT(DISTINCT o.customer_id)"
|
||||
agg: count_distinct
|
||||
dimensions: ["c.city"]
|
||||
time_grain:
|
||||
column: o.apply_time
|
||||
granularity: day
|
||||
filters:
|
||||
- "o.status = 'SUCCESS'"
|
||||
missing_fields: []
|
||||
status: <READY | NEED_USER_CONFIRMATION>
|
||||
```
|
||||
|
||||
## Forbidden Behaviors
|
||||
|
||||
| Rationalization | Reality |
|
||||
|---|---|
|
||||
| "I'll let the SQL generator pick aliases" | Aliases are frozen here. Re-aliasing downstream breaks `sql-review`. |
|
||||
| "The join key is obvious, skip the freeze" | The whole point of this skill is to freeze join keys. |
|
||||
| "Field type doesn't matter, it's all strings in SQL" | Type drives aggregations, casts, and partition-pruning decisions. |
|
||||
| "Derived columns can be invented in SQL" | Every derived column must already appear in the Logic Plan. |
|
||||
| "Multiple facts? Just join them all" | Multi-fact queries are a Logic Plan issue. Surface and stop. |
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
`status: READY` only when:
|
||||
- Every Logic Plan step has matching entries in this context
|
||||
- `missing_fields` is empty
|
||||
- Every metric has a defined `agg` and `expr`
|
||||
- Every join has a frozen key and a type
|
||||
- The time grain is set
|
||||
@@ -0,0 +1,124 @@
|
||||
---
|
||||
name: sql-review
|
||||
description: Use when performing a static review of a generated PySpark SQL string before it is executed. Triggers include "review this SQL", "SQL 评审", "静态审查", "risk level", "check join cardinality", "数据膨胀", "重复聚合", "分区裁剪". Consumes a SQL string plus the SQL Context that produced it, and emits a risk-graded review (HIGH / MEDIUM / LOW) with concrete fixes. Does NOT modify the SQL — it reports findings.
|
||||
---
|
||||
|
||||
# SQL Review
|
||||
|
||||
## Overview
|
||||
|
||||
Static, pre-execution review of a generated PySpark SQL string. The reviewer does **not** rewrite SQL — it emits findings, severity, and suggested fixes. The author of the SQL decides what to act on.
|
||||
|
||||
Core principle: **review against the SQL Context, not against vibes.** Every field, alias, and join key referenced in the SQL must exist in the SQL Context that produced it. If it does not, that is itself a `HIGH` finding (the SQL bypassed the contract).
|
||||
|
||||
## When to Use
|
||||
|
||||
Use when:
|
||||
- A SQL string has been generated from a `sql_context` and is about to be executed
|
||||
- The user pastes a SQL string and asks for a review, optimization, or risk assessment
|
||||
- A code change adds a new `spark.sql(...)` call to a PySpark script
|
||||
|
||||
Do NOT use when:
|
||||
- The SQL is not Spark SQL (e.g. plain Postgres, HiveQL outside Spark) — at minimum replace Spark-specific checks with the relevant engine's equivalents, but flag it
|
||||
- The SQL is a one-off interactive query that won't be reused — the user can opt out
|
||||
|
||||
## Inputs
|
||||
|
||||
1. The **SQL string** to review
|
||||
2. The **SQL Context** that the SQL was generated from (or, if absent, an explicit acknowledgement that the SQL is "untrusted / ad-hoc")
|
||||
|
||||
If the SQL Context is missing, raise a single `HIGH` finding: *"SQL has no associated SQL Context; downstream skills cannot guarantee field/join correctness."* Then continue with structural checks.
|
||||
|
||||
## Severity Model
|
||||
|
||||
| Level | Meaning | Action |
|
||||
|---|---|---|
|
||||
| `HIGH` | Will produce wrong numbers, fail to run, or violate policy | Block execution. Must fix. |
|
||||
| `MEDIUM` | Likely correct, but inefficient, fragile, or risky at scale | Fix before production. OK in dev. |
|
||||
| `LOW` | Style / best-practice nit | Optional fix. |
|
||||
|
||||
Emit findings in this order: `HIGH` → `MEDIUM` → `LOW`. A single `HIGH` finding makes the overall review verdict `FAIL`.
|
||||
|
||||
## Workflow
|
||||
|
||||
```dot
|
||||
digraph sql_review {
|
||||
"Parse SQL (logical, not just regex)" [shape=box];
|
||||
"Cross-check aliases, columns, join keys against SQL Context" [shape=box];
|
||||
"Detect multi-statement or forbidden verbs (guardrail)" [shape=box];
|
||||
"Analyze join cardinality & inflation risk" [shape=box];
|
||||
"Detect repeated aggregation on same measure" [shape=box];
|
||||
"Detect full table scan & missing partition filter" [shape=box];
|
||||
"Verify GROUP BY matches SELECT" [shape=box];
|
||||
"Check null handling on join keys & measures" [shape=box];
|
||||
"Check field types vs operations" [shape=box];
|
||||
"Aggregate findings into verdict" [shape=box];
|
||||
|
||||
"Parse SQL (logical, not just regex)" -> "Cross-check aliases, columns, join keys against SQL Context";
|
||||
"Cross-check aliases, columns, join keys against SQL Context" -> "Detect multi-statement or forbidden verbs (guardrail)";
|
||||
"Detect multi-statement or forbidden verbs (guardrail)" -> "Analyze join cardinality & inflation risk";
|
||||
"Analyze join cardinality & inflation risk" -> "Detect repeated aggregation on same measure";
|
||||
"Detect repeated aggregation on same measure" -> "Detect full table scan & missing partition filter";
|
||||
"Detect full table scan & missing partition filter" -> "Verify GROUP BY matches SELECT";
|
||||
"Verify GROUP BY matches SELECT" -> "Check null handling on join keys & measures";
|
||||
"Check null handling on join keys & measures" -> "Check field types vs operations";
|
||||
"Check field types vs operations" -> "Aggregate findings into verdict";
|
||||
}
|
||||
```
|
||||
|
||||
## Checks (run all)
|
||||
|
||||
1. **Reference integrity** — every `<alias>.<column>` in the SQL must exist in the SQL Context. Mismatches are `HIGH`.
|
||||
2. **Join key match** — every `ON` predicate must match a frozen join from the SQL Context. Invented joins are `HIGH`.
|
||||
3. **Forbidden verbs / multi-statement** — defer to `pyspark-sql-guardrails`. Any DDL/DML/maintenance verb, or any `;`-stacked statement, is `HIGH`.
|
||||
4. **Join cardinality risk** — for every join, if the SQL Context marks cardinality as `1:N` or `N:N` and there is no dedup, raise `MEDIUM` (1:N) or `HIGH` (N:N).
|
||||
5. **Repeated aggregation** — if the same measure is aggregated twice (e.g. once in a subquery, once outside) without intentional re-aggregation, raise `HIGH`.
|
||||
6. **Partition / time filter** — for any fact table with a known time field, missing `WHERE` on that field is `HIGH` (full scan risk).
|
||||
7. **GROUP BY completeness** — every selected non-aggregated column must be in `GROUP BY`. Mismatch is `HIGH`.
|
||||
8. **Null handling** — nullable join keys with `INNER JOIN` silently drop rows (`MEDIUM`); nullable measures aggregated without `COALESCE` can produce `NULL` that breaks downstream `WHERE ... > 0` (`MEDIUM`).
|
||||
9. **Type mismatch** — string columns used in numeric aggregation without cast (`MEDIUM`); date string compared to `TIMESTAMP` literal (`MEDIUM`).
|
||||
10. **Spark best practice** — `COUNT(*)` vs `COUNT(1)` is a style nit (`LOW`); `SELECT *` in production is `MEDIUM`; missing `DISTRIBUTE BY` / `CLUSTER BY` for skewed joins is `LOW` to `MEDIUM`.
|
||||
|
||||
## Output Contract
|
||||
|
||||
```yaml
|
||||
sql_review:
|
||||
verdict: <PASS | FAIL>
|
||||
highest_severity: <HIGH | MEDIUM | LOW>
|
||||
summary: "<one-sentence summary>"
|
||||
findings:
|
||||
- id: F1
|
||||
severity: HIGH
|
||||
check: reference_integrity
|
||||
location: "<line/column or SQL fragment>"
|
||||
message: "<what is wrong>"
|
||||
fix: "<concrete suggested change>"
|
||||
- id: F2
|
||||
severity: MEDIUM
|
||||
check: join_cardinality
|
||||
location: "JOIN customer_info"
|
||||
message: "1:N join without pre-aggregation; risk of row inflation"
|
||||
fix: "Aggregate loan_order by customer_id before joining"
|
||||
notes:
|
||||
- "SQL Context provided: yes (sql_context.aliases = [...])"
|
||||
- "Reviewed by: sql-review v1"
|
||||
```
|
||||
|
||||
## Forbidden Behaviors
|
||||
|
||||
| Rationalization | Reality |
|
||||
|---|---|
|
||||
| "The SQL looks fine, no need to cross-check the context" | A column can look fine and still not exist in the source table. Cross-check is mandatory. |
|
||||
| "I'll fix the SQL while reviewing" | The reviewer reports. The author fixes. Separation of concerns. |
|
||||
| "No SQL Context was provided, so I'll skip field checks" | Missing context is itself a `HIGH` finding. Never silently skip. |
|
||||
| "It's just a dev query, skip partition check" | The check is cheap. Skipping it produces silent full scans. |
|
||||
| "1:N joins are normal, no risk" | 1:N is fine *if* the grain is preserved. The reviewer must verify, not assume. |
|
||||
| "Regex is enough to check GROUP BY" | GROUP BY membership is structural. Use a parser, not regex. |
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
A review is complete when:
|
||||
- All 10 checks have been evaluated
|
||||
- Every `HIGH` finding has a `fix`
|
||||
- `verdict` is `PASS` iff `highest_severity` is `MEDIUM` or `LOW`
|
||||
- The output YAML can be pasted into a PR comment without edits
|
||||
Reference in New Issue
Block a user