init
This commit is contained in:
tao.chen
@@ -0,0 +1,121 @@
|
||||
---
|
||||
name: pyspark-sql-pipeline
|
||||
description: Use when orchestrating the full PySpark SQL pipeline from a business requirement to a reviewed, ready-to-execute SQL string. Triggers include "完整跑一遍取数流程", "生成 SQL 并 review", "end-to-end 取数", "full pipeline", "从头到尾", "数据需求到 SQL". Coordinates requirements-analysis → metadata-validator → logic-planner → sql-context-builder → pyspark-sql-guardrails → sql-review as a single deterministic flow with explicit user-confirmation gates between stages. Does NOT execute SQL — it drives the pipeline to a reviewed SQL string.
|
||||
---
|
||||
|
||||
# PySpark SQL Pipeline
|
||||
|
||||
## Overview
|
||||
|
||||
End-to-end orchestrator for turning a business data requirement into a reviewed SQL string. Each stage has a fixed input/output contract; between any two stages there is a **user-confirmation gate** when the upstream status is not green.
|
||||
|
||||
Core principle: **never skip a stage, never let a stage quietly proceed past its own `NEED_USER_CONFIRMATION`.** The pipeline is a sequence of hand-offs, not a single agent's internal monologue.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use when:
|
||||
- The user gives a data requirement and wants SQL produced end-to-end (e.g. "统计近 30 天各城市贷款总额及客户数")
|
||||
- The user asks to "run the full pipeline" or "从需求到 SQL 一条龙"
|
||||
- A non-trivial SQL change touches multiple skills' contracts
|
||||
|
||||
Do NOT use when:
|
||||
- The user only needs one stage (e.g. "review this SQL" → use `sql-review` directly)
|
||||
- The user explicitly says "skip validation" or "just give me SQL fast" — surface the risk and ask once; if they confirm, you may collapse stages but you MUST still apply `pyspark-sql-guardrails` and `sql-review`
|
||||
|
||||
## The Pipeline
|
||||
|
||||
```dot
|
||||
digraph pipeline {
|
||||
"1. requirements-analysis" [shape=box];
|
||||
"2. metadata-validator" [shape=box];
|
||||
"3. logic-planner" [shape=box];
|
||||
"4. sql-context-builder" [shape=box];
|
||||
"5. SQL generation" [shape=box];
|
||||
"6. pyspark-sql-guardrails" [shape=box];
|
||||
"7. sql-review" [shape=box];
|
||||
|
||||
"1. requirements-analysis" -> "2. metadata-validator" [label="status=READY_FOR_VALIDATION"];
|
||||
"2. metadata-validator" -> "3. logic-planner" [label="status=VALIDATED"];
|
||||
"3. logic-planner" -> "4. sql-context-builder" [label="status=PLANNED"];
|
||||
"4. sql-context-builder" -> "5. SQL generation" [label="status=READY"];
|
||||
"5. SQL generation" -> "6. pyspark-sql-guardrails" [label="SQL string"];
|
||||
"6. pyspark-sql-guardrails" -> "7. sql-review" [label="guardrail pass"];
|
||||
}
|
||||
```
|
||||
|
||||
Every arrow is a **contract boundary**. Do not let a stage start before its predecessor has produced the expected status.
|
||||
|
||||
## Stage Contracts
|
||||
|
||||
| # | Stage | Skill | Input | Output Status (green) | Output Status (stop) |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | Requirements | `requirements-analysis` | user message | `READY_FOR_VALIDATION` | `NEED_USER_CONFIRMATION` |
|
||||
| 2 | Metadata | `metadata-validator` | requirements_output + workspace | `VALIDATED` | `NEED_USER_CONFIRMATION` |
|
||||
| 3 | Logic | `logic-planner` | validation_result | `PLANNED` | `NEED_USER_CONFIRMATION` |
|
||||
| 4 | Context | `sql-context-builder` | logic_plan + field_mapping | `READY` | `NEED_USER_CONFIRMATION` |
|
||||
| 5 | Generate | (no skill — see "SQL Generation" below) | sql_context | SQL string | n/a |
|
||||
| 6 | Guardrail | `pyspark-sql-guardrails` | SQL string | `pass` | `fail` (raise) |
|
||||
| 7 | Review | `sql-review` | SQL string + sql_context | `verdict: PASS` | `verdict: FAIL` |
|
||||
|
||||
## Gate Rules (READ THESE CAREFULLY)
|
||||
|
||||
A stage that returns a stop status **must not** be followed by the next stage. Instead:
|
||||
|
||||
1. Surface the `pending_questions` from that stage to the user.
|
||||
2. Stop the pipeline.
|
||||
3. After the user answers, re-run from the same stage (not from the beginning) using the user's answers as additional input.
|
||||
4. Once the stage produces its green status, advance to the next stage.
|
||||
|
||||
Stages are idempotent. Re-running a stage with the same inputs must produce the same output; with new inputs, it must produce a new output that supersedes the old one. Carry forward all prior `field_mapping`, `validation_result`, `logic_plan`, and `sql_context` as the durable artifacts of the pipeline.
|
||||
|
||||
## SQL Generation (Stage 5)
|
||||
|
||||
There is no dedicated skill for generating SQL. The pipeline performs this step directly, with these constraints:
|
||||
|
||||
- Use the `sql_context` as the **only** source of field names, aliases, and join keys.
|
||||
- Compose the SQL from the `logic_plan.steps` in order: `source` → `filter` → `join` → `transform` → `dedupe` → `window` → `aggregate` → `output`.
|
||||
- Every `<alias>.<column>` in the output must appear in `sql_context.fields`.
|
||||
- The generated SQL must be runnable through `pyspark-sql-guardrails.safe_spark_sql` without further edits.
|
||||
|
||||
## User-Confirmation Messages
|
||||
|
||||
Each gate should produce a single, structured message in this shape:
|
||||
|
||||
```markdown
|
||||
## Pipeline paused at: <stage name>
|
||||
**Status:** NEED_USER_CONFIRMATION
|
||||
**Open questions:**
|
||||
1. <question 1>
|
||||
2. <question 2>
|
||||
**Pending options:** A / B / C / other
|
||||
**To resume:** answer the questions above, then say "继续" (continue).
|
||||
```
|
||||
|
||||
Do not present a stage's output as final when its status is a stop status. Do not bury the `pending_questions` inside a longer narrative.
|
||||
|
||||
## Failure Handling
|
||||
|
||||
- **Guardrail failure (`pyspark-sql-guardrails`)** — do not retry the same SQL. Return to Stage 5 and regenerate, this time respecting the guardrail.
|
||||
- **Review verdict FAIL with HIGH findings** — return to Stage 5 (or earlier if a finding is about context, in which case return to Stage 4). Do not "patch" the SQL to silence a finding; address the root cause.
|
||||
- **Review verdict FAIL with only MEDIUM/LOW** — surface the findings to the user. They decide whether to ship or fix.
|
||||
- **Inconsistency between stages** (e.g. SQL references a column not in `sql_context`) — treat as a Stage 5 bug; the SQL was generated without honoring the context. Regenerate, do not amend.
|
||||
|
||||
## Forbidden Behaviors
|
||||
|
||||
| Rationalization | Reality |
|
||||
|---|---|
|
||||
| "User asked for SQL fast, skip the gates" | Gates exist because skipping them produces wrong numbers. Ask the user once; if they confirm, you may collapse only adjacent non-validation stages. Never skip guardrail or review. |
|
||||
| "Just one stage failed, no need to ask" | Every stop status is a question to the user. Always ask. |
|
||||
| "I'll inline the validation into SQL generation" | Stages are separate so they can be re-run independently. Inlining couples them. |
|
||||
| "Review found MEDIUM, ship it" | Surface the findings. The user — not the agent — decides. |
|
||||
| "Field mapping is obvious, skip the validator" | Field mapping is the validator's job. Bypassing it is the #1 cause of hallucinated column names. |
|
||||
| "Same status, just continue" | Status names are fixed strings. Re-check the actual status text, do not pattern-match on memory. |
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The pipeline is complete only when:
|
||||
- All 7 stages have run in order
|
||||
- `sql-review.verdict` is `PASS` (or the user has explicitly accepted remaining MEDIUM/LOW findings)
|
||||
- The final SQL string, the `sql_context`, the `logic_plan`, the `validation_result`, and the `requirements_output` are all available to the user in the same response
|
||||
|
||||
If the user later changes the requirement, restart at Stage 1 — do not patch downstream artifacts.
|
||||
Reference in New Issue
Block a user