---
name: pyspark-sql-pipeline
description: Use when orchestrating the full PySpark SQL pipeline from a business requirement to a reviewed, ready-to-execute SQL string. Triggers include "完整跑一遍取数流程", "生成 SQL 并 review", "end-to-end 取数", "full pipeline", "从头到尾", "数据需求到 SQL". Coordinates requirements-analysis → metadata-validator → logic-planner → sql-context-builder → pyspark-sql-guardrails → sql-review as a single deterministic flow with explicit user-confirmation gates between stages. Does NOT execute SQL — it drives the pipeline to a reviewed SQL string.
---

# PySpark SQL Pipeline

## Overview

End-to-end orchestrator for turning a business data requirement into a reviewed SQL string. Each stage has a fixed input/output contract; between any two stages there is a **user-confirmation gate** when the upstream status is not green.

Core principle: **never skip a stage, never let a stage quietly proceed past its own `NEED_USER_CONFIRMATION`.** The pipeline is a sequence of hand-offs, not a single agent's internal monologue.

## When to Use

Use when:
- The user gives a data requirement and wants SQL produced end-to-end (e.g. "统计近 30 天各城市贷款总额及客户数")
- The user asks to "run the full pipeline" or "从需求到 SQL 一条龙"
- A non-trivial SQL change touches multiple skills' contracts

Do NOT use when:
- The user only needs one stage (e.g. "review this SQL" → use `sql-review` directly)
- The user explicitly says "skip validation" or "just give me SQL fast" — surface the risk and ask once; if they confirm, you may collapse stages but you MUST still apply `pyspark-sql-guardrails` and `sql-review`

## The Pipeline

```dot
digraph pipeline {
    "1. requirements-analysis" [shape=box];
    "2. metadata-validator" [shape=box];
    "3. logic-planner" [shape=box];
    "4. sql-context-builder" [shape=box];
    "5. SQL generation" [shape=box];
    "6. pyspark-sql-guardrails" [shape=box];
    "7. sql-review" [shape=box];

    "1. requirements-analysis" -> "2. metadata-validator" [label="status=READY_FOR_VALIDATION"];
    "2. metadata-validator" -> "3. logic-planner" [label="status=VALIDATED"];
    "3. logic-planner" -> "4. sql-context-builder" [label="status=PLANNED"];
    "4. sql-context-builder" -> "5. SQL generation" [label="status=READY"];
    "5. SQL generation" -> "6. pyspark-sql-guardrails" [label="SQL string"];
    "6. pyspark-sql-guardrails" -> "7. sql-review" [label="guardrail pass"];
}
```

Every arrow is a **contract boundary**. Do not let a stage start before its predecessor has produced the expected status.

## Stage Contracts

| # | Stage | Skill | Input | Output Status (green) | Output Status (stop) |
|---|---|---|---|---|---|
| 1 | Requirements | `requirements-analysis` | user message | `READY_FOR_VALIDATION` | `NEED_USER_CONFIRMATION` |
| 2 | Metadata | `metadata-validator` | requirements_output + workspace | `VALIDATED` | `NEED_USER_CONFIRMATION` |
| 3 | Logic | `logic-planner` | validation_result | `PLANNED` | `NEED_USER_CONFIRMATION` |
| 4 | Context | `sql-context-builder` | logic_plan + field_mapping | `READY` | `NEED_USER_CONFIRMATION` |
| 5 | Generate | (no skill — see "SQL Generation" below) | sql_context | SQL string | n/a |
| 6 | Guardrail | `pyspark-sql-guardrails` | SQL string | `pass` | `fail` (raise) |
| 7 | Review | `sql-review` | SQL string + sql_context | `verdict: PASS` | `verdict: FAIL` |

## Gate Rules (READ THESE CAREFULLY)

A stage that returns a stop status **must not** be followed by the next stage. Instead:

1. Surface the `pending_questions` from that stage to the user.
2. Stop the pipeline.
3. After the user answers, re-run from the same stage (not from the beginning) using the user's answers as additional input.
4. Once the stage produces its green status, advance to the next stage.

Stages are idempotent. Re-running a stage with the same inputs must produce the same output; with new inputs, it must produce a new output that supersedes the old one. Carry forward all prior `field_mapping`, `validation_result`, `logic_plan`, and `sql_context` as the durable artifacts of the pipeline.

## SQL Generation (Stage 5)

There is no dedicated skill for generating SQL. The pipeline performs this step directly, with these constraints:

- Use the `sql_context` as the **only** source of field names, aliases, and join keys.
- Compose the SQL from the `logic_plan.steps` in order: `source` → `filter` → `join` → `transform` → `dedupe` → `window` → `aggregate` → `output`.
- Every `<alias>.<column>` in the output must appear in `sql_context.fields`.
- The generated SQL must be runnable through `pyspark-sql-guardrails.safe_spark_sql` without further edits.

## User-Confirmation Messages

Each gate should produce a single, structured message in this shape:

```markdown
## Pipeline paused at: <stage name>
**Status:** NEED_USER_CONFIRMATION
**Open questions:**
1. <question 1>
2. <question 2>
**Pending options:** A / B / C / other
**To resume:** answer the questions above, then say "继续" (continue).
```

Do not present a stage's output as final when its status is a stop status. Do not bury the `pending_questions` inside a longer narrative.

## Failure Handling

- **Guardrail failure (`pyspark-sql-guardrails`)** — do not retry the same SQL. Return to Stage 5 and regenerate, this time respecting the guardrail.
- **Review verdict FAIL with HIGH findings** — return to Stage 5 (or earlier if a finding is about context, in which case return to Stage 4). Do not "patch" the SQL to silence a finding; address the root cause.
- **Review verdict FAIL with only MEDIUM/LOW** — surface the findings to the user. They decide whether to ship or fix.
- **Inconsistency between stages** (e.g. SQL references a column not in `sql_context`) — treat as a Stage 5 bug; the SQL was generated without honoring the context. Regenerate, do not amend.

## Forbidden Behaviors

| Rationalization | Reality |
|---|---|
| "User asked for SQL fast, skip the gates" | Gates exist because skipping them produces wrong numbers. Ask the user once; if they confirm, you may collapse only adjacent non-validation stages. Never skip guardrail or review. |
| "Just one stage failed, no need to ask" | Every stop status is a question to the user. Always ask. |
| "I'll inline the validation into SQL generation" | Stages are separate so they can be re-run independently. Inlining couples them. |
| "Review found MEDIUM, ship it" | Surface the findings. The user — not the agent — decides. |
| "Field mapping is obvious, skip the validator" | Field mapping is the validator's job. Bypassing it is the #1 cause of hallucinated column names. |
| "Same status, just continue" | Status names are fixed strings. Re-check the actual status text, do not pattern-match on memory. |

## Completion Criteria

The pipeline is complete only when:
- All 7 stages have run in order
- `sql-review.verdict` is `PASS` (or the user has explicitly accepted remaining MEDIUM/LOW findings)
- The final SQL string, the `sql_context`, the `logic_plan`, the `validation_result`, and the `requirements_output` are all available to the user in the same response

If the user later changes the requirement, restart at Stage 1 — do not patch downstream artifacts.
