opencode-build/skills/pyspark-sql-pipeline/SKILL.md

name, description

name

description

pyspark-sql-pipeline

Use when orchestrating the full PySpark SQL pipeline from a business requirement to a reviewed, ready-to-execute SQL string. Triggers include "完整跑一遍取数流程", "生成 SQL 并 review", "end-to-end 取数", "full pipeline", "从头到尾", "数据需求到 SQL". Coordinates requirements-analysis → metadata-validator → logic-planner → sql-context-builder → pyspark-sql-guardrails → sql-review as a single deterministic flow with explicit user-confirmation gates between stages. Does NOT execute SQL — it drives the pipeline to a reviewed SQL string.

PySpark SQL Pipeline

Overview

End-to-end orchestrator for turning a business data requirement into a reviewed SQL string. Each stage has a fixed input/output contract; between any two stages there is a user-confirmation gate when the upstream status is not green.

Core principle: never skip a stage, never let a stage quietly proceed past its own NEED_USER_CONFIRMATION. The pipeline is a sequence of hand-offs, not a single agent's internal monologue.

When to Use

Use when:

• The user gives a data requirement and wants SQL produced end-to-end (e.g. "统计近 30 天各城市贷款总额及客户数")
• The user asks to "run the full pipeline" or "从需求到 SQL 一条龙"
• A non-trivial SQL change touches multiple skills' contracts

Do NOT use when:

• The user only needs one stage (e.g. "review this SQL" → use sql-review directly)
• The user explicitly says "skip validation" or "just give me SQL fast" — surface the risk and ask once; if they confirm, you may collapse stages but you MUST still apply pyspark-sql-guardrails and sql-review

The Pipeline

digraph pipeline {
    "1. requirements-analysis" [shape=box];
    "2. metadata-validator" [shape=box];
    "3. logic-planner" [shape=box];
    "4. sql-context-builder" [shape=box];
    "5. SQL generation" [shape=box];
    "6. pyspark-sql-guardrails" [shape=box];
    "7. sql-review" [shape=box];

    "1. requirements-analysis" -> "2. metadata-validator" [label="status=READY_FOR_VALIDATION"];
    "2. metadata-validator" -> "3. logic-planner" [label="status=VALIDATED"];
    "3. logic-planner" -> "4. sql-context-builder" [label="status=PLANNED"];
    "4. sql-context-builder" -> "5. SQL generation" [label="status=READY"];
    "5. SQL generation" -> "6. pyspark-sql-guardrails" [label="SQL string"];
    "6. pyspark-sql-guardrails" -> "7. sql-review" [label="guardrail pass"];
}

Every arrow is a contract boundary. Do not let a stage start before its predecessor has produced the expected status.

Stage Contracts

#	Stage	Skill	Input	Output Status (green)	Output Status (stop)
1	Requirements	requirements-analysis	user message	READY_FOR_VALIDATION	NEED_USER_CONFIRMATION
2	Metadata	metadata-validator	requirements_output + workspace	VALIDATED	NEED_USER_CONFIRMATION
3	Logic	logic-planner	validation_result	PLANNED	NEED_USER_CONFIRMATION
4	Context	sql-context-builder	logic_plan + field_mapping	READY	NEED_USER_CONFIRMATION
5	Generate	(no skill — see "SQL Generation" below)	sql_context	SQL string	n/a
6	Guardrail	pyspark-sql-guardrails	SQL string	pass	fail (raise)
7	Review	sql-review	SQL string + sql_context	verdict: PASS	verdict: FAIL

Gate Rules (READ THESE CAREFULLY)

A stage that returns a stop status must not be followed by the next stage. Instead:

1. Surface the pending_questions from that stage to the user.
2. Stop the pipeline.
3. After the user answers, re-run from the same stage (not from the beginning) using the user's answers as additional input.
4. Once the stage produces its green status, advance to the next stage.

Stages are idempotent. Re-running a stage with the same inputs must produce the same output; with new inputs, it must produce a new output that supersedes the old one. Carry forward all prior field_mapping, validation_result, logic_plan, and sql_context as the durable artifacts of the pipeline.

SQL Generation (Stage 5)

There is no dedicated skill for generating SQL. The pipeline performs this step directly, with these constraints:

• Use the sql_context as the only source of field names, aliases, and join keys.
• Compose the SQL from the logic_plan.steps in order: source → filter → join → transform → dedupe → window → aggregate → output.
• Every <alias>.<column> in the output must appear in sql_context.fields.
• The generated SQL must be runnable through pyspark-sql-guardrails.safe_spark_sql without further edits.

User-Confirmation Messages

Each gate should produce a single, structured message in this shape:

## Pipeline paused at: <stage name>
**Status:** NEED_USER_CONFIRMATION
**Open questions:**
1. <question 1>
2. <question 2>
**Pending options:** A / B / C / other
**To resume:** answer the questions above, then say "继续" (continue).

Do not present a stage's output as final when its status is a stop status. Do not bury the pending_questions inside a longer narrative.

Failure Handling

• Guardrail failure (pyspark-sql-guardrails) — do not retry the same SQL. Return to Stage 5 and regenerate, this time respecting the guardrail.
• Review verdict FAIL with HIGH findings — return to Stage 5 (or earlier if a finding is about context, in which case return to Stage 4). Do not "patch" the SQL to silence a finding; address the root cause.
• Review verdict FAIL with only MEDIUM/LOW — surface the findings to the user. They decide whether to ship or fix.
• Inconsistency between stages (e.g. SQL references a column not in sql_context) — treat as a Stage 5 bug; the SQL was generated without honoring the context. Regenerate, do not amend.

Forbidden Behaviors

Rationalization	Reality
"User asked for SQL fast, skip the gates"	Gates exist because skipping them produces wrong numbers. Ask the user once; if they confirm, you may collapse only adjacent non-validation stages. Never skip guardrail or review.
"Just one stage failed, no need to ask"	Every stop status is a question to the user. Always ask.
"I'll inline the validation into SQL generation"	Stages are separate so they can be re-run independently. Inlining couples them.
"Review found MEDIUM, ship it"	Surface the findings. The user — not the agent — decides.
"Field mapping is obvious, skip the validator"	Field mapping is the validator's job. Bypassing it is the #1 cause of hallucinated column names.
"Same status, just continue"	Status names are fixed strings. Re-check the actual status text, do not pattern-match on memory.

Completion Criteria

The pipeline is complete only when:

• All 7 stages have run in order
• sql-review.verdict is PASS (or the user has explicitly accepted remaining MEDIUM/LOW findings)
• The final SQL string, the sql_context, the logic_plan, the validation_result, and the requirements_output are all available to the user in the same response

If the user later changes the requirement, restart at Stage 1 — do not patch downstream artifacts.