opencode-build/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this repo is

`skill-factory` is a collection of Claude Code **skills** that together implement a deterministic pipeline for turning a Chinese-language business data requirement into a reviewed PySpark SQL string. Each subdirectory at the repository root is a self-contained skill:

```
requirements-analysis → metadata-validator → logic-planner → sql-context-builder → (SQL gen) → pyspark-sql-guardrails → sql-review
```

`pyspark-sql-pipeline` is the orchestrator that drives the full flow with explicit user-confirmation gates between stages.

Every stage has a fixed **input/output contract** with one of two statuses: a *green* status that lets the pipeline advance, or `NEED_USER_CONFIRMATION` which **must stop** the pipeline and surface `pending_questions` to the user. Stage hand-offs are durable artifacts (`requirements_output`, `validation_result`, `field_mapping`, `logic_plan`, `sql_context`) that must be carried forward — never re-derived downstream.

Status names are fixed strings (`READY_FOR_VALIDATION`, `VALIDATED`, `PLANNED`, `READY`, `PASS`, `FAIL`); re-check the actual status text, do not pattern-match on memory.

## Skill file format

Every skill is a single `SKILL.md` with YAML frontmatter (`name`, `description`) describing when to invoke the skill. New skills must follow this format; do not introduce other conventions. The first paragraph of `description` is the trigger surface — keep it concrete (input contract + trigger phrases).

## Hard rule: the validation gate

`pyspark-sql-guardrails/SKILL.md` enforces a non-negotiable gate: **every generated SQL string must pass through `assert_select_or_insert()` before it is shown to the user**. The pipeline ordering and the gate are independent — even if `sql-review` already passed, the SQL still has to pass through this guardrail.

The forbidden-behaviors table in `pyspark-sql-guardrails/SKILL.md` enumerates every rationalization for skipping the validator; treat the list as closed.

## Commands

### Validate a SQL string (the gate)

```bash
# Pipe multi-line SQL via stdin (preferred)
cat <<'EOF' | python3 pyspark-sql-guardrails/scripts/validate_sql.py
SELECT 1 FROM dual
EOF

# Or single-line as first argument
python3 pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dual"
```

Exit codes:
- `0` → `PASS - first keyword: <verb>, body length: <N>` — you may now show the SQL
- `1` → `FAIL - <reason>` — stop, fix the SQL, re-run
- `2` → usage error (no SQL provided)

### Run the validator's standard test set

```bash
# From project root
python3 pyspark-sql-guardrails/scripts/test_sql_guard.py

# Or from the scripts directory
cd pyspark-sql-guardrails/scripts && python3 test_sql_guard.py
```

Expected output: 6 `[OK] EXPECT_OK` lines, 12 `[OK] EXPECT_RAISE: raised as expected` lines, terminated with `ALL TESTS PASSED`. Run this on **any** change to `sql_guard.py` / `validate_sql.py`.

The canonical validator lives in `pyspark-sql-guardrails/scripts/sql_guard.py` and exports `assert_select_or_insert(sql)` and `safe_spark_sql(spark, sql)`. Do not redefine the regex inline at call sites; import from this module.

### Validating metadata files

`metadata-validator` recursively scans the current working directory for metadata files (JSON / YAML / Markdown / CSV / Excel / DDL / data dictionaries). There is no CLI — the skill is invoked conversationally. Filenames hint at table identity but **never trust the filename**; open the file and confirm the table identifier inside.

### Evals

`requirements-analysis/evals/evals.json` contains 3 prompt/expected-output pairs for the requirements-analysis skill. There is no test runner — the file documents expected behavior for manual or harness-driven eval.

## Pipeline architecture

Each skill in the pipeline has a single responsibility and refuses to do work that belongs to its neighbors:

- **requirements-analysis** — clarify ambiguous business terms (time window, user definition, conversion semantics) into a DRD. Never writes SQL. Outputs `requirements_output` with status `READY_FOR_VALIDATION` only when all 7 completion items are checked.
- **metadata-validator** — confirm candidate tables/fields/joins actually exist in workspace metadata. Outputs `validation_result` with `field_mapping` as the single source of truth for downstream stages. Never guesses column names — surface as `pending_question` instead.
- **logic-planner** — decompose into a fixed step taxonomy (`source` / `filter` / `join` / `transform` / `dedupe` / `window` / `aggregate` / `output`). No SQL syntax, business terms only. Required reasoning: grain, metric formula, time grain, join cardinality, dedup position, null strategy.
- **sql-context-builder** — freeze aliases, join keys, field sources, types, and role tags exactly once. Anything not in the SQL Context does not exist for downstream skills.
- **(SQL generation)** — no dedicated skill. The pipeline composes SQL from `logic_plan.steps` in taxonomy order, using only `sql_context.fields`. Must be runnable through `safe_spark_sql` without edits.
- **pyspark-sql-guardrails** — allowlist validator. Allowed: `SELECT`, `WITH ... SELECT`, `INSERT [INTO|OVERWRITE] ... SELECT`. Forbidden: `DELETE`, `TRUNCATE`, `DROP`, `ALTER`, `CREATE`, `REPLACE`, `MERGE`, `UPDATE`, `MSCK`, `REFRESH`, `ANALYZE`, `CACHE`, `UNCACHE`, `VACUUM`, `OPTIMIZE`, stacked statements, arbitrary SQL from config/YAML/env passed directly to `spark.sql`. `INSERT OVERWRITE` requires explicit user approval.
- **sql-review** — static review against the SQL Context (not vibes). 10 checks; severity `HIGH`/`MEDIUM`/`LOW`; verdict `FAIL` if any `HIGH`. Reports findings — does not modify SQL.

The two non-pipeline skills (`logic-planner` taxonomy details, `sql-review` check list) are deliberately stricter than typical code review because they are the only line of defense against hallucinated column names, 1:N row inflation, and silent full scans on partitioned fact tables.

## Working with this repo

- **Adding a new skill** — create `<skill-name>/SKILL.md` with the YAML frontmatter and follow the same input/output/status contract used by existing skills. If the skill emits SQL, it must be downstream of `pyspark-sql-guardrails` (or use it as a library).
- **Modifying `sql_guard.py`** — run `test_sql_guard.py` after every change. The 18-case test set is the contract; do not weaken it to make a SQL string pass.
- **Pipeline status changes** — never collapse gates without user confirmation. The pipeline's value comes from each stage being a separate, re-runnable hand-off; inlining them couples them.
- **Triggers in `description:` frontmatter** — these are what Claude Code matches against user prompts. Be specific (input contract + concrete trigger phrases). Vague triggers cause skill over-invocation.