opencode-build/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this repo is

skill-factory is a collection of Claude Code skills that together implement a deterministic pipeline for turning a Chinese-language business data requirement into a reviewed PySpark SQL string. Each subdirectory at the repository root is a self-contained skill:

requirements-analysis → metadata-validator → logic-planner → sql-context-builder → (SQL gen) → pyspark-sql-guardrails → sql-review

pyspark-sql-pipeline is the orchestrator that drives the full flow with explicit user-confirmation gates between stages.

Every stage has a fixed input/output contract with one of two statuses: a green status that lets the pipeline advance, or NEED_USER_CONFIRMATION which must stop the pipeline and surface pending_questions to the user. Stage hand-offs are durable artifacts (requirements_output, validation_result, field_mapping, logic_plan, sql_context) that must be carried forward — never re-derived downstream.

Status names are fixed strings (READY_FOR_VALIDATION, VALIDATED, PLANNED, READY, PASS, FAIL); re-check the actual status text, do not pattern-match on memory.

Skill file format

Every skill is a single SKILL.md with YAML frontmatter (name, description) describing when to invoke the skill. New skills must follow this format; do not introduce other conventions. The first paragraph of description is the trigger surface — keep it concrete (input contract + trigger phrases).

Hard rule: the validation gate

pyspark-sql-guardrails/SKILL.md enforces a non-negotiable gate: every generated SQL string must pass through assert_select_or_insert() before it is shown to the user. The pipeline ordering and the gate are independent — even if sql-review already passed, the SQL still has to pass through this guardrail.

The forbidden-behaviors table in pyspark-sql-guardrails/SKILL.md enumerates every rationalization for skipping the validator; treat the list as closed.

Commands

Validate a SQL string (the gate)

# Pipe multi-line SQL via stdin (preferred)
cat <<'EOF' | python3 pyspark-sql-guardrails/scripts/validate_sql.py
SELECT 1 FROM dual
EOF

# Or single-line as first argument
python3 pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dual"

Exit codes:

• 0 → PASS - first keyword: <verb>, body length: <N> — you may now show the SQL
• 1 → FAIL - <reason> — stop, fix the SQL, re-run
• 2 → usage error (no SQL provided)

Run the validator's standard test set

# From project root
python3 pyspark-sql-guardrails/scripts/test_sql_guard.py

# Or from the scripts directory
cd pyspark-sql-guardrails/scripts && python3 test_sql_guard.py

Expected output: 6 [OK] EXPECT_OK lines, 12 [OK] EXPECT_RAISE: raised as expected lines, terminated with ALL TESTS PASSED. Run this on any change to sql_guard.py / validate_sql.py.

The canonical validator lives in pyspark-sql-guardrails/scripts/sql_guard.py and exports assert_select_or_insert(sql) and safe_spark_sql(spark, sql). Do not redefine the regex inline at call sites; import from this module.

Validating metadata files

metadata-validator recursively scans the current working directory for metadata files (JSON / YAML / Markdown / CSV / Excel / DDL / data dictionaries). There is no CLI — the skill is invoked conversationally. Filenames hint at table identity but never trust the filename; open the file and confirm the table identifier inside.

Evals

requirements-analysis/evals/evals.json contains 3 prompt/expected-output pairs for the requirements-analysis skill. There is no test runner — the file documents expected behavior for manual or harness-driven eval.

Pipeline architecture

Each skill in the pipeline has a single responsibility and refuses to do work that belongs to its neighbors:

• requirements-analysis — clarify ambiguous business terms (time window, user definition, conversion semantics) into a DRD. Never writes SQL. Outputs requirements_output with status READY_FOR_VALIDATION only when all 7 completion items are checked.
• metadata-validator — confirm candidate tables/fields/joins actually exist in workspace metadata. Outputs validation_result with field_mapping as the single source of truth for downstream stages. Never guesses column names — surface as pending_question instead.
• logic-planner — decompose into a fixed step taxonomy (source / filter / join / transform / dedupe / window / aggregate / output). No SQL syntax, business terms only. Required reasoning: grain, metric formula, time grain, join cardinality, dedup position, null strategy.
• sql-context-builder — freeze aliases, join keys, field sources, types, and role tags exactly once. Anything not in the SQL Context does not exist for downstream skills.
• (SQL generation) — no dedicated skill. The pipeline composes SQL from logic_plan.steps in taxonomy order, using only sql_context.fields. Must be runnable through safe_spark_sql without edits.
• pyspark-sql-guardrails — allowlist validator. Allowed: SELECT, WITH ... SELECT, INSERT [INTO|OVERWRITE] ... SELECT. Forbidden: DELETE, TRUNCATE, DROP, ALTER, CREATE, REPLACE, MERGE, UPDATE, MSCK, REFRESH, ANALYZE, CACHE, UNCACHE, VACUUM, OPTIMIZE, stacked statements, arbitrary SQL from config/YAML/env passed directly to spark.sql. INSERT OVERWRITE requires explicit user approval.
• sql-review — static review against the SQL Context (not vibes). 10 checks; severity HIGH/MEDIUM/LOW; verdict FAIL if any HIGH. Reports findings — does not modify SQL.

The two non-pipeline skills (logic-planner taxonomy details, sql-review check list) are deliberately stricter than typical code review because they are the only line of defense against hallucinated column names, 1:N row inflation, and silent full scans on partitioned fact tables.

Working with this repo

• Adding a new skill — create <skill-name>/SKILL.md with the YAML frontmatter and follow the same input/output/status contract used by existing skills. If the skill emits SQL, it must be downstream of pyspark-sql-guardrails (or use it as a library).
• Modifying sql_guard.py — run test_sql_guard.py after every change. The 18-case test set is the contract; do not weaken it to make a SQL string pass.
• Pipeline status changes — never collapse gates without user confirmation. The pipeline's value comes from each stage being a separate, re-runnable hand-off; inlining them couples them.
• Triggers in description: frontmatter — these are what Claude Code matches against user prompts. Be specific (input contract + concrete trigger phrases). Vague triggers cause skill over-invocation.