From 213b4db4237f3a75fa14e9e7d2576df7876aa76e Mon Sep 17 00:00:00 2001 From: "tao.chen" <93983997+taochen-ct@users.noreply.github.com> Date: Thu, 18 Jun 2026 14:10:16 +0800 Subject: [PATCH] update documents --- CLAUDE.md | 66 +++++++++++++++++++++++++++++++++++++++---------------- README.md | 60 +++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 106 insertions(+), 20 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index a1ee698..8714a5d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,14 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## What this repo is -`skill-factory` is a collection of Claude Code **skills** that together implement a deterministic pipeline for turning a Chinese-language business data requirement into a reviewed PySpark SQL string. Each subdirectory at the repository root is a self-contained skill: +`opencode-build` is a Docker image build for the [`opencode-ai`](https://github.com/sst/opencode) CLI. The image: + +- Installs Node.js 22, `uv`-managed Python 3.11, and `ripgrep`. +- Installs Python data-stack deps from `requirements.txt` (PySpark, pandas, polars, clickhouse-connect, pymongo, redis, hdfs, ...). +- Installs `opencode-ai` globally via npm. +- Copies `skills/` into `/root/.config/opencode/skills/` so they are auto-discovered by Claude Code at runtime. + +The `skills/` directory contains a deterministic Chinese-language business-data → reviewed PySpark SQL pipeline. See [Pipeline architecture](#pipeline-architecture) for that. Most edits touch the skills, not the image plumbing. ``` requirements-analysis → metadata-validator → logic-planner → sql-context-builder → (SQL gen) → pyspark-sql-guardrails → sql-review @@ -16,28 +23,36 @@ Every stage has a fixed **input/output contract** with one of two statuses: a *g Status names are fixed strings (`READY_FOR_VALIDATION`, `VALIDATED`, `PLANNED`, `READY`, `PASS`, `FAIL`); re-check the actual status text, do not pattern-match on memory. -## Skill file format - -Every skill is a single `SKILL.md` with YAML frontmatter (`name`, `description`) describing when to invoke the skill. New skills must follow this format; do not introduce other conventions. The first paragraph of `description` is the trigger surface — keep it concrete (input contract + trigger phrases). - -## Hard rule: the validation gate - -`pyspark-sql-guardrails/SKILL.md` enforces a non-negotiable gate: **every generated SQL string must pass through `assert_select_or_insert()` before it is shown to the user**. The pipeline ordering and the gate are independent — even if `sql-review` already passed, the SQL still has to pass through this guardrail. - -The forbidden-behaviors table in `pyspark-sql-guardrails/SKILL.md` enumerates every rationalization for skipping the validator; treat the list as closed. - ## Commands +### Build the Docker image + +```bash +docker build -t opencode-custom:latest . +``` + +The `Dockerfile` is two-stage: a builder that fetches Node.js 22 and a Python 3.11 tarball via `uv`, and a final image that copies the artifacts, installs `opencode-ai` via npm, and copies `skills/` to `/root/.config/opencode/skills/`. The `ripgrep-*.tar.gz` file in the repo root is unpacked into `/opt/ripgrep-15.1.0-x86_64-unknown-linux-musl/` and symlinked to `/usr/local/bin/rg`. + +### Run the image + +```bash +docker run --rm -it opencode-custom:latest +``` + +The image's `ENTRYPOINT` is `opencode`. + ### Validate a SQL string (the gate) +The validator lives at `skills/pyspark-sql-guardrails/scripts/validate_sql.py` in this repo. **In the built image, it is at `/root/.config/opencode/skills/pyspark-sql-guardrails/scripts/validate_sql.py`** — the Dockerfile copies `skills/` to that path. + ```bash -# Pipe multi-line SQL via stdin (preferred) -cat <<'EOF' | python3 pyspark-sql-guardrails/scripts/validate_sql.py +# From the repo root — pipe multi-line SQL via stdin (preferred) +cat <<'EOF' | python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py SELECT 1 FROM dual EOF # Or single-line as first argument -python3 pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dual" +python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dual" ``` Exit codes: @@ -45,19 +60,21 @@ Exit codes: - `1` → `FAIL - ` — stop, fix the SQL, re-run - `2` → usage error (no SQL provided) +> ⚠️ The bundled `SKILL.md` and `validate_sql.py` / `test_sql_guard.py` docstrings reference `.claude/skills/...` paths from an earlier layout. Those paths are stale for this repo — use the `skills/` (host) or `/root/.config/opencode/skills/` (image) paths above. + ### Run the validator's standard test set ```bash # From project root -python3 pyspark-sql-guardrails/scripts/test_sql_guard.py +python3 skills/pyspark-sql-guardrails/scripts/test_sql_guard.py # Or from the scripts directory -cd pyspark-sql-guardrails/scripts && python3 test_sql_guard.py +cd skills/pyspark-sql-guardrails/scripts && python3 test_sql_guard.py ``` Expected output: 6 `[OK] EXPECT_OK` lines, 12 `[OK] EXPECT_RAISE: raised as expected` lines, terminated with `ALL TESTS PASSED`. Run this on **any** change to `sql_guard.py` / `validate_sql.py`. -The canonical validator lives in `pyspark-sql-guardrails/scripts/sql_guard.py` and exports `assert_select_or_insert(sql)` and `safe_spark_sql(spark, sql)`. Do not redefine the regex inline at call sites; import from this module. +The canonical validator lives in `skills/pyspark-sql-guardrails/scripts/sql_guard.py` and exports `assert_select_or_insert(sql)` and `safe_spark_sql(spark, sql)`. Do not redefine the regex inline at call sites; import from this module. ### Validating metadata files @@ -65,7 +82,17 @@ The canonical validator lives in `pyspark-sql-guardrails/scripts/sql_guard.py` a ### Evals -`requirements-analysis/evals/evals.json` contains 3 prompt/expected-output pairs for the requirements-analysis skill. There is no test runner — the file documents expected behavior for manual or harness-driven eval. +`skills/requirements-analysis/evals/evals.json` contains 3 prompt/expected-output pairs for the requirements-analysis skill. There is no test runner — the file documents expected behavior for manual or harness-driven eval. + +## Skill file format + +Every skill is a single `SKILL.md` with YAML frontmatter (`name`, `description`) describing when to invoke the skill. New skills must follow this format; do not introduce other conventions. The first paragraph of `description` is the trigger surface — keep it concrete (input contract + trigger phrases). + +## Hard rule: the validation gate + +`skills/pyspark-sql-guardrails/SKILL.md` enforces a non-negotiable gate: **every generated SQL string must pass through `assert_select_or_insert()` before it is shown to the user**. The pipeline ordering and the gate are independent — even if `sql-review` already passed, the SQL still has to pass through this guardrail. + +The forbidden-behaviors table in `skills/pyspark-sql-guardrails/SKILL.md` enumerates every rationalization for skipping the validator; treat the list as closed. ## Pipeline architecture @@ -83,7 +110,8 @@ The two non-pipeline skills (`logic-planner` taxonomy details, `sql-review` chec ## Working with this repo -- **Adding a new skill** — create `/SKILL.md` with the YAML frontmatter and follow the same input/output/status contract used by existing skills. If the skill emits SQL, it must be downstream of `pyspark-sql-guardrails` (or use it as a library). +- **Adding a new skill** — create `skills//SKILL.md` with the YAML frontmatter and follow the same input/output/status contract used by existing skills. If the skill emits SQL, it must be downstream of `pyspark-sql-guardrails` (or use it as a library). - **Modifying `sql_guard.py`** — run `test_sql_guard.py` after every change. The 18-case test set is the contract; do not weaken it to make a SQL string pass. - **Pipeline status changes** — never collapse gates without user confirmation. The pipeline's value comes from each stage being a separate, re-runnable hand-off; inlining them couples them. - **Triggers in `description:` frontmatter** — these are what Claude Code matches against user prompts. Be specific (input contract + concrete trigger phrases). Vague triggers cause skill over-invocation. +- **Stale path references** — the bundled SKILL.md / scripts use `.claude/skills/...` paths from an earlier layout. When updating validator instructions, prefer the actual paths (`skills/...` on host, `/root/.config/opencode/skills/...` in image). diff --git a/README.md b/README.md index 6a59fb7..109dc2d 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,61 @@ # opencode -custome opencode image build \ No newline at end of file +自定义 `opencode-ai` Docker 镜像构建仓库。镜像内置 Python 数据栈依赖与一套用于把业务数据需求转成可执行 PySpark SQL 的 Claude Code skills。 + +## 包含什么 + +- **Node.js 22** + **`opencode-ai` CLI**（全局安装） +- **Python 3.11**（通过 [`uv`](https://github.com/astral-sh/uv) 管理）+ 数据栈依赖（`requirements.txt`）：PySpark、pandas、polars、clickhouse-connect、pymongo、redis、hdfs 等 +- **ripgrep**（预编译二进制） +- **`skills/`** 自动部署到 `/root/.config/opencode/skills/`，Claude Code 启动时自动发现 + +## 构建 + +```bash +docker build -t opencode-custom:latest . +``` + +Dockerfile 为多阶段构建：builder 阶段拉取 Node.js 22 与 Python 3.11，最终镜像只保留运行所需文件。 + +## 运行 + +```bash +docker run --rm -it opencode-custom:latest +``` + +ENTRYPOINT 为 `opencode`，可直接传入 Claude Code 参数。 + +## Skills 概览 + +`skills/` 下是一套端到端流水线，把中文业务数据需求一步步推到可评审的 PySpark SQL： + +``` +requirements-analysis → metadata-validator → logic-planner + → sql-context-builder → (SQL 生成) → pyspark-sql-guardrails → sql-review +``` + +`pyspark-sql-pipeline` 是 orchestrator，负责在阶段之间维护用户确认门。 + +详细的输入/输出契约、状态名（`READY_FOR_VALIDATION` / `VALIDATED` / `PLANNED` / `READY` / `PASS` / `FAIL`）、以及 SQL 校验闸门规则见 [`CLAUDE.md`](./CLAUDE.md)。 + +## 目录结构 + +``` +. +├── Dockerfile # 多阶段镜像构建 +├── requirements.txt # Python 数据栈依赖 +├── ripgrep-*.tar.gz # ripgrep 预编译包（构建时复制进镜像） +├── skills/ +│ ├── requirements-analysis/ # 需求解读与 DRD 输出 +│ ├── metadata-validator/ # 表/字段/关联验证 +│ ├── logic-planner/ # 业务逻辑拆解（固定步骤分类法） +│ ├── sql-context-builder/ # 冻结字段别名、类型、角色标签 +│ ├── pyspark-sql-guardrails/ # SQL 白名单校验器（闸门） +│ │ └── scripts/ +│ │ ├── sql_guard.py # 规范实现 +│ │ ├── validate_sql.py # CLI 入口 +│ │ └── test_sql_guard.py # 18 个用例的标准测试集 +│ ├── sql-review/ # 静态评审（10 项检查） +│ └── pyspark-sql-pipeline/ # 全流程 orchestrator +└── CLAUDE.md # 给 Claude Code 的详细指南 +```