update chinese skill
This commit is contained in:
tao.chen
@@ -113,7 +113,7 @@ logic_plan:
|
||||
right: { alias: c, table: customer_info }
|
||||
keys: ["o.customer_id = c.customer_id"]
|
||||
type: left
|
||||
cardinality_assumption: 1:N # 已知时填;否则写 "unknown"
|
||||
cardinality_assumption: 1:N # if known; else state "unknown"
|
||||
- kind: transform
|
||||
alias: o
|
||||
expressions:
|
||||
|
||||
@@ -12,13 +12,13 @@ description: 在编写或评审 PySpark 脚本、Spark SQL 字符串、`spark.sq
|
||||
整条流程就是两条命令 + 一个条件:
|
||||
|
||||
```bash
|
||||
# 第 1 步 — 生成 SQL(脑中起草 / 写出来)
|
||||
# 第 2 步 — 立即把 SQL 喂给校验器(在同一轮里跑)
|
||||
# Step 1 — Generate the SQL (do this in your head / draft)
|
||||
# Step 2 — Run the validator on it (do this IMMEDIATELY, in the same turn)
|
||||
echo "<the SQL you just wrote>" | python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py
|
||||
|
||||
# 第 3 步 — 读输出:
|
||||
# PASS - first keyword: select, body length: N → 可以把 SQL 给用户看了
|
||||
# FAIL - <reason> → 停下,修 SQL,再跑一次
|
||||
# Step 3 — Read the output:
|
||||
# PASS - first keyword: select, body length: N → you may now show the SQL
|
||||
# FAIL - <reason> → STOP. Fix the SQL. Re-run.
|
||||
```
|
||||
|
||||
**如果你还没把 `PASS -` 这一行贴到对话里,就不允许贴 SQL。** 校验器就是关口,SQL 站在关口的另一边,绝不能空着手闯关。
|
||||
@@ -46,21 +46,21 @@ echo "<the SQL you just wrote>" | python3 skills/pyspark-sql-guardrails/scripts/
|
||||
唯一可接受的调用形式:
|
||||
|
||||
```python
|
||||
# 推荐:封装函数,校验+执行一步到位
|
||||
# Preferred: wrapper that validates + executes atomically
|
||||
safe_spark_sql(spark, sql)
|
||||
|
||||
# 或者:先校验,再显式执行
|
||||
assert_select_or_insert(sql) # 违规即抛错
|
||||
# Or: validate first, then execute explicitly
|
||||
assert_select_or_insert(sql) # raises on violation
|
||||
spark.sql(sql)
|
||||
```
|
||||
|
||||
**禁止的调用形式(出现即视为流水线失败):**
|
||||
|
||||
```python
|
||||
spark.sql(sql) # 直接调用,未校验
|
||||
spark.sql(f"SELECT ... {user_input} ...") # f-string 直接进 spark.sql
|
||||
spark.sql(config["sql"]) # 配置驱动的 SQL 未校验
|
||||
spark.sql(open("queries/xxx.sql").read()) # 从文件加载的 SQL 未校验
|
||||
spark.sql(sql) # direct, no validation
|
||||
spark.sql(f"SELECT ... {user_input} ...") # f-string into spark.sql
|
||||
spark.sql(config["sql"]) # config-driven without validation
|
||||
spark.sql(open("queries/xxx.sql").read()) # file-loaded without validation
|
||||
```
|
||||
|
||||
如果 `assert_select_or_insert()` 抛错,流水线立即停止。**不要**削弱规则,**不要**用正则去剥除禁用关键字,**不要**把 SQL"改写"成看似安全的样子。要么拒绝,要么重新生成。
|
||||
@@ -115,7 +115,7 @@ def assert_select_or_insert(sql: str) -> str:
|
||||
if not text:
|
||||
raise ValueError("SQL is empty")
|
||||
|
||||
# 允许一个可选的尾随分号,但拒绝多条堆叠的语句
|
||||
# Allow one optional trailing semicolon, but reject stacked statements.
|
||||
body = text[:-1].strip() if text.endswith(";") else text
|
||||
if ";" in body:
|
||||
raise ValueError("Multiple SQL statements are not allowed")
|
||||
@@ -138,13 +138,13 @@ def assert_select_or_insert(sql: str) -> str:
|
||||
def safe_spark_sql(spark, sql: str):
|
||||
return spark.sql(assert_select_or_insert(sql))
|
||||
|
||||
# 正确
|
||||
# Good
|
||||
safe_spark_sql(spark, """
|
||||
INSERT INTO analytics.daily_customer_snapshot
|
||||
SELECT * FROM staging.daily_customer_snapshot
|
||||
""")
|
||||
|
||||
# 错误:在 Spark 看到 SQL 之前就抛错
|
||||
# Bad: raises before Spark sees it
|
||||
safe_spark_sql(spark, "ALTER TABLE analytics.daily_customer_snapshot DROP PARTITION (dt='2026-06-10')")
|
||||
```
|
||||
|
||||
@@ -158,11 +158,11 @@ safe_spark_sql(spark, "ALTER TABLE analytics.daily_customer_snapshot DROP PARTIT
|
||||
|
||||
```
|
||||
skills/pyspark-sql-guardrails/
|
||||
├── SKILL.md ← 本文件
|
||||
├── SKILL.md ← this file
|
||||
└── scripts/
|
||||
├── sql_guard.py ← 权威校验器(assert_select_or_insert + safe_spark_sql)
|
||||
├── validate_sql.py ← 一行 CLI:管道喂 SQL,输出 PASS/FAIL
|
||||
└── test_sql_guard.py ← 18 用例的标准测试集
|
||||
├── sql_guard.py ← canonical validator (assert_select_or_insert + safe_spark_sql)
|
||||
├── validate_sql.py ← one-line CLI: pipe SQL in, get PASS/FAIL out
|
||||
└── test_sql_guard.py ← 18-case standard test set
|
||||
```
|
||||
|
||||
### 第 1 步 — 用内置校验器(不复制、不重写)
|
||||
@@ -172,7 +172,7 @@ skill 自带一个一行 CLI:`validate_sql.py`。用它。不要写不同的调
|
||||
**两种把 SQL 喂给校验器的方法:**
|
||||
|
||||
```bash
|
||||
# 方法 A(推荐,适合多行):走 stdin
|
||||
# Way A (preferred for multi-line): pipe via stdin
|
||||
cat <<'EOF' | python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py
|
||||
SELECT
|
||||
c.city AS city,
|
||||
@@ -183,7 +183,7 @@ WHERE o.status = 'SUCCESS'
|
||||
GROUP BY c.city
|
||||
EOF
|
||||
|
||||
# 方法 B(仅适合单行):作为第一个参数传入
|
||||
# Way B (single-line only): pass as first argument
|
||||
python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dual"
|
||||
```
|
||||
|
||||
@@ -195,7 +195,7 @@ python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dua
|
||||
内置的 `sql_guard.py` 和 `validate_sql.py` 一并展示:
|
||||
|
||||
```python
|
||||
# sql_guard.py (内置)
|
||||
# sql_guard.py (bundled)
|
||||
import re
|
||||
|
||||
_FORBIDDEN_SQL = re.compile(
|
||||
@@ -227,7 +227,7 @@ def safe_spark_sql(spark, sql: str):
|
||||
return spark.sql(assert_select_or_insert(sql))
|
||||
|
||||
|
||||
# validate_sql.py (内置,精简版)
|
||||
# validate_sql.py (bundled, abridged)
|
||||
import sys, os
|
||||
SKILL_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, SKILL_DIR)
|
||||
@@ -265,10 +265,10 @@ PASS - first keyword: select, body length: 420
|
||||
skill 自带 `test_sql_guard.py`,含 6 个 EXPECT_OK + 12 个 EXPECT_RAISE 用例。每次校验器改动,以及在 CI 中,都要在 skill 目录下跑一遍:
|
||||
|
||||
```bash
|
||||
# 在项目根目录
|
||||
# From project root
|
||||
python3 skills/pyspark-sql-guardrails/scripts/test_sql_guard.py
|
||||
|
||||
# 或在 scripts/ 目录下
|
||||
# Or from the scripts/ directory
|
||||
cd skills/pyspark-sql-guardrails/scripts
|
||||
python3 test_sql_guard.py
|
||||
```
|
||||
|
||||
@@ -32,15 +32,15 @@ digraph pipeline {
|
||||
"2. metadata-validator" [shape=box];
|
||||
"3. logic-planner" [shape=box];
|
||||
"4. sql-context-builder" [shape=box];
|
||||
"5. SQL 生成" [shape=box];
|
||||
"5. SQL generation" [shape=box];
|
||||
"6. pyspark-sql-guardrails" [shape=box];
|
||||
"7. sql-review" [shape=box];
|
||||
|
||||
"1. requirements-analysis" -> "2. metadata-validator" [label="status=READY_FOR_VALIDATION"];
|
||||
"2. metadata-validator" -> "3. logic-planner" [label="status=VALIDATED"];
|
||||
"3. logic-planner" -> "4. sql-context-builder" [label="status=PLANNED"];
|
||||
"4. sql-context-builder" -> "5. SQL 生成" [label="status=READY"];
|
||||
"5. SQL 生成" -> "6. pyspark-sql-guardrails" [label="SQL 字符串"];
|
||||
"4. sql-context-builder" -> "5. SQL generation" [label="status=READY"];
|
||||
"5. SQL generation" -> "6. pyspark-sql-guardrails" [label="SQL 字符串"];
|
||||
"6. pyspark-sql-guardrails" -> "7. sql-review" [label="guardrail pass"];
|
||||
}
|
||||
```
|
||||
@@ -84,13 +84,13 @@ digraph pipeline {
|
||||
每个关口应产出一条结构化消息,格式如下:
|
||||
|
||||
```markdown
|
||||
## Pipeline paused at: <阶段名>
|
||||
## Pipeline paused at: <stage name>
|
||||
**Status:** NEED_USER_CONFIRMATION
|
||||
**Open questions:**
|
||||
1. <问题 1>
|
||||
2. <问题 2>
|
||||
1. <question 1>
|
||||
2. <question 2>
|
||||
**Pending options:** A / B / C / other
|
||||
**To resume:** 回答以上问题后,说"继续"。
|
||||
**To resume:** answer the questions above, then say "继续" (continue).
|
||||
```
|
||||
|
||||
不要把某个阶段的输出当作最终结果,如果它的状态是停止状态。不要把 `pending_questions` 埋在冗长叙述里。
|
||||
|
||||
Reference in New Issue
Block a user