update chinese skill
This commit is contained in:
@@ -113,7 +113,7 @@ logic_plan:
|
|||||||
right: { alias: c, table: customer_info }
|
right: { alias: c, table: customer_info }
|
||||||
keys: ["o.customer_id = c.customer_id"]
|
keys: ["o.customer_id = c.customer_id"]
|
||||||
type: left
|
type: left
|
||||||
cardinality_assumption: 1:N # 已知时填;否则写 "unknown"
|
cardinality_assumption: 1:N # if known; else state "unknown"
|
||||||
- kind: transform
|
- kind: transform
|
||||||
alias: o
|
alias: o
|
||||||
expressions:
|
expressions:
|
||||||
|
|||||||
@@ -12,13 +12,13 @@ description: 在编写或评审 PySpark 脚本、Spark SQL 字符串、`spark.sq
|
|||||||
整条流程就是两条命令 + 一个条件:
|
整条流程就是两条命令 + 一个条件:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 第 1 步 — 生成 SQL(脑中起草 / 写出来)
|
# Step 1 — Generate the SQL (do this in your head / draft)
|
||||||
# 第 2 步 — 立即把 SQL 喂给校验器(在同一轮里跑)
|
# Step 2 — Run the validator on it (do this IMMEDIATELY, in the same turn)
|
||||||
echo "<the SQL you just wrote>" | python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py
|
echo "<the SQL you just wrote>" | python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py
|
||||||
|
|
||||||
# 第 3 步 — 读输出:
|
# Step 3 — Read the output:
|
||||||
# PASS - first keyword: select, body length: N → 可以把 SQL 给用户看了
|
# PASS - first keyword: select, body length: N → you may now show the SQL
|
||||||
# FAIL - <reason> → 停下,修 SQL,再跑一次
|
# FAIL - <reason> → STOP. Fix the SQL. Re-run.
|
||||||
```
|
```
|
||||||
|
|
||||||
**如果你还没把 `PASS -` 这一行贴到对话里,就不允许贴 SQL。** 校验器就是关口,SQL 站在关口的另一边,绝不能空着手闯关。
|
**如果你还没把 `PASS -` 这一行贴到对话里,就不允许贴 SQL。** 校验器就是关口,SQL 站在关口的另一边,绝不能空着手闯关。
|
||||||
@@ -46,21 +46,21 @@ echo "<the SQL you just wrote>" | python3 skills/pyspark-sql-guardrails/scripts/
|
|||||||
唯一可接受的调用形式:
|
唯一可接受的调用形式:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
# 推荐:封装函数,校验+执行一步到位
|
# Preferred: wrapper that validates + executes atomically
|
||||||
safe_spark_sql(spark, sql)
|
safe_spark_sql(spark, sql)
|
||||||
|
|
||||||
# 或者:先校验,再显式执行
|
# Or: validate first, then execute explicitly
|
||||||
assert_select_or_insert(sql) # 违规即抛错
|
assert_select_or_insert(sql) # raises on violation
|
||||||
spark.sql(sql)
|
spark.sql(sql)
|
||||||
```
|
```
|
||||||
|
|
||||||
**禁止的调用形式(出现即视为流水线失败):**
|
**禁止的调用形式(出现即视为流水线失败):**
|
||||||
|
|
||||||
```python
|
```python
|
||||||
spark.sql(sql) # 直接调用,未校验
|
spark.sql(sql) # direct, no validation
|
||||||
spark.sql(f"SELECT ... {user_input} ...") # f-string 直接进 spark.sql
|
spark.sql(f"SELECT ... {user_input} ...") # f-string into spark.sql
|
||||||
spark.sql(config["sql"]) # 配置驱动的 SQL 未校验
|
spark.sql(config["sql"]) # config-driven without validation
|
||||||
spark.sql(open("queries/xxx.sql").read()) # 从文件加载的 SQL 未校验
|
spark.sql(open("queries/xxx.sql").read()) # file-loaded without validation
|
||||||
```
|
```
|
||||||
|
|
||||||
如果 `assert_select_or_insert()` 抛错,流水线立即停止。**不要**削弱规则,**不要**用正则去剥除禁用关键字,**不要**把 SQL"改写"成看似安全的样子。要么拒绝,要么重新生成。
|
如果 `assert_select_or_insert()` 抛错,流水线立即停止。**不要**削弱规则,**不要**用正则去剥除禁用关键字,**不要**把 SQL"改写"成看似安全的样子。要么拒绝,要么重新生成。
|
||||||
@@ -115,7 +115,7 @@ def assert_select_or_insert(sql: str) -> str:
|
|||||||
if not text:
|
if not text:
|
||||||
raise ValueError("SQL is empty")
|
raise ValueError("SQL is empty")
|
||||||
|
|
||||||
# 允许一个可选的尾随分号,但拒绝多条堆叠的语句
|
# Allow one optional trailing semicolon, but reject stacked statements.
|
||||||
body = text[:-1].strip() if text.endswith(";") else text
|
body = text[:-1].strip() if text.endswith(";") else text
|
||||||
if ";" in body:
|
if ";" in body:
|
||||||
raise ValueError("Multiple SQL statements are not allowed")
|
raise ValueError("Multiple SQL statements are not allowed")
|
||||||
@@ -138,13 +138,13 @@ def assert_select_or_insert(sql: str) -> str:
|
|||||||
def safe_spark_sql(spark, sql: str):
|
def safe_spark_sql(spark, sql: str):
|
||||||
return spark.sql(assert_select_or_insert(sql))
|
return spark.sql(assert_select_or_insert(sql))
|
||||||
|
|
||||||
# 正确
|
# Good
|
||||||
safe_spark_sql(spark, """
|
safe_spark_sql(spark, """
|
||||||
INSERT INTO analytics.daily_customer_snapshot
|
INSERT INTO analytics.daily_customer_snapshot
|
||||||
SELECT * FROM staging.daily_customer_snapshot
|
SELECT * FROM staging.daily_customer_snapshot
|
||||||
""")
|
""")
|
||||||
|
|
||||||
# 错误:在 Spark 看到 SQL 之前就抛错
|
# Bad: raises before Spark sees it
|
||||||
safe_spark_sql(spark, "ALTER TABLE analytics.daily_customer_snapshot DROP PARTITION (dt='2026-06-10')")
|
safe_spark_sql(spark, "ALTER TABLE analytics.daily_customer_snapshot DROP PARTITION (dt='2026-06-10')")
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -158,11 +158,11 @@ safe_spark_sql(spark, "ALTER TABLE analytics.daily_customer_snapshot DROP PARTIT
|
|||||||
|
|
||||||
```
|
```
|
||||||
skills/pyspark-sql-guardrails/
|
skills/pyspark-sql-guardrails/
|
||||||
├── SKILL.md ← 本文件
|
├── SKILL.md ← this file
|
||||||
└── scripts/
|
└── scripts/
|
||||||
├── sql_guard.py ← 权威校验器(assert_select_or_insert + safe_spark_sql)
|
├── sql_guard.py ← canonical validator (assert_select_or_insert + safe_spark_sql)
|
||||||
├── validate_sql.py ← 一行 CLI:管道喂 SQL,输出 PASS/FAIL
|
├── validate_sql.py ← one-line CLI: pipe SQL in, get PASS/FAIL out
|
||||||
└── test_sql_guard.py ← 18 用例的标准测试集
|
└── test_sql_guard.py ← 18-case standard test set
|
||||||
```
|
```
|
||||||
|
|
||||||
### 第 1 步 — 用内置校验器(不复制、不重写)
|
### 第 1 步 — 用内置校验器(不复制、不重写)
|
||||||
@@ -172,7 +172,7 @@ skill 自带一个一行 CLI:`validate_sql.py`。用它。不要写不同的调
|
|||||||
**两种把 SQL 喂给校验器的方法:**
|
**两种把 SQL 喂给校验器的方法:**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 方法 A(推荐,适合多行):走 stdin
|
# Way A (preferred for multi-line): pipe via stdin
|
||||||
cat <<'EOF' | python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py
|
cat <<'EOF' | python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py
|
||||||
SELECT
|
SELECT
|
||||||
c.city AS city,
|
c.city AS city,
|
||||||
@@ -183,7 +183,7 @@ WHERE o.status = 'SUCCESS'
|
|||||||
GROUP BY c.city
|
GROUP BY c.city
|
||||||
EOF
|
EOF
|
||||||
|
|
||||||
# 方法 B(仅适合单行):作为第一个参数传入
|
# Way B (single-line only): pass as first argument
|
||||||
python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dual"
|
python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dual"
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -195,7 +195,7 @@ python3 skills/pyspark-sql-guardrails/scripts/validate_sql.py "SELECT 1 FROM dua
|
|||||||
内置的 `sql_guard.py` 和 `validate_sql.py` 一并展示:
|
内置的 `sql_guard.py` 和 `validate_sql.py` 一并展示:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
# sql_guard.py (内置)
|
# sql_guard.py (bundled)
|
||||||
import re
|
import re
|
||||||
|
|
||||||
_FORBIDDEN_SQL = re.compile(
|
_FORBIDDEN_SQL = re.compile(
|
||||||
@@ -227,7 +227,7 @@ def safe_spark_sql(spark, sql: str):
|
|||||||
return spark.sql(assert_select_or_insert(sql))
|
return spark.sql(assert_select_or_insert(sql))
|
||||||
|
|
||||||
|
|
||||||
# validate_sql.py (内置,精简版)
|
# validate_sql.py (bundled, abridged)
|
||||||
import sys, os
|
import sys, os
|
||||||
SKILL_DIR = os.path.dirname(os.path.abspath(__file__))
|
SKILL_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
sys.path.insert(0, SKILL_DIR)
|
sys.path.insert(0, SKILL_DIR)
|
||||||
@@ -265,10 +265,10 @@ PASS - first keyword: select, body length: 420
|
|||||||
skill 自带 `test_sql_guard.py`,含 6 个 EXPECT_OK + 12 个 EXPECT_RAISE 用例。每次校验器改动,以及在 CI 中,都要在 skill 目录下跑一遍:
|
skill 自带 `test_sql_guard.py`,含 6 个 EXPECT_OK + 12 个 EXPECT_RAISE 用例。每次校验器改动,以及在 CI 中,都要在 skill 目录下跑一遍:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 在项目根目录
|
# From project root
|
||||||
python3 skills/pyspark-sql-guardrails/scripts/test_sql_guard.py
|
python3 skills/pyspark-sql-guardrails/scripts/test_sql_guard.py
|
||||||
|
|
||||||
# 或在 scripts/ 目录下
|
# Or from the scripts/ directory
|
||||||
cd skills/pyspark-sql-guardrails/scripts
|
cd skills/pyspark-sql-guardrails/scripts
|
||||||
python3 test_sql_guard.py
|
python3 test_sql_guard.py
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -32,15 +32,15 @@ digraph pipeline {
|
|||||||
"2. metadata-validator" [shape=box];
|
"2. metadata-validator" [shape=box];
|
||||||
"3. logic-planner" [shape=box];
|
"3. logic-planner" [shape=box];
|
||||||
"4. sql-context-builder" [shape=box];
|
"4. sql-context-builder" [shape=box];
|
||||||
"5. SQL 生成" [shape=box];
|
"5. SQL generation" [shape=box];
|
||||||
"6. pyspark-sql-guardrails" [shape=box];
|
"6. pyspark-sql-guardrails" [shape=box];
|
||||||
"7. sql-review" [shape=box];
|
"7. sql-review" [shape=box];
|
||||||
|
|
||||||
"1. requirements-analysis" -> "2. metadata-validator" [label="status=READY_FOR_VALIDATION"];
|
"1. requirements-analysis" -> "2. metadata-validator" [label="status=READY_FOR_VALIDATION"];
|
||||||
"2. metadata-validator" -> "3. logic-planner" [label="status=VALIDATED"];
|
"2. metadata-validator" -> "3. logic-planner" [label="status=VALIDATED"];
|
||||||
"3. logic-planner" -> "4. sql-context-builder" [label="status=PLANNED"];
|
"3. logic-planner" -> "4. sql-context-builder" [label="status=PLANNED"];
|
||||||
"4. sql-context-builder" -> "5. SQL 生成" [label="status=READY"];
|
"4. sql-context-builder" -> "5. SQL generation" [label="status=READY"];
|
||||||
"5. SQL 生成" -> "6. pyspark-sql-guardrails" [label="SQL 字符串"];
|
"5. SQL generation" -> "6. pyspark-sql-guardrails" [label="SQL 字符串"];
|
||||||
"6. pyspark-sql-guardrails" -> "7. sql-review" [label="guardrail pass"];
|
"6. pyspark-sql-guardrails" -> "7. sql-review" [label="guardrail pass"];
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
@@ -84,13 +84,13 @@ digraph pipeline {
|
|||||||
每个关口应产出一条结构化消息,格式如下:
|
每个关口应产出一条结构化消息,格式如下:
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
## Pipeline paused at: <阶段名>
|
## Pipeline paused at: <stage name>
|
||||||
**Status:** NEED_USER_CONFIRMATION
|
**Status:** NEED_USER_CONFIRMATION
|
||||||
**Open questions:**
|
**Open questions:**
|
||||||
1. <问题 1>
|
1. <question 1>
|
||||||
2. <问题 2>
|
2. <question 2>
|
||||||
**Pending options:** A / B / C / other
|
**Pending options:** A / B / C / other
|
||||||
**To resume:** 回答以上问题后,说"继续"。
|
**To resume:** answer the questions above, then say "继续" (continue).
|
||||||
```
|
```
|
||||||
|
|
||||||
不要把某个阶段的输出当作最终结果,如果它的状态是停止状态。不要把 `pending_questions` 埋在冗长叙述里。
|
不要把某个阶段的输出当作最终结果,如果它的状态是停止状态。不要把 `pending_questions` 埋在冗长叙述里。
|
||||||
|
|||||||
Reference in New Issue
Block a user