48 lines
1.8 KiB
Markdown
48 lines
1.8 KiB
Markdown
---
|
|
name: pyspark-script-builder
|
|
description: Use after sql-review to wrap reviewed_sql into a complete PySpark script. Inputs are reviewed_sql, app_name, and script_name; ask the user for app_name or script_name if missing; preserve reviewed_sql business logic and output app_name, script_name, and pyspark_code only.
|
|
---
|
|
|
|
# PySpark Script Builder
|
|
|
|
Use this skill only after `sql-review`, when `reviewed_sql` is already available and the user needs it wrapped as a complete PySpark script.
|
|
|
|
## Required Inputs
|
|
|
|
- `reviewed_sql`: SQL text that has already been reviewed.
|
|
- `app_name`: Spark application name. If missing, ask the user before producing output. Do not invent or derive it.
|
|
- `script_name`: Script file name. If missing, ask the user before producing output. Do not invent or derive it.
|
|
|
|
## Rules
|
|
|
|
- Preserve the business logic of `reviewed_sql`. Do not rewrite tables, columns, predicates, joins, aggregations, ordering, limits, comments, or SQL semantics.
|
|
- Only wrap `reviewed_sql` in the fixed PySpark script structure below.
|
|
- Do not add file persistence, job creation, confirmation, cluster launch, queue/resource selection, monitoring, logs, or lifecycle steps.
|
|
- Escape `app_name` only as needed to keep a valid Python double-quoted string.
|
|
- Put `reviewed_sql` inside a raw triple-quoted string exactly in the `sql = r"""..."""` assignment.
|
|
- Do not run the SQL or validate it again; this skill only builds script text.
|
|
|
|
## Output
|
|
|
|
Return only these fields:
|
|
|
|
- `app_name`
|
|
- `script_name`
|
|
- `pyspark_code`
|
|
|
|
Build `pyspark_code` with this exact structure:
|
|
|
|
```python
|
|
from pyspark.sql import SparkSession
|
|
|
|
APP_NAME = "<app_name>"
|
|
|
|
def main():
|
|
spark = SparkSession.builder.appName(APP_NAME).enableHiveSupport().getOrCreate()
|
|
sql = r"""<reviewed_sql>"""
|
|
spark.sql(sql)
|
|
spark.stop()
|
|
|
|
if __name__ == "__main__": main()
|
|
```
|