tao.chenandClaude
97f6184f65
cluster.DefaultSubmitArgs is now deprecated. It used to be prepended to
spark_submit Tool calls, but since 7920dc9 the spark-submit builder
constructs argv from the structured fields in the tool call. The field is
still accepted by the admin API and still persisted to the DB, so this is
a non-breaking change.
This commit deliberately does NOT remove:
- the Cluster.DefaultSubmitArgs field in internal/cluster/types.go
- the default_submit_args DB column
- the admin API read/write support in internal/admin/router.go
- existing storage/repo tests
Runtime now logs a one-time slog.Info when the PUT handler receives a
non-empty default_submit_args value, warning operators that the field is
ignored by spark_submit and that they should use spark_conf / extra_args
instead.
Follow-up removal should touch:
- the Cluster struct field and its JSON tag
- storage/cluster_repo.go scan/insert/update SQL and params
- admin/router.go JSON round-tripping
- storage/cluster_repo_test.go assertions
- the DB migration dropping the default_submit_args column
Co-Authored-By: Claude <noreply@anthropic.com>
62 lines
2.9 KiB
Go
62 lines
2.9 KiB
Go
// Package cluster defines the Cluster domain type — a configured Spark/YARN
|
|
// endpoint pair plus authentication, SSL, rate-limit, and Spark-submit metadata.
|
|
//
|
|
// Cluster is the only entity persisted in this project besides the audit log.
|
|
// It is the discovery root for the LLM agent: list_clusters returns the full
|
|
// struct (minus the password) so the agent can resolve RM/SHS URLs, choose
|
|
// the correct auth flavor, and respect the URL allowlist.
|
|
package cluster
|
|
|
|
import "time"
|
|
|
|
// AuthType enumerates the YARN/Hadoop authentication flavors supported in v1.
|
|
//
|
|
// "none" — no credentials; public or pre-trusted clusters
|
|
// "simple" — YARN SimpleAuth: appends ?user.name=<username> to every request
|
|
// "basic" — HTTP Basic; AuthUsername + AuthPassword (encrypted at rest)
|
|
//
|
|
// Kerberos is intentionally absent in v1; the deployment uses SimpleAuth.
|
|
type AuthType string
|
|
|
|
const (
|
|
AuthNone AuthType = "none"
|
|
AuthSimple AuthType = "simple"
|
|
AuthBasic AuthType = "basic"
|
|
)
|
|
|
|
// Cluster is one configured Spark-on-YARN endpoint pair.
|
|
//
|
|
// Field semantics:
|
|
// - SparkSubmitExecuteBin: absolute path to spark-submit (or spark2-submit);
|
|
// exec.Command uses this binary directly, never the shell.
|
|
// - IsActive: only active clusters are returned by LLM-facing list_clusters.
|
|
// - AuthPassword: zero value ("") on Update means "do not touch the stored
|
|
// password" — keeps the existing encrypted blob intact.
|
|
// - URLAllowlist: extra glob patterns beyond the RM/SHS hosts; fetch_url
|
|
// uses this to permit long-tail SHS endpoints the agent may construct.
|
|
// - DefaultSubmitArgs: deprecated; no longer prepended by spark_submit
|
|
// (post-7920dc9 the builder constructs argv from structured fields).
|
|
// Kept for backward compatibility with the admin API and DB schema;
|
|
// actual removal is a follow-up. New cluster configurations should
|
|
// leave this empty.
|
|
// - RateLimitPerMin: 0 disables the per-cluster limiter.
|
|
type Cluster struct {
|
|
ID string `json:"id"`
|
|
Name string `json:"name"`
|
|
RMURL string `json:"rm_url"`
|
|
SHSURL string `json:"shs_url"`
|
|
SparkSubmitExecuteBin string `json:"spark_submit_execute_bin"`
|
|
IsActive bool `json:"is_active"`
|
|
AuthType AuthType `json:"auth_type"`
|
|
AuthUsername string `json:"auth_username,omitempty"`
|
|
AuthPassword string `json:"-"` // never serialized; encrypted at rest in auth_password_enc
|
|
SSLVerify bool `json:"ssl_verify"`
|
|
SSLCABundle string `json:"ssl_ca_bundle,omitempty"`
|
|
URLAllowlist []string `json:"url_allowlist,omitempty"`
|
|
// Deprecated: see godoc.
|
|
DefaultSubmitArgs []string `json:"default_submit_args,omitempty"`
|
|
RateLimitPerMin int `json:"rate_limit_per_min"`
|
|
CreatedAt time.Time `json:"created_at"`
|
|
UpdatedAt time.Time `json:"updated_at"`
|
|
}
|