重构模型平台前后端并移除Redis依赖
This commit is contained in:
Winnie
@@ -0,0 +1,28 @@
|
||||
# Migrations
|
||||
|
||||
MySQL 8 Alembic 迁移目录。
|
||||
|
||||
当前链路：
|
||||
|
||||
```text
|
||||
20260724_0001 完整业务 Schema 基线
|
||||
20260728_0002 Demo 用户、角色、Workspace 数据
|
||||
20260728_0003 调度稳定版本可见性字段
|
||||
20260730_0004 编辑租约字段 redis_lock_key -> lock_key 兼容迁移
|
||||
```
|
||||
|
||||
Docker Compose 中的 `migrate` 一次性容器会在 Backend 和 Runtime 启动前执行：
|
||||
|
||||
```bash
|
||||
alembic upgrade head
|
||||
```
|
||||
|
||||
本地执行：
|
||||
|
||||
```bash
|
||||
export DATABASE_URL='mysql+asyncmy://user:password@127.0.0.1:3308/model_platform?charset=utf8mb4'
|
||||
uv run --package backend alembic current
|
||||
uv run --package backend alembic upgrade head
|
||||
```
|
||||
|
||||
已发布的旧 revision 不应再修改；后续表结构调整必须新增 revision。
|
||||
@@ -0,0 +1 @@
|
||||
"""Alembic schema and controlled data migrations."""
|
||||
@@ -0,0 +1,69 @@
|
||||
# Data Migrations
|
||||
|
||||
该目录保存从旧版 JSON 状态文件迁移到 MySQL 的一次性工具。
|
||||
|
||||
数据迁移工具必须满足：
|
||||
|
||||
- 默认只预检，必须显式传入 `--apply` 才能写数据库；
|
||||
- 单个事务提交，失败时完整回滚；
|
||||
- 可以安全重复执行，不产生重复数据；
|
||||
- 输出源文件摘要、源数据数量和实际变更数量；
|
||||
- 不修改或删除旧版 JSON 源文件。
|
||||
|
||||
## system.json
|
||||
|
||||
```powershell
|
||||
$env:DATABASE_URL = "mysql+asyncmy://<user>:<password>@<host>:3306/<database>?charset=utf8mb4"
|
||||
python -m migrations.data.migrate_system_json --source "<path>/system.json"
|
||||
python -m migrations.data.migrate_system_json --source "<path>/system.json" --apply
|
||||
```
|
||||
|
||||
新迁移用户使用不可登录的占位密码哈希。后续接入认证时，必须通过密码初始化、
|
||||
管理员重置或外部身份认证启用登录，不能把旧版无密码账号视为已有凭据。
|
||||
|
||||
## WORKSPACE_DEFINITIONS
|
||||
|
||||
旧版 Workspace 定义位于 `文件1/server.py` 的
|
||||
`WORKSPACE_DEFINITIONS` 常量中。迁移工具通过 Python AST 仅读取这一静态常量，
|
||||
不执行旧服务代码：
|
||||
|
||||
```powershell
|
||||
python -m migrations.data.migrate_legacy_workspaces --source "<path>/server.py"
|
||||
python -m migrations.data.migrate_legacy_workspaces --source "<path>/server.py" --apply
|
||||
```
|
||||
|
||||
- 创建者取每个 Workspace 成员中的第一个管理员；
|
||||
- 成员角色沿用第 9 步迁入的平台注册角色；
|
||||
- 活跃目录统一为 `file:///workspace/workspaces/{workspace_code}`；
|
||||
- 制品前缀统一为 `workspaces/{workspace_id}`；
|
||||
- 旧审计记录仅在用户唯一属于一个 Workspace 时补齐归属。
|
||||
|
||||
## 第 9 小步执行记录
|
||||
|
||||
- 执行日期：2026-07-24
|
||||
- 源文件：`文件1/platform_data/system.json`
|
||||
- SHA-256：`ef4ee0e92f4a3679c17f23aff6066688fa9231db4477208e36fcbd343cd446a7`
|
||||
- 迁移结果：2 个角色、13 个权限、21 条角色权限、4 个用户、26 条审计记录
|
||||
- 角色权限：`admin=13`，`developer=8`
|
||||
- 中文字段：UTF-8 校验通过，问号乱码记录为 0
|
||||
- 幂等验证：第二次执行插入和更新均为 0，跳过已有审计记录 26 条
|
||||
- 旧版源文件：未修改、未删除
|
||||
|
||||
## 第 10 小步执行记录
|
||||
|
||||
- 执行日期：2026-07-24
|
||||
- 源文件及常量：`文件1/server.py` / `WORKSPACE_DEFINITIONS`
|
||||
- SHA-256：`d7ec05f1ab7b2cdeb78f6293de40fbe23fa74b6a7fe4ccb03a9cc64e4cc1f464`
|
||||
- 迁移结果：2 个 Workspace、4 条成员关系
|
||||
- 成员分布：`model-dev` 2 人，`risk-validation` 2 人
|
||||
- 审计归属：补齐 26 条，其中 `model-dev` 24 条、`risk-validation` 2 条
|
||||
- 中文字段：UTF-8 校验通过，异常记录为 0
|
||||
- 幂等验证：第二次执行及镜像内 dry-run 的变更数均为 0
|
||||
- 镜像验证：仅挂载旧版 `server.py` 时，迁移工具可独立读取并完成校验
|
||||
- 旧版源文件：未修改、未删除
|
||||
|
||||
## 第 11、12 小步数据处理决定
|
||||
|
||||
根据实施确认，第 11、12 小步不迁移旧版资源、脚本或稳定版本数据。新实现直接
|
||||
使用 MySQL、Workspace 文件目录和 RustFS，从空的 `data_resources`、`scripts`
|
||||
及 `versions` 表开始运行。功能验收产生的临时对象和数据库记录均已清理。
|
||||
@@ -0,0 +1 @@
|
||||
"""One-time, idempotent data migration tools."""
|
||||
@@ -0,0 +1,413 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, TypeAdapter
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from common.db import create_database_engine, create_session_factory
|
||||
from common.db.models import (
|
||||
AuditLogs,
|
||||
Roles,
|
||||
Users,
|
||||
WorkspaceMembers,
|
||||
Workspaces,
|
||||
)
|
||||
from migrations.data.migrate_system_json import (
|
||||
deterministic_legacy_ulid,
|
||||
set_changed,
|
||||
)
|
||||
|
||||
|
||||
class LegacyWorkspace(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
id: str
|
||||
name: str
|
||||
description: str | None = None
|
||||
userIds: list[str]
|
||||
portOffset: int
|
||||
|
||||
|
||||
WORKSPACE_LIST = TypeAdapter(list[LegacyWorkspace])
|
||||
|
||||
|
||||
def require_unique(values: list[str], label: str) -> None:
|
||||
duplicates = sorted(
|
||||
value for value in set(values) if values.count(value) > 1
|
||||
)
|
||||
if duplicates:
|
||||
raise ValueError(f"duplicate {label}: {duplicates}")
|
||||
|
||||
|
||||
def extract_workspace_definitions(
|
||||
source_path: Path,
|
||||
) -> tuple[list[LegacyWorkspace], str]:
|
||||
raw = source_path.read_bytes()
|
||||
module = ast.parse(
|
||||
raw.decode("utf-8-sig"),
|
||||
filename=str(source_path),
|
||||
)
|
||||
definition: Any | None = None
|
||||
|
||||
for statement in module.body:
|
||||
if not isinstance(statement, ast.Assign):
|
||||
continue
|
||||
if any(
|
||||
isinstance(target, ast.Name)
|
||||
and target.id == "WORKSPACE_DEFINITIONS"
|
||||
for target in statement.targets
|
||||
):
|
||||
definition = ast.literal_eval(statement.value)
|
||||
break
|
||||
|
||||
if definition is None:
|
||||
raise ValueError("WORKSPACE_DEFINITIONS was not found")
|
||||
|
||||
workspaces = WORKSPACE_LIST.validate_python(definition)
|
||||
require_unique([item.id for item in workspaces], "workspace ids")
|
||||
for workspace in workspaces:
|
||||
require_unique(
|
||||
workspace.userIds,
|
||||
f"members of workspace {workspace.id}",
|
||||
)
|
||||
if not workspace.userIds:
|
||||
raise ValueError(
|
||||
f"workspace {workspace.id!r} has no members"
|
||||
)
|
||||
|
||||
return workspaces, hashlib.sha256(raw).hexdigest()
|
||||
|
||||
|
||||
def new_stats() -> dict[str, int]:
|
||||
return {
|
||||
"workspaces_inserted": 0,
|
||||
"workspaces_updated": 0,
|
||||
"workspace_members_inserted": 0,
|
||||
"workspace_members_updated": 0,
|
||||
"audit_logs_workspace_backfilled": 0,
|
||||
}
|
||||
|
||||
|
||||
async def load_users_and_roles(
|
||||
session: AsyncSession,
|
||||
) -> tuple[dict[str, Users], dict[str, str]]:
|
||||
users = {
|
||||
item.username: item
|
||||
for item in (
|
||||
await session.scalars(select(Users).order_by(Users.username))
|
||||
).all()
|
||||
}
|
||||
roles_by_id = {
|
||||
item.role_id: item.role_code
|
||||
for item in (
|
||||
await session.scalars(select(Roles).order_by(Roles.role_code))
|
||||
).all()
|
||||
}
|
||||
role_codes_by_username = {
|
||||
username: roles_by_id[user.platform_role_id]
|
||||
for username, user in users.items()
|
||||
if user.platform_role_id in roles_by_id
|
||||
}
|
||||
return users, role_codes_by_username
|
||||
|
||||
|
||||
def validate_members(
|
||||
workspaces: list[LegacyWorkspace],
|
||||
users: dict[str, Users],
|
||||
role_codes_by_username: dict[str, str],
|
||||
) -> None:
|
||||
source_members = {
|
||||
username
|
||||
for workspace in workspaces
|
||||
for username in workspace.userIds
|
||||
}
|
||||
missing_users = sorted(source_members - set(users))
|
||||
if missing_users:
|
||||
raise ValueError(
|
||||
f"workspace members were not migrated as users: {missing_users}"
|
||||
)
|
||||
users_without_roles = sorted(
|
||||
source_members - set(role_codes_by_username)
|
||||
)
|
||||
if users_without_roles:
|
||||
raise ValueError(
|
||||
"workspace members have no platform role: "
|
||||
f"{users_without_roles}"
|
||||
)
|
||||
|
||||
|
||||
def workspace_creator(
|
||||
workspace: LegacyWorkspace,
|
||||
users: dict[str, Users],
|
||||
role_codes_by_username: dict[str, str],
|
||||
) -> Users:
|
||||
admin_username = next(
|
||||
(
|
||||
username
|
||||
for username in workspace.userIds
|
||||
if role_codes_by_username[username] == "admin"
|
||||
),
|
||||
workspace.userIds[0],
|
||||
)
|
||||
return users[admin_username]
|
||||
|
||||
|
||||
async def migrate_workspaces(
|
||||
session: AsyncSession,
|
||||
source: list[LegacyWorkspace],
|
||||
users: dict[str, Users],
|
||||
role_codes_by_username: dict[str, str],
|
||||
stats: dict[str, int],
|
||||
) -> dict[str, str]:
|
||||
existing = {
|
||||
item.workspace_code: item
|
||||
for item in (
|
||||
await session.scalars(
|
||||
select(Workspaces).order_by(Workspaces.workspace_code)
|
||||
)
|
||||
).all()
|
||||
}
|
||||
workspace_ids: dict[str, str] = {}
|
||||
|
||||
for legacy in source:
|
||||
workspace = existing.get(legacy.id)
|
||||
if workspace is None:
|
||||
workspace_id = deterministic_legacy_ulid(
|
||||
"workspace", legacy.id
|
||||
)
|
||||
creator = workspace_creator(
|
||||
legacy,
|
||||
users,
|
||||
role_codes_by_username,
|
||||
)
|
||||
workspace = Workspaces(
|
||||
workspace_id=workspace_id,
|
||||
workspace_code=legacy.id,
|
||||
workspace_name=legacy.name,
|
||||
description=legacy.description,
|
||||
active_root_uri=(
|
||||
"file:///workspace/workspaces/"
|
||||
f"{quote(legacy.id, safe='')}"
|
||||
),
|
||||
quota_bytes=0,
|
||||
used_bytes=0,
|
||||
status="active",
|
||||
created_by=creator.user_id,
|
||||
artifact_bucket="model-platform",
|
||||
artifact_prefix=f"workspaces/{workspace_id}",
|
||||
)
|
||||
session.add(workspace)
|
||||
stats["workspaces_inserted"] += 1
|
||||
else:
|
||||
values = {
|
||||
"workspace_name": legacy.name,
|
||||
"description": legacy.description,
|
||||
"active_root_uri": (
|
||||
"file:///workspace/workspaces/"
|
||||
f"{quote(legacy.id, safe='')}"
|
||||
),
|
||||
"artifact_bucket": "model-platform",
|
||||
"artifact_prefix": (
|
||||
f"workspaces/{workspace.workspace_id}"
|
||||
),
|
||||
}
|
||||
if set_changed(workspace, values):
|
||||
stats["workspaces_updated"] += 1
|
||||
workspace_ids[legacy.id] = workspace.workspace_id
|
||||
|
||||
return workspace_ids
|
||||
|
||||
|
||||
async def migrate_members(
|
||||
session: AsyncSession,
|
||||
source: list[LegacyWorkspace],
|
||||
workspace_ids: dict[str, str],
|
||||
users: dict[str, Users],
|
||||
stats: dict[str, int],
|
||||
) -> None:
|
||||
existing = {
|
||||
(item.workspace_id, item.user_id): item
|
||||
for item in (
|
||||
await session.scalars(select(WorkspaceMembers))
|
||||
).all()
|
||||
}
|
||||
|
||||
for legacy in source:
|
||||
workspace_id = workspace_ids[legacy.id]
|
||||
for username in legacy.userIds:
|
||||
user = users[username]
|
||||
pair = (workspace_id, user.user_id)
|
||||
member = existing.get(pair)
|
||||
values = {
|
||||
"role_id": user.platform_role_id,
|
||||
"member_status": "active",
|
||||
}
|
||||
if member is None:
|
||||
member = WorkspaceMembers(
|
||||
workspace_id=workspace_id,
|
||||
user_id=user.user_id,
|
||||
**values,
|
||||
)
|
||||
session.add(member)
|
||||
existing[pair] = member
|
||||
stats["workspace_members_inserted"] += 1
|
||||
elif set_changed(member, values):
|
||||
stats["workspace_members_updated"] += 1
|
||||
|
||||
|
||||
async def backfill_audit_workspaces(
|
||||
session: AsyncSession,
|
||||
source: list[LegacyWorkspace],
|
||||
workspace_ids: dict[str, str],
|
||||
users: dict[str, Users],
|
||||
stats: dict[str, int],
|
||||
) -> None:
|
||||
workspace_codes_by_username: dict[str, set[str]] = defaultdict(set)
|
||||
for workspace in source:
|
||||
for username in workspace.userIds:
|
||||
workspace_codes_by_username[username].add(workspace.id)
|
||||
|
||||
workspace_by_user_id = {
|
||||
users[username].user_id: workspace_ids[next(iter(codes))]
|
||||
for username, codes in workspace_codes_by_username.items()
|
||||
if len(codes) == 1
|
||||
}
|
||||
|
||||
audit_logs = (
|
||||
await session.scalars(
|
||||
select(AuditLogs).where(AuditLogs.workspace_id.is_(None))
|
||||
)
|
||||
).all()
|
||||
for audit_log in audit_logs:
|
||||
if (
|
||||
not isinstance(audit_log.detail_json, dict)
|
||||
or audit_log.detail_json.get("migration_source")
|
||||
!= "platform_data/system.json"
|
||||
):
|
||||
continue
|
||||
workspace_id = workspace_by_user_id.get(
|
||||
audit_log.actor_user_id
|
||||
)
|
||||
if workspace_id is not None:
|
||||
audit_log.workspace_id = workspace_id
|
||||
stats["audit_logs_workspace_backfilled"] += 1
|
||||
|
||||
|
||||
async def run_migration(
|
||||
database_url: str,
|
||||
source: list[LegacyWorkspace],
|
||||
*,
|
||||
apply_changes: bool,
|
||||
) -> dict[str, int]:
|
||||
engine = create_database_engine(database_url)
|
||||
factory = create_session_factory(engine)
|
||||
stats = new_stats()
|
||||
|
||||
try:
|
||||
async with factory() as session:
|
||||
try:
|
||||
users, role_codes_by_username = (
|
||||
await load_users_and_roles(session)
|
||||
)
|
||||
validate_members(
|
||||
source,
|
||||
users,
|
||||
role_codes_by_username,
|
||||
)
|
||||
workspace_ids = await migrate_workspaces(
|
||||
session,
|
||||
source,
|
||||
users,
|
||||
role_codes_by_username,
|
||||
stats,
|
||||
)
|
||||
await migrate_members(
|
||||
session,
|
||||
source,
|
||||
workspace_ids,
|
||||
users,
|
||||
stats,
|
||||
)
|
||||
await backfill_audit_workspaces(
|
||||
session,
|
||||
source,
|
||||
workspace_ids,
|
||||
users,
|
||||
stats,
|
||||
)
|
||||
|
||||
if apply_changes:
|
||||
await session.commit()
|
||||
else:
|
||||
await session.rollback()
|
||||
except Exception:
|
||||
await session.rollback()
|
||||
raise
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Migrate legacy WORKSPACE_DEFINITIONS into MySQL."
|
||||
)
|
||||
)
|
||||
parser.add_argument(
|
||||
"--source",
|
||||
required=True,
|
||||
type=Path,
|
||||
help="Path to the legacy server.py file.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--apply",
|
||||
action="store_true",
|
||||
help="Commit changes. Without this flag the transaction is rolled back.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
async def async_main() -> None:
|
||||
args = parse_args()
|
||||
source_path = args.source.resolve(strict=True)
|
||||
workspaces, source_sha256 = extract_workspace_definitions(source_path)
|
||||
stats = await run_migration(
|
||||
os.environ["DATABASE_URL"],
|
||||
workspaces,
|
||||
apply_changes=args.apply,
|
||||
)
|
||||
result = {
|
||||
"mode": "apply" if args.apply else "dry-run",
|
||||
"source": str(source_path),
|
||||
"source_symbol": "WORKSPACE_DEFINITIONS",
|
||||
"source_sha256": source_sha256,
|
||||
"source_counts": {
|
||||
"workspaces": len(workspaces),
|
||||
"workspace_members": sum(
|
||||
len(workspace.userIds) for workspace in workspaces
|
||||
),
|
||||
},
|
||||
"changes": stats,
|
||||
}
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
asyncio.run(async_main())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,504 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from common.db import create_database_engine, create_session_factory
|
||||
from common.db.models import (
|
||||
AuditLogs,
|
||||
Permissions,
|
||||
RolePermissions,
|
||||
Roles,
|
||||
Users,
|
||||
)
|
||||
|
||||
CROCKFORD_BASE32 = "0123456789ABCDEFGHJKMNPQRSTVWXYZ"
|
||||
LEGACY_PASSWORD_HASH = "!legacy-account-without-password!"
|
||||
|
||||
PERMISSION_NAMES = {
|
||||
"dashboard.view": "查看工作台",
|
||||
"script.build": "构建脚本",
|
||||
"script.public.manage": "管理公共脚本",
|
||||
"schedule.own": "管理本人调度",
|
||||
"schedule.all": "管理全部调度",
|
||||
"experiment.own": "管理本人实验",
|
||||
"experiment.all": "管理全部实验",
|
||||
"resource.personal": "管理个人资源",
|
||||
"resource.public.upload": "上传公共资源",
|
||||
"resource.public.manage": "管理公共资源",
|
||||
"system.view": "查看系统管理",
|
||||
"system.manage": "管理系统配置",
|
||||
"audit.view": "查看审计日志",
|
||||
}
|
||||
|
||||
ACTION_CATALOG = {
|
||||
"保存调度配置": ("schedule.save", "schedule"),
|
||||
"删除脚本对象": ("script.delete", "script"),
|
||||
"删除实验记录": ("experiment.delete", "experiment"),
|
||||
"删除数据资源": ("data_resource.delete", "data_resource"),
|
||||
"上传数据资源": ("data_resource.upload", "data_resource"),
|
||||
"新建脚本对象": ("script.create", "script"),
|
||||
"修改用户角色": ("user.role.update", "user"),
|
||||
"运行 Python": ("script.run_python", "script"),
|
||||
"运行调度": ("schedule.run", "schedule"),
|
||||
}
|
||||
|
||||
|
||||
class LegacyUser(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
id: str
|
||||
name: str
|
||||
role: str
|
||||
roleKey: str
|
||||
avatar: str | None = None
|
||||
|
||||
|
||||
class LegacyRole(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
key: str
|
||||
name: str
|
||||
description: str | None = None
|
||||
permissions: list[str]
|
||||
|
||||
|
||||
class LegacyAuditLog(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
id: str
|
||||
actorId: str
|
||||
actorName: str
|
||||
role: str
|
||||
action: str
|
||||
target: str
|
||||
detail: str
|
||||
status: str
|
||||
createdAt: str
|
||||
|
||||
|
||||
class LegacySystem(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
users: list[LegacyUser]
|
||||
roles: list[LegacyRole]
|
||||
audit_logs: list[LegacyAuditLog] = Field(alias="auditLogs")
|
||||
|
||||
|
||||
def deterministic_legacy_ulid(entity_type: str, legacy_key: str) -> str:
|
||||
"""Create a stable ULID-compatible ID with an epoch timestamp prefix."""
|
||||
digest = hashlib.sha256(
|
||||
f"model-platform-v1:{entity_type}:{legacy_key}".encode("utf-8")
|
||||
).digest()
|
||||
value = int.from_bytes(b"\x00" * 6 + digest[:10], byteorder="big")
|
||||
encoded = ["0"] * 26
|
||||
for index in range(25, -1, -1):
|
||||
encoded[index] = CROCKFORD_BASE32[value & 31]
|
||||
value >>= 5
|
||||
return "".join(encoded)
|
||||
|
||||
|
||||
def require_unique(values: list[str], label: str) -> None:
|
||||
duplicates = sorted(
|
||||
value for value in set(values) if values.count(value) > 1
|
||||
)
|
||||
if duplicates:
|
||||
raise ValueError(f"duplicate {label}: {duplicates}")
|
||||
|
||||
|
||||
def validate_source(source: LegacySystem) -> None:
|
||||
role_codes = [role.key for role in source.roles]
|
||||
user_codes = [user.id for user in source.users]
|
||||
audit_ids = [item.id for item in source.audit_logs]
|
||||
require_unique(role_codes, "role keys")
|
||||
require_unique(user_codes, "user ids")
|
||||
require_unique(audit_ids, "audit ids")
|
||||
|
||||
role_code_set = set(role_codes)
|
||||
unknown_roles = sorted(
|
||||
user.roleKey
|
||||
for user in source.users
|
||||
if user.roleKey not in role_code_set
|
||||
)
|
||||
if unknown_roles:
|
||||
raise ValueError(f"users reference unknown roles: {unknown_roles}")
|
||||
|
||||
user_code_set = set(user_codes)
|
||||
unknown_actors = sorted(
|
||||
item.actorId
|
||||
for item in source.audit_logs
|
||||
if item.actorId not in user_code_set
|
||||
)
|
||||
if unknown_actors:
|
||||
raise ValueError(f"audit logs reference unknown users: {unknown_actors}")
|
||||
|
||||
for role in source.roles:
|
||||
require_unique(role.permissions, f"permissions of role {role.key}")
|
||||
for permission_code in role.permissions:
|
||||
if "." not in permission_code:
|
||||
raise ValueError(
|
||||
f"invalid permission code {permission_code!r}"
|
||||
)
|
||||
|
||||
|
||||
def load_source(path: Path) -> tuple[LegacySystem, str]:
|
||||
raw = path.read_bytes()
|
||||
source = LegacySystem.model_validate_json(raw)
|
||||
validate_source(source)
|
||||
return source, hashlib.sha256(raw).hexdigest()
|
||||
|
||||
|
||||
def set_changed(instance: Any, values: dict[str, Any]) -> bool:
|
||||
changed = False
|
||||
for attribute, value in values.items():
|
||||
if getattr(instance, attribute) != value:
|
||||
setattr(instance, attribute, value)
|
||||
changed = True
|
||||
return changed
|
||||
|
||||
|
||||
def new_stats() -> dict[str, int]:
|
||||
return {
|
||||
"roles_inserted": 0,
|
||||
"roles_updated": 0,
|
||||
"permissions_inserted": 0,
|
||||
"permissions_updated": 0,
|
||||
"role_permissions_inserted": 0,
|
||||
"users_inserted": 0,
|
||||
"users_updated": 0,
|
||||
"audit_logs_inserted": 0,
|
||||
"audit_logs_skipped": 0,
|
||||
}
|
||||
|
||||
|
||||
async def migrate_roles(
|
||||
session: AsyncSession,
|
||||
source: LegacySystem,
|
||||
stats: dict[str, int],
|
||||
) -> dict[str, str]:
|
||||
existing = {
|
||||
item.role_code: item
|
||||
for item in (
|
||||
await session.scalars(select(Roles).order_by(Roles.role_code))
|
||||
).all()
|
||||
}
|
||||
role_ids: dict[str, str] = {}
|
||||
|
||||
for legacy in source.roles:
|
||||
values = {
|
||||
"role_name": legacy.name,
|
||||
"role_scope": "platform",
|
||||
"is_builtin": 1,
|
||||
"description": legacy.description,
|
||||
}
|
||||
role = existing.get(legacy.key)
|
||||
if role is None:
|
||||
role = Roles(
|
||||
role_id=deterministic_legacy_ulid("role", legacy.key),
|
||||
role_code=legacy.key,
|
||||
**values,
|
||||
)
|
||||
session.add(role)
|
||||
stats["roles_inserted"] += 1
|
||||
elif set_changed(role, values):
|
||||
stats["roles_updated"] += 1
|
||||
role_ids[legacy.key] = role.role_id
|
||||
|
||||
return role_ids
|
||||
|
||||
|
||||
async def migrate_permissions(
|
||||
session: AsyncSession,
|
||||
source: LegacySystem,
|
||||
stats: dict[str, int],
|
||||
) -> dict[str, str]:
|
||||
permission_codes = sorted(
|
||||
{
|
||||
permission_code
|
||||
for role in source.roles
|
||||
for permission_code in role.permissions
|
||||
}
|
||||
)
|
||||
existing = {
|
||||
item.permission_code: item
|
||||
for item in (
|
||||
await session.scalars(
|
||||
select(Permissions).order_by(Permissions.permission_code)
|
||||
)
|
||||
).all()
|
||||
}
|
||||
permission_ids: dict[str, str] = {}
|
||||
|
||||
for permission_code in permission_codes:
|
||||
values = {
|
||||
"permission_name": PERMISSION_NAMES.get(
|
||||
permission_code, permission_code
|
||||
),
|
||||
"module_code": permission_code.split(".", 1)[0],
|
||||
"description": f"由旧版 system.json 迁移：{permission_code}",
|
||||
}
|
||||
permission = existing.get(permission_code)
|
||||
if permission is None:
|
||||
permission = Permissions(
|
||||
permission_id=deterministic_legacy_ulid(
|
||||
"permission", permission_code
|
||||
),
|
||||
permission_code=permission_code,
|
||||
**values,
|
||||
)
|
||||
session.add(permission)
|
||||
stats["permissions_inserted"] += 1
|
||||
elif set_changed(permission, values):
|
||||
stats["permissions_updated"] += 1
|
||||
permission_ids[permission_code] = permission.permission_id
|
||||
|
||||
return permission_ids
|
||||
|
||||
|
||||
async def migrate_role_permissions(
|
||||
session: AsyncSession,
|
||||
source: LegacySystem,
|
||||
role_ids: dict[str, str],
|
||||
permission_ids: dict[str, str],
|
||||
stats: dict[str, int],
|
||||
) -> None:
|
||||
existing = set(
|
||||
(
|
||||
await session.execute(
|
||||
select(
|
||||
RolePermissions.role_id,
|
||||
RolePermissions.permission_id,
|
||||
)
|
||||
)
|
||||
).all()
|
||||
)
|
||||
|
||||
for role in source.roles:
|
||||
for permission_code in role.permissions:
|
||||
pair = (
|
||||
role_ids[role.key],
|
||||
permission_ids[permission_code],
|
||||
)
|
||||
if pair not in existing:
|
||||
session.add(
|
||||
RolePermissions(
|
||||
role_id=pair[0],
|
||||
permission_id=pair[1],
|
||||
)
|
||||
)
|
||||
existing.add(pair)
|
||||
stats["role_permissions_inserted"] += 1
|
||||
|
||||
|
||||
async def migrate_users(
|
||||
session: AsyncSession,
|
||||
source: LegacySystem,
|
||||
role_ids: dict[str, str],
|
||||
stats: dict[str, int],
|
||||
) -> dict[str, str]:
|
||||
existing = {
|
||||
item.username: item
|
||||
for item in (
|
||||
await session.scalars(select(Users).order_by(Users.username))
|
||||
).all()
|
||||
}
|
||||
user_ids: dict[str, str] = {}
|
||||
|
||||
for legacy in source.users:
|
||||
values = {
|
||||
"display_name": legacy.name,
|
||||
"platform_role_id": role_ids[legacy.roleKey],
|
||||
"avatar_uri": (
|
||||
f"initial://{quote(legacy.avatar)}"
|
||||
if legacy.avatar
|
||||
else None
|
||||
),
|
||||
}
|
||||
user = existing.get(legacy.id)
|
||||
if user is None:
|
||||
user = Users(
|
||||
user_id=deterministic_legacy_ulid("user", legacy.id),
|
||||
username=legacy.id,
|
||||
password_hash=LEGACY_PASSWORD_HASH,
|
||||
status="active",
|
||||
**values,
|
||||
)
|
||||
session.add(user)
|
||||
stats["users_inserted"] += 1
|
||||
elif set_changed(user, values):
|
||||
stats["users_updated"] += 1
|
||||
user_ids[legacy.id] = user.user_id
|
||||
|
||||
return user_ids
|
||||
|
||||
|
||||
def audit_catalog(action: str) -> tuple[str, str]:
|
||||
known = ACTION_CATALOG.get(action)
|
||||
if known is not None:
|
||||
return known
|
||||
digest = hashlib.sha256(action.encode("utf-8")).hexdigest()[:16]
|
||||
return (f"legacy.action.{digest}", "legacy")
|
||||
|
||||
|
||||
async def migrate_audit_logs(
|
||||
session: AsyncSession,
|
||||
source: LegacySystem,
|
||||
user_ids: dict[str, str],
|
||||
stats: dict[str, int],
|
||||
) -> None:
|
||||
existing_payloads = (
|
||||
await session.scalars(select(AuditLogs.detail_json))
|
||||
).all()
|
||||
existing_legacy_ids = {
|
||||
payload.get("legacy_id")
|
||||
for payload in existing_payloads
|
||||
if isinstance(payload, dict) and payload.get("legacy_id")
|
||||
}
|
||||
|
||||
for legacy in sorted(source.audit_logs, key=lambda item: item.createdAt):
|
||||
if legacy.id in existing_legacy_ids:
|
||||
stats["audit_logs_skipped"] += 1
|
||||
continue
|
||||
|
||||
action_code, target_type = audit_catalog(legacy.action)
|
||||
session.add(
|
||||
AuditLogs(
|
||||
actor_user_id=user_ids[legacy.actorId],
|
||||
action_code=action_code,
|
||||
target_type=target_type,
|
||||
target_id=legacy.target[:128] or None,
|
||||
operation_status=(
|
||||
"success" if legacy.status == "成功" else "failed"
|
||||
),
|
||||
created_at=datetime.strptime(
|
||||
legacy.createdAt, "%Y-%m-%d %H:%M:%S"
|
||||
),
|
||||
detail_json={
|
||||
"legacy_id": legacy.id,
|
||||
"legacy_actor_name": legacy.actorName,
|
||||
"legacy_role": legacy.role,
|
||||
"legacy_action": legacy.action,
|
||||
"legacy_target": legacy.target,
|
||||
"legacy_detail": legacy.detail,
|
||||
"legacy_status": legacy.status,
|
||||
"migration_source": "platform_data/system.json",
|
||||
},
|
||||
)
|
||||
)
|
||||
existing_legacy_ids.add(legacy.id)
|
||||
stats["audit_logs_inserted"] += 1
|
||||
|
||||
|
||||
async def run_migration(
|
||||
database_url: str,
|
||||
source: LegacySystem,
|
||||
*,
|
||||
apply_changes: bool,
|
||||
) -> dict[str, int]:
|
||||
engine = create_database_engine(database_url)
|
||||
factory = create_session_factory(engine)
|
||||
stats = new_stats()
|
||||
|
||||
try:
|
||||
async with factory() as session:
|
||||
try:
|
||||
role_ids = await migrate_roles(session, source, stats)
|
||||
permission_ids = await migrate_permissions(
|
||||
session, source, stats
|
||||
)
|
||||
await migrate_role_permissions(
|
||||
session,
|
||||
source,
|
||||
role_ids,
|
||||
permission_ids,
|
||||
stats,
|
||||
)
|
||||
user_ids = await migrate_users(
|
||||
session, source, role_ids, stats
|
||||
)
|
||||
await migrate_audit_logs(
|
||||
session, source, user_ids, stats
|
||||
)
|
||||
|
||||
if apply_changes:
|
||||
await session.commit()
|
||||
else:
|
||||
await session.rollback()
|
||||
except Exception:
|
||||
await session.rollback()
|
||||
raise
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Migrate legacy system.json records into MySQL."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--source",
|
||||
required=True,
|
||||
type=Path,
|
||||
help="Path to the legacy platform_data/system.json file.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--apply",
|
||||
action="store_true",
|
||||
help="Commit changes. Without this flag the transaction is rolled back.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
async def async_main() -> None:
|
||||
args = parse_args()
|
||||
source_path = args.source.resolve(strict=True)
|
||||
source, source_sha256 = load_source(source_path)
|
||||
database_url = os.environ["DATABASE_URL"]
|
||||
stats = await run_migration(
|
||||
database_url,
|
||||
source,
|
||||
apply_changes=args.apply,
|
||||
)
|
||||
result = {
|
||||
"mode": "apply" if args.apply else "dry-run",
|
||||
"source": str(source_path),
|
||||
"source_sha256": source_sha256,
|
||||
"source_counts": {
|
||||
"roles": len(source.roles),
|
||||
"permissions": len(
|
||||
{
|
||||
permission
|
||||
for role in source.roles
|
||||
for permission in role.permissions
|
||||
}
|
||||
),
|
||||
"role_permissions": sum(
|
||||
len(role.permissions) for role in source.roles
|
||||
),
|
||||
"users": len(source.users),
|
||||
"audit_logs": len(source.audit_logs),
|
||||
},
|
||||
"changes": stats,
|
||||
}
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
asyncio.run(async_main())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,127 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from decimal import Decimal, InvalidOperation
|
||||
from logging.config import fileConfig
|
||||
from typing import Any
|
||||
|
||||
from alembic import context
|
||||
from sqlalchemy import Connection, pool
|
||||
from sqlalchemy.ext.asyncio import async_engine_from_config
|
||||
|
||||
from common.db import Base
|
||||
|
||||
config = context.config
|
||||
|
||||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
|
||||
target_metadata = Base.metadata
|
||||
|
||||
|
||||
def canonical_default(value: Any) -> tuple[str, Any] | None:
|
||||
"""Normalize harmless MySQL quoting and numeric formatting differences."""
|
||||
if value is None:
|
||||
return None
|
||||
|
||||
text_value = str(value).strip()
|
||||
while (
|
||||
len(text_value) >= 2
|
||||
and text_value.startswith("(")
|
||||
and text_value.endswith(")")
|
||||
):
|
||||
text_value = text_value[1:-1].strip()
|
||||
if (
|
||||
len(text_value) >= 2
|
||||
and text_value[0] == text_value[-1]
|
||||
and text_value[0] in {"'", '"'}
|
||||
):
|
||||
text_value = text_value[1:-1]
|
||||
|
||||
try:
|
||||
return ("number", Decimal(text_value).normalize())
|
||||
except InvalidOperation:
|
||||
return ("text", text_value.casefold())
|
||||
|
||||
|
||||
def compare_server_default(
|
||||
migration_context: Any,
|
||||
inspected_column: Any,
|
||||
metadata_column: Any,
|
||||
inspected_default: str | None,
|
||||
metadata_default: Any,
|
||||
rendered_metadata_default: str | None,
|
||||
) -> bool | None:
|
||||
"""Suppress formatting-only differences and defer real changes to Alembic."""
|
||||
del migration_context, inspected_column, metadata_column, metadata_default
|
||||
if canonical_default(inspected_default) == canonical_default(
|
||||
rendered_metadata_default
|
||||
):
|
||||
return False
|
||||
return None
|
||||
|
||||
|
||||
def database_url() -> str:
|
||||
"""Return the runtime database URL without storing credentials in the repo."""
|
||||
try:
|
||||
return os.environ["DATABASE_URL"]
|
||||
except KeyError as exc:
|
||||
raise RuntimeError(
|
||||
"DATABASE_URL is required for Alembic commands"
|
||||
) from exc
|
||||
|
||||
|
||||
def configure_context(*, connection: Connection | None = None) -> None:
|
||||
options = {
|
||||
"target_metadata": target_metadata,
|
||||
"compare_type": True,
|
||||
"compare_server_default": compare_server_default,
|
||||
}
|
||||
if connection is None:
|
||||
context.configure(
|
||||
url=database_url(),
|
||||
literal_binds=True,
|
||||
dialect_opts={"paramstyle": "named"},
|
||||
**options,
|
||||
)
|
||||
else:
|
||||
context.configure(connection=connection, **options)
|
||||
|
||||
|
||||
def run_migrations_offline() -> None:
|
||||
configure_context()
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
def do_run_migrations(connection: Connection) -> None:
|
||||
configure_context(connection=connection)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
async def run_async_migrations() -> None:
|
||||
section = config.get_section(config.config_ini_section, {})
|
||||
section["sqlalchemy.url"] = database_url()
|
||||
connectable = async_engine_from_config(
|
||||
section,
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
|
||||
try:
|
||||
async with connectable.connect() as connection:
|
||||
await connection.run_sync(do_run_migrations)
|
||||
finally:
|
||||
await connectable.dispose()
|
||||
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
asyncio.run(run_async_migrations())
|
||||
|
||||
|
||||
if context.is_offline_mode():
|
||||
run_migrations_offline()
|
||||
else:
|
||||
run_migrations_online()
|
||||
@@ -0,0 +1,28 @@
|
||||
"""${message}
|
||||
|
||||
Revision ID: ${up_revision}
|
||||
Revises: ${down_revision | comma,n}
|
||||
Create Date: ${create_date}
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = ${repr(up_revision)}
|
||||
down_revision: str | Sequence[str] | None = ${repr(down_revision)}
|
||||
branch_labels: str | Sequence[str] | None = ${repr(branch_labels)}
|
||||
depends_on: str | Sequence[str] | None = ${repr(depends_on)}
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Upgrade schema."""
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""Downgrade schema."""
|
||||
${downgrades if downgrades else "pass"}
|
||||
@@ -0,0 +1,652 @@
|
||||
"""v1 schema baseline
|
||||
|
||||
Revision ID: 20260724_0001
|
||||
Revises:
|
||||
Create Date: 2026-07-24 05:57:13.620909
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import mysql
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = '20260724_0001'
|
||||
down_revision: str | Sequence[str] | None = None
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Upgrade schema."""
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.create_table('consumer_inbox',
|
||||
sa.Column('consumer_name', sa.String(length=128), nullable=False),
|
||||
sa.Column('event_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('process_status', sa.String(length=16), server_default=sa.text("'processing'"), nullable=False, comment='processing/succeeded/failed'),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('message_id', sa.String(length=128), nullable=True, comment='数据库事件处理批次标识'),
|
||||
sa.Column('processed_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('error_message', sa.String(length=2000), nullable=True),
|
||||
sa.PrimaryKeyConstraint('consumer_name', 'event_id'),
|
||||
comment='消费者幂等 Inbox，防止数据库事件重复处理'
|
||||
)
|
||||
op.create_index('idx_consumer_inbox_status', 'consumer_inbox', ['consumer_name', 'process_status', 'created_at'], unique=False)
|
||||
op.create_table('outbox_events',
|
||||
sa.Column('event_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('aggregate_type', sa.String(length=64), nullable=False),
|
||||
sa.Column('aggregate_id', sa.String(length=128), nullable=False),
|
||||
sa.Column('event_type', sa.String(length=128), nullable=False),
|
||||
sa.Column('schema_version', mysql.SMALLINT(), server_default=sa.text('1'), nullable=False),
|
||||
sa.Column('payload_json', sa.JSON(), nullable=False),
|
||||
sa.Column('event_status', sa.String(length=16), server_default=sa.text("'pending'"), nullable=False, comment='pending/published/failed'),
|
||||
sa.Column('available_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('retry_count', mysql.INTEGER(), server_default=sa.text('0'), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('trace_id', sa.String(length=64), nullable=True),
|
||||
sa.Column('idempotency_key', sa.String(length=128), nullable=True),
|
||||
sa.Column('published_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('last_error', sa.String(length=2000), nullable=True),
|
||||
sa.PrimaryKeyConstraint('event_id'),
|
||||
comment='事务 Outbox；由 Schedule Executor 直接轮询处理'
|
||||
)
|
||||
op.create_index('idx_outbox_aggregate', 'outbox_events', ['aggregate_type', 'aggregate_id', 'created_at'], unique=False)
|
||||
op.create_index('idx_outbox_idempotency', 'outbox_events', ['idempotency_key'], unique=False)
|
||||
op.create_index('idx_outbox_pending', 'outbox_events', ['event_status', 'available_at', 'created_at'], unique=False)
|
||||
op.create_table('permissions',
|
||||
sa.Column('permission_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('permission_code', sa.String(length=128), nullable=False),
|
||||
sa.Column('permission_name', sa.String(length=100), nullable=False),
|
||||
sa.Column('module_code', sa.String(length=64), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('description', sa.String(length=500), nullable=True),
|
||||
sa.PrimaryKeyConstraint('permission_id'),
|
||||
comment='权限点'
|
||||
)
|
||||
op.create_index('idx_permissions_module', 'permissions', ['module_code'], unique=False)
|
||||
op.create_index('uk_permissions_code', 'permissions', ['permission_code'], unique=True)
|
||||
op.create_table('roles',
|
||||
sa.Column('role_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('role_code', sa.String(length=64), nullable=False),
|
||||
sa.Column('role_name', sa.String(length=100), nullable=False),
|
||||
sa.Column('role_scope', sa.String(length=16), nullable=False, comment='platform/workspace'),
|
||||
sa.Column('is_builtin', mysql.TINYINT(display_width=1), server_default=sa.text('0'), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('updated_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('description', sa.String(length=500), nullable=True),
|
||||
sa.PrimaryKeyConstraint('role_id'),
|
||||
comment='角色'
|
||||
)
|
||||
op.create_index('uk_roles_code', 'roles', ['role_code'], unique=True)
|
||||
op.create_table('role_permissions',
|
||||
sa.Column('role_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('permission_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.ForeignKeyConstraint(['permission_id'], ['permissions.permission_id'], name='fk_role_permissions_permission', ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['role_id'], ['roles.role_id'], name='fk_role_permissions_role', ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('role_id', 'permission_id'),
|
||||
comment='角色权限'
|
||||
)
|
||||
op.create_index('fk_role_permissions_permission', 'role_permissions', ['permission_id'], unique=False)
|
||||
op.create_table('users',
|
||||
sa.Column('user_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('username', sa.String(length=64), nullable=False),
|
||||
sa.Column('display_name', sa.String(length=100), nullable=False),
|
||||
sa.Column('password_hash', sa.String(length=255), nullable=False),
|
||||
sa.Column('status', sa.String(length=16), server_default=sa.text("'active'"), nullable=False, comment='active/disabled/locked'),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('updated_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('email', sa.String(length=255), nullable=True),
|
||||
sa.Column('platform_role_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('avatar_uri', sa.String(length=1000), nullable=True),
|
||||
sa.Column('last_login_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('deleted_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.ForeignKeyConstraint(['platform_role_id'], ['roles.role_id'], name='fk_users_platform_role', ondelete='SET NULL'),
|
||||
sa.PrimaryKeyConstraint('user_id'),
|
||||
comment='平台用户'
|
||||
)
|
||||
op.create_index('fk_users_platform_role', 'users', ['platform_role_id'], unique=False)
|
||||
op.create_index('idx_users_status', 'users', ['status'], unique=False)
|
||||
op.create_index('uk_users_email', 'users', ['email'], unique=True)
|
||||
op.create_index('uk_users_username', 'users', ['username'], unique=True)
|
||||
op.create_table('workspaces',
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workspace_code', sa.String(length=64), nullable=False),
|
||||
sa.Column('workspace_name', sa.String(length=150), nullable=False),
|
||||
sa.Column('active_root_uri', sa.String(length=1500), nullable=False, comment='活动工作区，建议 NFS/PVC/file URI'),
|
||||
sa.Column('quota_bytes', mysql.BIGINT(), server_default=sa.text('0'), nullable=False, comment='0 表示不限额'),
|
||||
sa.Column('used_bytes', mysql.BIGINT(), server_default=sa.text('0'), nullable=False),
|
||||
sa.Column('status', sa.String(length=24), server_default=sa.text("'active'"), nullable=False, comment='creating/active/suspended/deleting/deleted'),
|
||||
sa.Column('created_by', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('updated_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('description', sa.String(length=1000), nullable=True),
|
||||
sa.Column('artifact_bucket', sa.String(length=128), nullable=True, comment='RustFS bucket'),
|
||||
sa.Column('artifact_prefix', sa.String(length=512), nullable=True, comment='RustFS object key prefix'),
|
||||
sa.Column('deleted_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.ForeignKeyConstraint(['created_by'], ['users.user_id'], name='fk_workspaces_created_by', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('workspace_id'),
|
||||
comment='Workspace'
|
||||
)
|
||||
op.create_index('fk_workspaces_created_by', 'workspaces', ['created_by'], unique=False)
|
||||
op.create_index('idx_workspaces_status', 'workspaces', ['status'], unique=False)
|
||||
op.create_index('uk_workspaces_code', 'workspaces', ['workspace_code'], unique=True)
|
||||
op.create_table('audit_logs',
|
||||
sa.Column('audit_id', mysql.BIGINT(), nullable=False),
|
||||
sa.Column('action_code', sa.String(length=128), nullable=False),
|
||||
sa.Column('target_type', sa.String(length=64), nullable=False),
|
||||
sa.Column('operation_status', sa.String(length=16), server_default=sa.text("'success'"), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('actor_user_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('target_id', sa.String(length=128), nullable=True),
|
||||
sa.Column('client_ip', sa.String(length=45), nullable=True),
|
||||
sa.Column('user_agent', sa.String(length=1000), nullable=True),
|
||||
sa.Column('detail_json', sa.JSON(), nullable=True),
|
||||
sa.ForeignKeyConstraint(['actor_user_id'], ['users.user_id'], name='fk_audit_actor', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_audit_workspace', ondelete='SET NULL'),
|
||||
sa.PrimaryKeyConstraint('audit_id'),
|
||||
comment='操作审计日志'
|
||||
)
|
||||
op.create_index('idx_audit_action_time', 'audit_logs', ['action_code', 'created_at'], unique=False)
|
||||
op.create_index('idx_audit_actor_time', 'audit_logs', ['actor_user_id', 'created_at'], unique=False)
|
||||
op.create_index('idx_audit_workspace_time', 'audit_logs', ['workspace_id', 'created_at'], unique=False)
|
||||
op.create_table('runtime_instances',
|
||||
sa.Column('runtime_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('runtime_type', sa.String(length=24), server_default=sa.text("'jupyter'"), nullable=False),
|
||||
sa.Column('runtime_provider', sa.String(length=24), nullable=False, comment='process/docker/kubernetes'),
|
||||
sa.Column('proxy_base_path', sa.String(length=512), nullable=False),
|
||||
sa.Column('desired_state', sa.String(length=16), server_default=sa.text("'running'"), nullable=False),
|
||||
sa.Column('actual_state', sa.String(length=24), server_default=sa.text("'provisioning'"), nullable=False, comment='provisioning/starting/running/unhealthy/stopping/stopped/failed'),
|
||||
sa.Column('state_version', mysql.INTEGER(), server_default=sa.text('0'), nullable=False, comment='乐观锁版本'),
|
||||
sa.Column('started_by', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('updated_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('owner_user_id', sa.CHAR(length=26), nullable=True, comment='为空表示 Workspace 级 Runtime'),
|
||||
sa.Column('runtime_ref', sa.String(length=255), nullable=True, comment='PID/container ID/pod UID'),
|
||||
sa.Column('host_node', sa.String(length=255), nullable=True),
|
||||
sa.Column('internal_url', sa.String(length=1000), nullable=True),
|
||||
sa.Column('started_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('last_heartbeat_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('lease_expires_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('stopped_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('error_message', sa.Text(), nullable=True),
|
||||
sa.ForeignKeyConstraint(['owner_user_id'], ['users.user_id'], name='fk_runtime_owner', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['started_by'], ['users.user_id'], name='fk_runtime_started_by', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_runtime_workspace', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('runtime_id'),
|
||||
comment='Jupyter/未来 VS Code、OpenCode Runtime 实例'
|
||||
)
|
||||
op.create_index('fk_runtime_started_by', 'runtime_instances', ['started_by'], unique=False)
|
||||
op.create_index('idx_runtime_lease', 'runtime_instances', ['actual_state', 'lease_expires_at'], unique=False)
|
||||
op.create_index('idx_runtime_owner_state', 'runtime_instances', ['owner_user_id', 'actual_state'], unique=False)
|
||||
op.create_index('idx_runtime_workspace_state', 'runtime_instances', ['workspace_id', 'runtime_type', 'actual_state'], unique=False)
|
||||
op.create_table('schedules',
|
||||
sa.Column('schedule_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('schedule_name', sa.String(length=255), nullable=False),
|
||||
sa.Column('trigger_type', sa.String(length=16), server_default=sa.text("'cron'"), nullable=False, comment='manual/cron/api'),
|
||||
sa.Column('timezone', sa.String(length=64), server_default=sa.text("'Asia/Shanghai'"), nullable=False),
|
||||
sa.Column('enabled', mysql.TINYINT(display_width=1), server_default=sa.text('0'), nullable=False),
|
||||
sa.Column('workflow_version', mysql.INTEGER(), server_default=sa.text('1'), nullable=False),
|
||||
sa.Column('max_concurrency', mysql.INTEGER(), server_default=sa.text('1'), nullable=False),
|
||||
sa.Column('failure_policy', sa.String(length=24), server_default=sa.text("'stop'"), nullable=False, comment='stop/continue'),
|
||||
sa.Column('created_by', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('updated_by', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('updated_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('description', sa.String(length=1000), nullable=True),
|
||||
sa.Column('cron_expression', sa.String(length=128), nullable=True),
|
||||
sa.Column('last_run_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('next_run_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('deleted_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.ForeignKeyConstraint(['created_by'], ['users.user_id'], name='fk_schedules_created_by', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['updated_by'], ['users.user_id'], name='fk_schedules_updated_by', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_schedules_workspace', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('schedule_id'),
|
||||
comment='调度方案'
|
||||
)
|
||||
op.create_index('fk_schedules_created_by', 'schedules', ['created_by'], unique=False)
|
||||
op.create_index('fk_schedules_updated_by', 'schedules', ['updated_by'], unique=False)
|
||||
op.create_index('idx_schedules_due', 'schedules', ['enabled', 'next_run_at'], unique=False)
|
||||
op.create_index('idx_schedules_workspace', 'schedules', ['workspace_id', 'enabled', 'updated_at'], unique=False)
|
||||
op.create_table('storage_objects',
|
||||
sa.Column('storage_object_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('object_type', sa.String(length=16), nullable=False, comment='file/directory'),
|
||||
sa.Column('usage_type', sa.String(length=32), nullable=False, comment='working_copy/public_script/data_resource/version_artifact/snapshot/run_log/run_result'),
|
||||
sa.Column('storage_backend', sa.String(length=16), nullable=False, comment='workspace_fs/rustfs'),
|
||||
sa.Column('storage_uri', sa.String(length=1500), nullable=False),
|
||||
sa.Column('file_name', sa.String(length=255), nullable=False),
|
||||
sa.Column('size_bytes', mysql.BIGINT(), server_default=sa.text('0'), nullable=False),
|
||||
sa.Column('visibility', sa.String(length=16), server_default=sa.text("'private'"), nullable=False, comment='private/workspace/public'),
|
||||
sa.Column('is_immutable', mysql.TINYINT(display_width=1), server_default=sa.text('0'), nullable=False),
|
||||
sa.Column('object_status', sa.String(length=24), server_default=sa.text("'available'"), nullable=False, comment='uploading/available/deleting/deleted/failed'),
|
||||
sa.Column('created_by', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('updated_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('owner_user_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('parent_object_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('relative_path', sa.String(length=1024), nullable=True, comment='Workspace 相对路径'),
|
||||
sa.Column('path_hash', sa.BINARY(length=32), nullable=True, comment='SHA-256(relative_path)，由应用写入'),
|
||||
sa.Column('bucket_name', sa.String(length=128), nullable=True),
|
||||
sa.Column('object_key', sa.String(length=1024), nullable=True),
|
||||
sa.Column('object_key_hash', sa.BINARY(length=32), nullable=True, comment='SHA-256(object_key)，由应用写入'),
|
||||
sa.Column('file_extension', sa.String(length=32), nullable=True),
|
||||
sa.Column('mime_type', sa.String(length=255), nullable=True),
|
||||
sa.Column('content_hash', sa.CHAR(length=64), nullable=True, comment='SHA-256 hex'),
|
||||
sa.Column('object_etag', sa.String(length=255), nullable=True),
|
||||
sa.Column('deleted_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.ForeignKeyConstraint(['created_by'], ['users.user_id'], name='fk_storage_created_by', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['owner_user_id'], ['users.user_id'], name='fk_storage_owner', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['parent_object_id'], ['storage_objects.storage_object_id'], name='fk_storage_parent', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_storage_workspace', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('storage_object_id'),
|
||||
comment='Workspace 文件和 RustFS 对象的统一元数据'
|
||||
)
|
||||
op.create_index('fk_storage_created_by', 'storage_objects', ['created_by'], unique=False)
|
||||
op.create_index('idx_storage_content_hash', 'storage_objects', ['content_hash'], unique=False)
|
||||
op.create_index('idx_storage_owner', 'storage_objects', ['owner_user_id', 'object_status'], unique=False)
|
||||
op.create_index('idx_storage_parent', 'storage_objects', ['parent_object_id'], unique=False)
|
||||
op.create_index('idx_storage_workspace_usage', 'storage_objects', ['workspace_id', 'usage_type', 'object_status'], unique=False)
|
||||
op.create_index('uk_storage_bucket_key', 'storage_objects', ['storage_backend', 'bucket_name', 'object_key_hash'], unique=True)
|
||||
op.create_index('uk_storage_workspace_path', 'storage_objects', ['workspace_id', 'storage_backend', 'path_hash'], unique=True)
|
||||
op.create_table('workspace_members',
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('user_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('role_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('member_status', sa.String(length=16), server_default=sa.text("'active'"), nullable=False),
|
||||
sa.Column('joined_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('updated_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.ForeignKeyConstraint(['role_id'], ['roles.role_id'], name='fk_workspace_members_role', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['user_id'], ['users.user_id'], name='fk_workspace_members_user', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_workspace_members_workspace', ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('workspace_id', 'user_id'),
|
||||
comment='Workspace 成员与角色'
|
||||
)
|
||||
op.create_index('idx_workspace_members_role', 'workspace_members', ['role_id'], unique=False)
|
||||
op.create_index('idx_workspace_members_user', 'workspace_members', ['user_id', 'member_status'], unique=False)
|
||||
op.create_table('data_resources',
|
||||
sa.Column('resource_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('storage_object_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('owner_user_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('resource_name', sa.String(length=255), nullable=False),
|
||||
sa.Column('visibility', sa.String(length=16), server_default=sa.text("'private'"), nullable=False),
|
||||
sa.Column('status', sa.String(length=16), server_default=sa.text("'active'"), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('updated_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('description', sa.String(length=1000), nullable=True),
|
||||
sa.Column('schema_json', sa.JSON(), nullable=True, comment='字段结构、行数等可选元数据'),
|
||||
sa.Column('deleted_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.ForeignKeyConstraint(['owner_user_id'], ['users.user_id'], name='fk_data_resources_owner', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['storage_object_id'], ['storage_objects.storage_object_id'], name='fk_data_resources_object', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_data_resources_workspace', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('resource_id'),
|
||||
comment='数据资源'
|
||||
)
|
||||
op.create_index('idx_data_resources_owner', 'data_resources', ['owner_user_id', 'status'], unique=False)
|
||||
op.create_index('idx_data_resources_workspace', 'data_resources', ['workspace_id', 'visibility', 'status'], unique=False)
|
||||
op.create_index('uk_data_resources_object', 'data_resources', ['storage_object_id'], unique=True)
|
||||
op.create_table('edit_sessions',
|
||||
sa.Column('edit_session_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('storage_object_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('user_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('lock_key', sa.String(length=512), nullable=False),
|
||||
sa.Column('lock_token_hash', sa.BINARY(length=32), nullable=False, comment='不保存原始 token'),
|
||||
sa.Column('session_status', sa.String(length=16), server_default=sa.text("'active'"), nullable=False, comment='active/closed/expired/failed'),
|
||||
sa.Column('started_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('last_heartbeat_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('expires_at', mysql.DATETIME(fsp=3), nullable=False),
|
||||
sa.Column('runtime_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('jupyter_session_id', sa.String(length=255), nullable=True),
|
||||
sa.Column('ended_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('end_reason', sa.String(length=64), nullable=True),
|
||||
sa.ForeignKeyConstraint(['runtime_id'], ['runtime_instances.runtime_id'], name='fk_edit_sessions_runtime', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['storage_object_id'], ['storage_objects.storage_object_id'], name='fk_edit_sessions_object', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['user_id'], ['users.user_id'], name='fk_edit_sessions_user', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_edit_sessions_workspace', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('edit_session_id'),
|
||||
comment='编辑会话与数据库租约；MySQL 为锁状态权威'
|
||||
)
|
||||
op.create_index('fk_edit_sessions_workspace', 'edit_sessions', ['workspace_id'], unique=False)
|
||||
op.create_index('idx_edit_sessions_object', 'edit_sessions', ['storage_object_id', 'session_status', 'expires_at'], unique=False)
|
||||
op.create_index('idx_edit_sessions_runtime', 'edit_sessions', ['runtime_id', 'session_status'], unique=False)
|
||||
op.create_index('idx_edit_sessions_user', 'edit_sessions', ['user_id', 'session_status'], unique=False)
|
||||
op.create_table('schedule_runs',
|
||||
sa.Column('run_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('schedule_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workflow_version', mysql.INTEGER(), nullable=False),
|
||||
sa.Column('trigger_type', sa.String(length=16), nullable=False, comment='manual/cron/api/retry'),
|
||||
sa.Column('idempotency_key', sa.String(length=128), nullable=False),
|
||||
sa.Column('run_status', sa.String(length=24), server_default=sa.text("'queued'"), nullable=False, comment='queued/running/succeeded/failed/cancelled/timed_out'),
|
||||
sa.Column('state_version', mysql.INTEGER(), server_default=sa.text('0'), nullable=False, comment='乐观锁版本'),
|
||||
sa.Column('schedule_snapshot', sa.JSON(), nullable=False, comment='执行时 DAG 快照'),
|
||||
sa.Column('queued_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('triggered_by', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('started_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('finished_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('duration_ms', mysql.BIGINT(), nullable=True),
|
||||
sa.Column('error_code', sa.String(length=64), nullable=True),
|
||||
sa.Column('error_message', sa.Text(), nullable=True),
|
||||
sa.Column('logs_object_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('result_object_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.ForeignKeyConstraint(['logs_object_id'], ['storage_objects.storage_object_id'], name='fk_schedule_runs_logs', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['result_object_id'], ['storage_objects.storage_object_id'], name='fk_schedule_runs_result', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['schedule_id'], ['schedules.schedule_id'], name='fk_schedule_runs_schedule', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['triggered_by'], ['users.user_id'], name='fk_schedule_runs_user', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_schedule_runs_workspace', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('run_id'),
|
||||
comment='调度运行'
|
||||
)
|
||||
op.create_index('fk_schedule_runs_logs', 'schedule_runs', ['logs_object_id'], unique=False)
|
||||
op.create_index('fk_schedule_runs_result', 'schedule_runs', ['result_object_id'], unique=False)
|
||||
op.create_index('fk_schedule_runs_user', 'schedule_runs', ['triggered_by'], unique=False)
|
||||
op.create_index('idx_schedule_runs_schedule', 'schedule_runs', ['schedule_id', 'created_at'], unique=False)
|
||||
op.create_index('idx_schedule_runs_status', 'schedule_runs', ['run_status', 'queued_at'], unique=False)
|
||||
op.create_index('idx_schedule_runs_workspace_status', 'schedule_runs', ['workspace_id', 'run_status', 'queued_at'], unique=False)
|
||||
op.create_index('uk_schedule_runs_idempotency', 'schedule_runs', ['idempotency_key'], unique=True)
|
||||
op.create_table('scripts',
|
||||
sa.Column('script_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('current_object_id', sa.CHAR(length=26), nullable=False, comment='当前工作副本'),
|
||||
sa.Column('owner_user_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('script_name', sa.String(length=255), nullable=False),
|
||||
sa.Column('script_type', sa.String(length=16), nullable=False, comment='python/notebook'),
|
||||
sa.Column('visibility', sa.String(length=16), server_default=sa.text("'private'"), nullable=False),
|
||||
sa.Column('status', sa.String(length=16), server_default=sa.text("'active'"), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('updated_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('deleted_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.ForeignKeyConstraint(['current_object_id'], ['storage_objects.storage_object_id'], name='fk_scripts_current_object', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['owner_user_id'], ['users.user_id'], name='fk_scripts_owner', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_scripts_workspace', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('script_id'),
|
||||
comment='可执行 Python/Notebook 脚本'
|
||||
)
|
||||
op.create_index('idx_scripts_owner', 'scripts', ['owner_user_id', 'status'], unique=False)
|
||||
op.create_index('idx_scripts_workspace', 'scripts', ['workspace_id', 'script_type', 'visibility', 'status'], unique=False)
|
||||
op.create_index('uk_scripts_current_object', 'scripts', ['current_object_id'], unique=True)
|
||||
op.create_table('upload_sessions',
|
||||
sa.Column('upload_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('user_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('idempotency_key', sa.String(length=128), nullable=False),
|
||||
sa.Column('bucket_name', sa.String(length=128), nullable=False),
|
||||
sa.Column('object_key', sa.String(length=1024), nullable=False),
|
||||
sa.Column('object_key_hash', sa.BINARY(length=32), nullable=False),
|
||||
sa.Column('upload_status', sa.String(length=24), server_default=sa.text("'created'"), nullable=False, comment='created/uploading/completed/expired/aborted/failed'),
|
||||
sa.Column('expires_at', mysql.DATETIME(fsp=3), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('updated_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('multipart_upload_id', sa.String(length=255), nullable=True),
|
||||
sa.Column('expected_size_bytes', mysql.BIGINT(), nullable=True),
|
||||
sa.Column('expected_hash', sa.CHAR(length=64), nullable=True),
|
||||
sa.Column('content_type', sa.String(length=255), nullable=True),
|
||||
sa.Column('storage_object_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('completed_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.ForeignKeyConstraint(['storage_object_id'], ['storage_objects.storage_object_id'], name='fk_upload_sessions_storage_object', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['user_id'], ['users.user_id'], name='fk_upload_sessions_user', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_upload_sessions_workspace', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('upload_id'),
|
||||
comment='RustFS 预签名上传会话；URL 本身不持久化'
|
||||
)
|
||||
op.create_index('fk_upload_sessions_storage_object', 'upload_sessions', ['storage_object_id'], unique=False)
|
||||
op.create_index('fk_upload_sessions_user', 'upload_sessions', ['user_id'], unique=False)
|
||||
op.create_index('idx_upload_sessions_expiry', 'upload_sessions', ['upload_status', 'expires_at'], unique=False)
|
||||
op.create_index('idx_upload_sessions_object_key', 'upload_sessions', ['bucket_name', 'object_key_hash'], unique=False)
|
||||
op.create_index('idx_upload_sessions_workspace', 'upload_sessions', ['workspace_id', 'user_id', 'created_at'], unique=False)
|
||||
op.create_index('uk_upload_sessions_idempotency', 'upload_sessions', ['idempotency_key'], unique=True)
|
||||
op.create_table('workspace_operations',
|
||||
sa.Column('operation_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('operation_type', sa.String(length=24), nullable=False, comment='open/close/mount/unmount/start/stop/restart/recycle'),
|
||||
sa.Column('operation_status', sa.String(length=24), server_default=sa.text("'pending'"), nullable=False, comment='pending/running/succeeded/failed/cancelled'),
|
||||
sa.Column('state_version', mysql.INTEGER(), server_default=sa.text('0'), nullable=False, comment='乐观锁版本'),
|
||||
sa.Column('requested_by', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('runtime_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('request_id', sa.String(length=128), nullable=True),
|
||||
sa.Column('started_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('finished_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('error_code', sa.String(length=64), nullable=True),
|
||||
sa.Column('error_message', sa.Text(), nullable=True),
|
||||
sa.ForeignKeyConstraint(['requested_by'], ['users.user_id'], name='fk_workspace_operations_user', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['runtime_id'], ['runtime_instances.runtime_id'], name='fk_workspace_operations_runtime', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_workspace_operations_workspace', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('operation_id'),
|
||||
comment='无状态 Backend 的 Workspace/Jupyter 异步操作记录'
|
||||
)
|
||||
op.create_index('fk_workspace_operations_user', 'workspace_operations', ['requested_by'], unique=False)
|
||||
op.create_index('idx_workspace_operations_runtime', 'workspace_operations', ['runtime_id', 'created_at'], unique=False)
|
||||
op.create_index('idx_workspace_operations_workspace', 'workspace_operations', ['workspace_id', 'operation_status', 'created_at'], unique=False)
|
||||
op.create_index('uk_workspace_operations_request', 'workspace_operations', ['request_id'], unique=True)
|
||||
op.create_table('notebook_snapshots',
|
||||
sa.Column('snapshot_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('script_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('source_object_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('artifact_object_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('snapshot_name', sa.String(length=255), nullable=False),
|
||||
sa.Column('content_hash', sa.CHAR(length=64), nullable=False),
|
||||
sa.Column('outputs_stripped', mysql.TINYINT(display_width=1), server_default=sa.text('1'), nullable=False),
|
||||
sa.Column('created_by', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('description', sa.String(length=1000), nullable=True),
|
||||
sa.ForeignKeyConstraint(['artifact_object_id'], ['storage_objects.storage_object_id'], name='fk_snapshots_artifact_object', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['created_by'], ['users.user_id'], name='fk_snapshots_created_by', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['script_id'], ['scripts.script_id'], name='fk_snapshots_script', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['source_object_id'], ['storage_objects.storage_object_id'], name='fk_snapshots_source_object', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_snapshots_workspace', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('snapshot_id'),
|
||||
comment='Notebook 开发快照，append-only'
|
||||
)
|
||||
op.create_index('fk_snapshots_created_by', 'notebook_snapshots', ['created_by'], unique=False)
|
||||
op.create_index('fk_snapshots_source_object', 'notebook_snapshots', ['source_object_id'], unique=False)
|
||||
op.create_index('idx_snapshots_workspace_created', 'notebook_snapshots', ['workspace_id', 'created_at'], unique=False)
|
||||
op.create_index('uk_snapshots_artifact', 'notebook_snapshots', ['artifact_object_id'], unique=True)
|
||||
op.create_index('uk_snapshots_script_hash', 'notebook_snapshots', ['script_id', 'content_hash'], unique=True)
|
||||
op.create_table('versions',
|
||||
sa.Column('versions_id', sa.CHAR(length=26), nullable=False, comment='稳定版本唯一 ID'),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('script_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('source_object_id', sa.CHAR(length=26), nullable=False, comment='发布时的源对象'),
|
||||
sa.Column('artifact_object_id', sa.CHAR(length=26), nullable=False, comment='RustFS 不可变版本制品'),
|
||||
sa.Column('version_no', mysql.INTEGER(), nullable=False),
|
||||
sa.Column('version_label', sa.String(length=32), nullable=False, comment='例如 v1.0'),
|
||||
sa.Column('source_path', sa.String(length=1024), nullable=False, comment='发布时路径快照'),
|
||||
sa.Column('artifact_path', sa.String(length=1500), nullable=False),
|
||||
sa.Column('content_hash', sa.CHAR(length=64), nullable=False),
|
||||
sa.Column('file_size_bytes', mysql.BIGINT(), server_default=sa.text('0'), nullable=False),
|
||||
sa.Column('visibility', sa.String(length=16), server_default=sa.text("'private'"), nullable=False),
|
||||
sa.Column('created_by', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('release_note', sa.String(length=1000), nullable=True),
|
||||
sa.ForeignKeyConstraint(['artifact_object_id'], ['storage_objects.storage_object_id'], name='fk_versions_artifact_object', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['created_by'], ['users.user_id'], name='fk_versions_created_by', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['script_id'], ['scripts.script_id'], name='fk_versions_script', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['source_object_id'], ['storage_objects.storage_object_id'], name='fk_versions_source_object', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_versions_workspace', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('versions_id'),
|
||||
comment='不可变稳定版本；调度节点必须引用 versions_id'
|
||||
)
|
||||
op.create_index('fk_versions_source_object', 'versions', ['source_object_id'], unique=False)
|
||||
op.create_index('idx_versions_creator', 'versions', ['created_by', 'created_at'], unique=False)
|
||||
op.create_index('idx_versions_workspace_created', 'versions', ['workspace_id', 'created_at'], unique=False)
|
||||
op.create_index('uk_versions_artifact', 'versions', ['artifact_object_id'], unique=True)
|
||||
op.create_index('uk_versions_script_hash', 'versions', ['script_id', 'content_hash'], unique=True)
|
||||
op.create_index('uk_versions_script_no', 'versions', ['script_id', 'version_no'], unique=True)
|
||||
op.create_table('experiments',
|
||||
sa.Column('experiment_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('workspace_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('owner_user_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('experiment_name', sa.String(length=255), nullable=False),
|
||||
sa.Column('source_type', sa.String(length=24), nullable=False, comment='python/notebook/schedule/rerun'),
|
||||
sa.Column('experiment_status', sa.String(length=24), server_default=sa.text("'queued'"), nullable=False),
|
||||
sa.Column('state_version', mysql.INTEGER(), server_default=sa.text('0'), nullable=False, comment='乐观锁版本'),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('script_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('versions_id', sa.CHAR(length=26), nullable=True, comment='工作副本运行时可为空'),
|
||||
sa.Column('schedule_run_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('parent_experiment_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('parameters_json', sa.JSON(), nullable=True),
|
||||
sa.Column('environment_json', sa.JSON(), nullable=True),
|
||||
sa.Column('result_summary', sa.String(length=2000), nullable=True),
|
||||
sa.Column('logs_object_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('result_object_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('started_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('finished_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('duration_ms', mysql.BIGINT(), nullable=True),
|
||||
sa.Column('deleted_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.ForeignKeyConstraint(['logs_object_id'], ['storage_objects.storage_object_id'], name='fk_experiments_logs', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['owner_user_id'], ['users.user_id'], name='fk_experiments_owner', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['parent_experiment_id'], ['experiments.experiment_id'], name='fk_experiments_parent', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['result_object_id'], ['storage_objects.storage_object_id'], name='fk_experiments_result', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['schedule_run_id'], ['schedule_runs.run_id'], name='fk_experiments_schedule_run', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['script_id'], ['scripts.script_id'], name='fk_experiments_script', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['versions_id'], ['versions.versions_id'], name='fk_experiments_version', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['workspace_id'], ['workspaces.workspace_id'], name='fk_experiments_workspace', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('experiment_id'),
|
||||
comment='实验记录'
|
||||
)
|
||||
op.create_index('fk_experiments_logs', 'experiments', ['logs_object_id'], unique=False)
|
||||
op.create_index('fk_experiments_parent', 'experiments', ['parent_experiment_id'], unique=False)
|
||||
op.create_index('fk_experiments_result', 'experiments', ['result_object_id'], unique=False)
|
||||
op.create_index('fk_experiments_script', 'experiments', ['script_id'], unique=False)
|
||||
op.create_index('idx_experiments_owner', 'experiments', ['owner_user_id', 'created_at'], unique=False)
|
||||
op.create_index('idx_experiments_schedule_run', 'experiments', ['schedule_run_id'], unique=False)
|
||||
op.create_index('idx_experiments_version', 'experiments', ['versions_id'], unique=False)
|
||||
op.create_index('idx_experiments_workspace', 'experiments', ['workspace_id', 'experiment_status', 'created_at'], unique=False)
|
||||
op.create_table('schedule_nodes',
|
||||
sa.Column('node_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('schedule_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('node_key', sa.String(length=64), nullable=False, comment='画布内稳定标识'),
|
||||
sa.Column('node_name', sa.String(length=255), nullable=False),
|
||||
sa.Column('versions_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('timeout_seconds', mysql.INTEGER(), server_default=sa.text('600'), nullable=False),
|
||||
sa.Column('retry_count', mysql.INTEGER(), server_default=sa.text('0'), nullable=False),
|
||||
sa.Column('retry_interval_sec', mysql.INTEGER(), server_default=sa.text('5'), nullable=False),
|
||||
sa.Column('position_x', sa.DECIMAL(precision=10, scale=2), server_default=sa.text('0.00'), nullable=False),
|
||||
sa.Column('position_y', sa.DECIMAL(precision=10, scale=2), server_default=sa.text('0.00'), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('updated_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('arguments_json', sa.JSON(), nullable=True),
|
||||
sa.Column('env_refs_json', sa.JSON(), nullable=True, comment='只存密钥引用，不存明文密钥'),
|
||||
sa.ForeignKeyConstraint(['schedule_id'], ['schedules.schedule_id'], name='fk_schedule_nodes_schedule', ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['versions_id'], ['versions.versions_id'], name='fk_schedule_nodes_version', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('node_id'),
|
||||
comment='DAG 节点，必须引用稳定版本'
|
||||
)
|
||||
op.create_index('idx_schedule_nodes_version', 'schedule_nodes', ['versions_id'], unique=False)
|
||||
op.create_index('uk_schedule_nodes_key', 'schedule_nodes', ['schedule_id', 'node_key'], unique=True)
|
||||
op.create_table('experiment_metrics',
|
||||
sa.Column('metric_id', mysql.BIGINT(), nullable=False),
|
||||
sa.Column('experiment_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('metric_name', sa.String(length=128), nullable=False),
|
||||
sa.Column('recorded_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('metric_value', sa.Double(asdecimal=True), nullable=True),
|
||||
sa.Column('metric_text', sa.String(length=1000), nullable=True),
|
||||
sa.Column('step_no', sa.BigInteger(), nullable=True),
|
||||
sa.ForeignKeyConstraint(['experiment_id'], ['experiments.experiment_id'], name='fk_experiment_metrics_experiment', ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('metric_id'),
|
||||
comment='实验指标，支持筛选和曲线'
|
||||
)
|
||||
op.create_index('idx_experiment_metrics_lookup', 'experiment_metrics', ['experiment_id', 'metric_name', 'step_no'], unique=False)
|
||||
op.create_table('experiment_resources',
|
||||
sa.Column('experiment_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('resource_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('resource_role', sa.String(length=16), server_default=sa.text("'input'"), nullable=False, comment='input/output'),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.ForeignKeyConstraint(['experiment_id'], ['experiments.experiment_id'], name='fk_experiment_resources_experiment', ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['resource_id'], ['data_resources.resource_id'], name='fk_experiment_resources_resource', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('experiment_id', 'resource_id', 'resource_role'),
|
||||
comment='实验与数据资源'
|
||||
)
|
||||
op.create_index('fk_experiment_resources_resource', 'experiment_resources', ['resource_id'], unique=False)
|
||||
op.create_table('schedule_edges',
|
||||
sa.Column('edge_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('schedule_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('source_node_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('target_node_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('condition_expr', sa.String(length=1000), nullable=True),
|
||||
sa.ForeignKeyConstraint(['schedule_id'], ['schedules.schedule_id'], name='fk_schedule_edges_schedule', ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['source_node_id'], ['schedule_nodes.node_id'], name='fk_schedule_edges_source', ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['target_node_id'], ['schedule_nodes.node_id'], name='fk_schedule_edges_target', ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('edge_id'),
|
||||
comment='DAG 有向边'
|
||||
)
|
||||
op.create_index('fk_schedule_edges_source', 'schedule_edges', ['source_node_id'], unique=False)
|
||||
op.create_index('idx_schedule_edges_target', 'schedule_edges', ['target_node_id'], unique=False)
|
||||
op.create_index('uk_schedule_edges_pair', 'schedule_edges', ['schedule_id', 'source_node_id', 'target_node_id'], unique=True)
|
||||
op.create_table('schedule_node_runs',
|
||||
sa.Column('node_run_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('run_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('node_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('versions_id', sa.CHAR(length=26), nullable=False),
|
||||
sa.Column('attempt_no', mysql.INTEGER(), server_default=sa.text('1'), nullable=False),
|
||||
sa.Column('node_status', sa.String(length=24), server_default=sa.text("'queued'"), nullable=False, comment='queued/running/succeeded/failed/skipped/cancelled/timed_out'),
|
||||
sa.Column('state_version', mysql.INTEGER(), server_default=sa.text('0'), nullable=False, comment='乐观锁版本'),
|
||||
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
|
||||
sa.Column('started_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('finished_at', mysql.DATETIME(fsp=3), nullable=True),
|
||||
sa.Column('duration_ms', mysql.BIGINT(), nullable=True),
|
||||
sa.Column('exit_code', sa.Integer(), nullable=True),
|
||||
sa.Column('message', sa.String(length=2000), nullable=True),
|
||||
sa.Column('metrics_json', sa.JSON(), nullable=True),
|
||||
sa.Column('logs_object_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.Column('result_object_id', sa.CHAR(length=26), nullable=True),
|
||||
sa.ForeignKeyConstraint(['logs_object_id'], ['storage_objects.storage_object_id'], name='fk_node_runs_logs', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['node_id'], ['schedule_nodes.node_id'], name='fk_node_runs_node', ondelete='RESTRICT'),
|
||||
sa.ForeignKeyConstraint(['result_object_id'], ['storage_objects.storage_object_id'], name='fk_node_runs_result', ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['run_id'], ['schedule_runs.run_id'], name='fk_node_runs_run', ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['versions_id'], ['versions.versions_id'], name='fk_node_runs_version', ondelete='RESTRICT'),
|
||||
sa.PrimaryKeyConstraint('node_run_id'),
|
||||
comment='调度节点运行与重试'
|
||||
)
|
||||
op.create_index('fk_node_runs_logs', 'schedule_node_runs', ['logs_object_id'], unique=False)
|
||||
op.create_index('fk_node_runs_node', 'schedule_node_runs', ['node_id'], unique=False)
|
||||
op.create_index('fk_node_runs_result', 'schedule_node_runs', ['result_object_id'], unique=False)
|
||||
op.create_index('idx_node_runs_status', 'schedule_node_runs', ['run_id', 'node_status'], unique=False)
|
||||
op.create_index('idx_node_runs_version', 'schedule_node_runs', ['versions_id'], unique=False)
|
||||
op.create_index('uk_node_runs_attempt', 'schedule_node_runs', ['run_id', 'node_id', 'attempt_no'], unique=True)
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""Downgrade schema."""
|
||||
# MySQL removes indexes with their table. Dropping FK-supporting indexes
|
||||
# explicitly first raises error 1553, so tables are removed in reverse
|
||||
# dependency order and MySQL performs the index cleanup.
|
||||
op.drop_table('schedule_node_runs')
|
||||
op.drop_table('schedule_edges')
|
||||
op.drop_table('experiment_resources')
|
||||
op.drop_table('experiment_metrics')
|
||||
op.drop_table('schedule_nodes')
|
||||
op.drop_table('experiments')
|
||||
op.drop_table('versions')
|
||||
op.drop_table('notebook_snapshots')
|
||||
op.drop_table('workspace_operations')
|
||||
op.drop_table('upload_sessions')
|
||||
op.drop_table('scripts')
|
||||
op.drop_table('schedule_runs')
|
||||
op.drop_table('edit_sessions')
|
||||
op.drop_table('data_resources')
|
||||
op.drop_table('workspace_members')
|
||||
op.drop_table('storage_objects')
|
||||
op.drop_table('schedules')
|
||||
op.drop_table('runtime_instances')
|
||||
op.drop_table('audit_logs')
|
||||
op.drop_table('workspaces')
|
||||
op.drop_table('users')
|
||||
op.drop_table('role_permissions')
|
||||
op.drop_table('roles')
|
||||
op.drop_table('permissions')
|
||||
op.drop_table('outbox_events')
|
||||
op.drop_table('consumer_inbox')
|
||||
# ### end Alembic commands ###
|
||||
@@ -0,0 +1,112 @@
|
||||
"""demo workspaces and users
|
||||
|
||||
Revision ID: 20260728_0002
|
||||
Revises: 20260724_0001
|
||||
Create Date: 2026-07-28
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
|
||||
|
||||
revision: str = "20260728_0002"
|
||||
down_revision: str | Sequence[str] | None = "20260724_0001"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
ADMIN_ROLE = "0000000000HNQN4KM476QNKW1C"
|
||||
DEVELOPER_ROLE = "00000000005RCQ4GPGBK3WZYMM"
|
||||
MODEL_WORKSPACE = "00000000000BM630VT9ARVFZPC"
|
||||
RISK_WORKSPACE = "0000000000AE0NC0V5T424KK86"
|
||||
ZHANG = "0000000000RF6FG1SDBXG59S13"
|
||||
LI = "0000000000H2QYCGPCWQM1JSGS"
|
||||
WANG = "0000000000RWG40ESZPGJT629J"
|
||||
ZHAO = "00000000004CQV7WASJA6N6FW4"
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
f"""
|
||||
INSERT INTO roles
|
||||
(role_id, role_code, role_name, role_scope, is_builtin)
|
||||
VALUES
|
||||
('{ADMIN_ROLE}', 'admin', '管理员', 'workspace', 1),
|
||||
('{DEVELOPER_ROLE}', 'developer', '开发人员', 'workspace', 1)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
role_name = VALUES(role_name),
|
||||
role_scope = VALUES(role_scope)
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
f"""
|
||||
INSERT INTO users
|
||||
(user_id, username, display_name, password_hash, status, email)
|
||||
VALUES
|
||||
('{ZHANG}', 'admin-zhang', '张三', 'demo-login-disabled', 'active',
|
||||
'zhangsan@example.local'),
|
||||
('{LI}', 'admin-li', '李四', 'demo-login-disabled', 'active',
|
||||
'lisi@example.local'),
|
||||
('{WANG}', 'dev-wang', '王五', 'demo-login-disabled', 'active',
|
||||
'wangwu@example.local'),
|
||||
('{ZHAO}', 'dev-zhao', '赵六', 'demo-login-disabled', 'active',
|
||||
'zhaoliu@example.local')
|
||||
ON DUPLICATE KEY UPDATE
|
||||
username = VALUES(username),
|
||||
display_name = VALUES(display_name),
|
||||
status = 'active',
|
||||
email = VALUES(email)
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
f"""
|
||||
INSERT INTO workspaces
|
||||
(workspace_id, workspace_code, workspace_name, active_root_uri,
|
||||
quota_bytes, used_bytes, status, created_by, description,
|
||||
artifact_bucket, artifact_prefix)
|
||||
VALUES
|
||||
('{MODEL_WORKSPACE}', 'model-dev', '模型开发 Workspace',
|
||||
'file:///workspace/workspaces/model-dev', 0, 0, 'active',
|
||||
'{ZHANG}', '模型开发与脚本调度', 'model-platform',
|
||||
'workspaces/model-dev'),
|
||||
('{RISK_WORKSPACE}', 'risk-validation', '风险验证 Workspace',
|
||||
'file:///workspace/workspaces/risk-validation', 0, 0, 'active',
|
||||
'{LI}', '风险模型验证与批处理', 'model-platform',
|
||||
'workspaces/risk-validation')
|
||||
ON DUPLICATE KEY UPDATE
|
||||
workspace_name = VALUES(workspace_name),
|
||||
active_root_uri = VALUES(active_root_uri),
|
||||
status = 'active',
|
||||
description = VALUES(description)
|
||||
"""
|
||||
)
|
||||
values = []
|
||||
for workspace_id in (MODEL_WORKSPACE, RISK_WORKSPACE):
|
||||
for user_id, role_id in (
|
||||
(ZHANG, ADMIN_ROLE),
|
||||
(LI, ADMIN_ROLE),
|
||||
(WANG, DEVELOPER_ROLE),
|
||||
(ZHAO, DEVELOPER_ROLE),
|
||||
):
|
||||
values.append(
|
||||
f"('{workspace_id}', '{user_id}', '{role_id}', 'active')"
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
INSERT INTO workspace_members
|
||||
(workspace_id, user_id, role_id, member_status)
|
||||
VALUES
|
||||
"""
|
||||
+ ",\n".join(values)
|
||||
+ """
|
||||
ON DUPLICATE KEY UPDATE
|
||||
role_id = VALUES(role_id),
|
||||
member_status = 'active'
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# 演示身份可能已产生业务数据，降级时保留，避免破坏外键引用。
|
||||
pass
|
||||
@@ -0,0 +1,41 @@
|
||||
"""decouple schedule artifact visibility from version history
|
||||
|
||||
Revision ID: 20260728_0003
|
||||
Revises: 20260728_0002
|
||||
Create Date: 2026-07-28
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import mysql
|
||||
|
||||
|
||||
revision: str = "20260728_0003"
|
||||
down_revision: str | Sequence[str] | None = "20260728_0002"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"versions",
|
||||
sa.Column(
|
||||
"schedule_hidden_at",
|
||||
mysql.DATETIME(fsp=3),
|
||||
nullable=True,
|
||||
comment="从调度稳定版本列表移除的时间；不影响版本和运行历史",
|
||||
),
|
||||
)
|
||||
op.create_index(
|
||||
"idx_versions_schedule_visible",
|
||||
"versions",
|
||||
["workspace_id", "schedule_hidden_at", "created_at"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_versions_schedule_visible", table_name="versions")
|
||||
op.drop_column("versions", "schedule_hidden_at")
|
||||
@@ -0,0 +1,49 @@
|
||||
"""remove Redis-specific lock column naming
|
||||
|
||||
Revision ID: 20260730_0004
|
||||
Revises: 20260728_0003
|
||||
Create Date: 2026-07-30
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision: str = "20260730_0004"
|
||||
down_revision: str | None = "20260728_0003"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _column_names() -> set[str]:
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
return {item["name"] for item in inspector.get_columns("edit_sessions")}
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if context.is_offline_mode():
|
||||
return
|
||||
columns = _column_names()
|
||||
if "redis_lock_key" in columns and "lock_key" not in columns:
|
||||
op.alter_column(
|
||||
"edit_sessions",
|
||||
"redis_lock_key",
|
||||
new_column_name="lock_key",
|
||||
existing_type=sa.String(length=512),
|
||||
existing_nullable=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if context.is_offline_mode():
|
||||
return
|
||||
columns = _column_names()
|
||||
if "lock_key" in columns and "redis_lock_key" not in columns:
|
||||
op.alter_column(
|
||||
"edit_sessions",
|
||||
"lock_key",
|
||||
new_column_name="redis_lock_key",
|
||||
existing_type=sa.String(length=512),
|
||||
existing_nullable=False,
|
||||
)
|
||||
@@ -0,0 +1,7 @@
|
||||
# Migration Revisions
|
||||
|
||||
该目录只保存经过评审和验证的 Alembic 迁移版本。
|
||||
|
||||
- 已发布迁移不得直接修改。
|
||||
- 新迁移必须同时提供可执行的 `upgrade()` 和 `downgrade()`。
|
||||
- 自动生成后必须检查数据类型、约束、索引和执行顺序。
|
||||
Reference in New Issue
Block a user