tao.chen/model-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: permission

Browse Source
This commit is contained in:
tao.chen
2026-09-02 10:10:41 +08:00
committed by tao.chen
parent 1c220490cf
commit dea6b4cfcf
11 changed files with 171 additions and 1037 deletions
Download Patch File Download Diff File Expand all files Collapse all files
API.md
+17 -30
View File
@@ -645,7 +645,7 @@ Base 前缀 `/api/v1/admin`。
| `DELETE` | `/api/v1/platform/workspaces/{workspace_id}/members/{user_id}` | 软删成员 |
| `GET` | `/api/v1/platform/roles` | 列全部 `role_scope='platform'` 的角色及其 `permission_codes`(仅系统管理员) |
| `GET` | `/api/v1/platform/roles/{role_code}/permissions` | 单个平台角色的 `permission_codes`(仅系统管理员) |
| `PATCH` | `/api/v1/platform/roles/{role_code}/permissions` | 整体替换角色权限集合;对 `admin` 角色保留 `system.view` + `system.manage` 的最后系统管理员保护(仅系统管理员) |
| `PATCH` | `/api/v1/platform/roles/{role_code}/permissions` | 整体替换角色权限集合;对 `admin` 角色强制保留 `system:view` 的最后系统管理员保护(仅系统管理员) |
> **不变量**:
> - 每个 workspace 必须始终保留至少一个 `admin` 角色的活跃成员;对最后 admin 做降级 / 停用 / 删除 → 409。
@@ -883,11 +883,8 @@ Base 前缀 `/api/v1/admin`。
"role_name": "管理员",
"is_builtin": true,
"permission_codes": [
"dashboard.view", "experiment.all", "experiment.own",
"resource.personal", "resource.public.manage",
"resource.public.upload", "schedule.all", "schedule.own",
"script.build", "script.public.manage", "system.manage",
"system.view"
"dashboard:view", "schedule:view", "script:view",
"system:project:view", "system:user:view", "system:view"
]
},
{
@@ -896,8 +893,7 @@ Base 前缀 `/api/v1/admin`。
"role_name": "开发人员",
"is_builtin": true,
"permission_codes": [
"dashboard.view", "experiment.own", "resource.personal",
"schedule.own", "script.build", "script.public.manage"
"dashboard:view", "schedule:view", "script:view"
]
}
],
@@ -923,11 +919,8 @@ Base 前缀 `/api/v1/admin`。
"role_name": "管理员",
"is_builtin": true,
"permission_codes": [
"dashboard.view", "experiment.all", "experiment.own",
"resource.personal", "resource.public.manage",
"resource.public.upload", "schedule.all", "schedule.own",
"script.build", "script.public.manage", "system.manage",
"system.view"
"dashboard:view", "schedule:view", "script:view",
"system:project:view", "system:user:view", "system:view"
]
},
"meta": {}
@@ -948,8 +941,8 @@ Base 前缀 `/api/v1/admin`。
- **守卫顺序(load-bearing,不可调换)**:
1. 角色不存在 / `role_scope != 'platform'` → 404。
2. **`admin` 角色**:提交的 `permission_codes` 必须同时包含 `system.view` 与 `system.manage`,否则 → 409 "admin 角色必须保留 system.view 与 system.manage 权限"。
3. **非 `admin` 角色**:`permission_codes` 中**禁止**含 `system.*` 项 → 422 "非 admin 角色不能拥有 system.* 权限: [...]"。menu permission 只控前端展示,后端 `/api/v1/platform/*` 鉴权仍按 `role_code == "admin"`;若让 developer 拿到 `system.manage`,前端会渲染"系统管理"入口但所有 platform API 调用 403,UX 割裂。
2. **`admin` 角色**:提交的 `permission_codes` 必须包含 `system:view`,否则 → 409 "admin 角色必须保留 system:view 权限"。
3. **非 `admin` 角色**:`permission_codes` 中**禁止**含 `system:*` 项 → 422 "非 admin 角色不能拥有 system:* 权限: [...]"。menu permission 只控前端展示,后端 `/api/v1/platform/*` 鉴权仍按 `role_code == "admin"`;若让 developer 拿到 `system:view`,前端会渲染"系统管理"入口但所有 platform API 调用 403,UX 割裂。
4. 任意 `permission_code` 不在 `permissions` 表活跃行中 → 422 "未知的 permission_code: [...]"。
5. 写入策略:diff-based —— 只 soft-delete `current \ new` 的关联,只 INSERT `new \ current` 的关联。重复提交同 payload 是 no-op;包含原有 codes 的 patch 不会触发 `(role_id, permission_id)` 主键冲突。**严禁**先全量 soft-delete 再全量 INSERT(会 `IntegrityError`,因为软删行仍占主键 slot)。
- 写入后,响应 `data.permission_codes` 为本次写入后的活跃集合,与再次 `GET §7.13` 完全一致。
@@ -963,32 +956,26 @@ Base 前缀 `/api/v1/admin`。
"role_code": "admin",
"role_name": "管理员",
"is_builtin": true,
"permission_codes": ["dashboard.view", "system.manage", "system.view"]
"permission_codes": ["dashboard:view", "system:view"]
},
"meta": {}
}
```
- **当前 seed 的 permission_code 全集**(来自迁移 `f6a7b8c9d0e1`,与 `migrations/data/migrate_system_json.py::PERMISSION_NAMES` 真值对齐,不要在客户端另造一份):
- **当前 seed 的 permission_code 全集**(由 baseline `e1f2a3b4c5d6_rebuild_baseline` 直接写入 MySQL,不要在客户端另造一份):
| `permission_code` | `module_code` | admin | developer |
|---|---|:-:|:-:|
| `dashboard.view` | dashboard | ✓ | ✓ |
| `script.build` | script | ✓ | ✓ |
| `script.public.manage` | script | ✓ | ✓ |
| `schedule.own` | schedule | ✓ | ✓ |
| `schedule.all` | schedule | ✓ | |
| `experiment.own` | experiment | ✓ | ✓ |
| `experiment.all` | experiment | ✓ | |
| `resource.personal` | resource | ✓ | ✓ |
| `resource.public.upload` | resource | ✓ | |
| `resource.public.manage` | resource | ✓ | |
| `system.view` | system | ✓ | |
| `system.manage` | system | ✓ | |
| `dashboard:view` | dashboard | ✓ | ✓ |
| `script:view` | script | ✓ | ✓ |
| `schedule:view` | schedule | ✓ | ✓ |
| `system:view` | system | ✓ | |
| `system:user:view` | system | ✓ | |
| `system:project:view` | system | ✓ | |
> **与现有 7.x 端点的语义差异**(避免 reviewer 误读):
> - 本端点不修改 `users.platform_role_id`,只调整 `role_permissions` 关联表。
> - "清空 developer 的全部权限"是合法操作;只有 `admin` 受 `system.*` 强制约束。
> - "清空 developer 的全部权限"是合法操作;只有 `admin` 受 `system:*` 强制约束。
> - `role_code` 不是 `permission_code`,前端不要用前者去判断菜单可见性。
---
backend/src/backend/api/platform.py
+5 -5
View File
@@ -58,7 +58,7 @@ Invariants
* ``DELETE /workspaces/{id}`` is allowed from any non-disabled status and
sets ``status='disabled'`` + ``is_deleted=1`` + ``deleted_at`` on the
workspace and every one of its active memberships.
* The ``admin`` role must always keep ``system.view`` + ``system.manage``
* The ``admin`` role must always keep ``system:view``.
menu permissions; non-admin roles may never hold ``system.*``
permissions. Menu permissions gate frontend rendering only — API
authorization always keys off ``role_code == 'admin'``.
@@ -1185,7 +1185,7 @@ async def patch_role_permissions(
1. Load the role. Reject 404 if it is missing or not
platform-scoped.
2. Admin role: the patched ``permission_codes`` MUST still include
both ``system.view`` and ``system.manage``. Otherwise every
``system:view``. Otherwise every
active admin loses the menu entry to this very endpoint and
the platform locks itself out. Reject with 409. (No last-admin
count is needed here — menu permissions never gate API access;
@@ -1207,12 +1207,12 @@ async def patch_role_permissions(
if role.role_code == "admin":
keeps_admin_entry = (
"system.view" in new_codes and "system.manage" in new_codes
"system:view" in new_codes
)
if not keeps_admin_entry:
raise HTTPException(
status.HTTP_409_CONFLICT,
"admin 角色必须保留 system.view 与 system.manage 权限",
"admin 角色必须保留 system:view 权限",
)
else:
# Menu permissions are a frontend-display signal only — backend
@@ -1222,7 +1222,7 @@ async def patch_role_permissions(
# /api/v1/platform/* call still returns 403. Reject with 422 so
# the failure is unambiguous about *what* the input violated.
leaked_system = [
code for code in new_codes if code.startswith("system.")
code for code in new_codes if code.startswith("system:")
]
if leaked_system:
raise HTTPException(
frontend/app/context/AuthContext.tsx
+6
View File
@@ -20,6 +20,7 @@ export type AuthUser = {
status: string;
role_code: string | null;
is_system_admin: boolean;
permissions: string[];
};
export type AuthWorkspace = {
@@ -312,3 +313,8 @@ export function useApi(): WorkspaceBoundApi {
deleteWorkspaceMember: (workspaceId, userId) => rawApi.deleteWorkspaceMember(workspaceId, userId),
}), [workspaceId]);
}
export function usePermission(code: string): boolean {
const { user } = useAuth();
return user?.permissions.includes(code) ?? false;
}
frontend/app/features/admin/DashboardPage.tsx
+3 -3
View File
@@ -1,5 +1,5 @@
import Icon from "../../components/common/Icon";
import { useAuth } from "../../context/AuthContext";
import { useAuth, usePermission } from "../../context/AuthContext";
const TREND_VALUES = [38, 55, 44, 73, 61, 86, 78] as const;
const TREND_LABELS = ["周一", "周二", "周三", "周四", "周五", "周六", "今天"] as const;
@@ -25,7 +25,7 @@ export function DashboardPage({
onNavigate: (page: "scripts" | "schedules" | "system") => void;
}) {
const { user, currentWorkspace } = useAuth();
const isSystemAdmin = user?.is_system_admin ?? false;
const hasSystemView = usePermission("system:view");
const notebookCount = Math.max(1, Math.round(scriptCount * 0.67));
const pythonCount = Math.max(0, scriptCount - Math.round(scriptCount * 0.67));
@@ -92,7 +92,7 @@ export function DashboardPage({
<Icon name="schedule" />
调度配置
</button>
{isSystemAdmin && (
{hasSystemView && (
<button
type="button"
className="flex cursor-pointer items-center gap-2 rounded-[7px] border border-[#d8e4ef] bg-white px-[18px] py-[13px] text-[#346587]"
frontend/app/routes/platform.navigation.tsx
+38
View File
@@ -0,0 +1,38 @@
import { Home, Code, Calendar, Settings, Users, Folder } from "lucide-react";
import type { ComponentType } from "react";
export type ActivePage = "home" | "scripts" | "schedules" | "system";
export type SubPage = "users" | "projects";
export type NavigationItem = {
/** 菜单显示名 */
label: string;
/** 渲染在菜单前的图标组件（lucide-react 或等价） */
icon: ComponentType;
/** 入口页 — 点击后由 guardedNavigate 处理 */
page: ActivePage;
/** 子页段，仅 children 项使用 */
sub?: SubPage;
/** 命中高亮的 pathname；不填则按 pathForPage(page) 派生 */
activePath?: string;
/** 当前端可访问的权限码；未配置时视为所有人可访问 */
permission?: string;
/** 子菜单条目 */
children?: NavigationItem[];
};
export const navigation: NavigationItem[] = [
{ label: "工作台", icon: Home, page: "home", permission: "dashboard:view" },
{ label: "构建脚本", icon: Code, page: "scripts", permission: "script:view" },
{ label: "调度配置", icon: Calendar, page: "schedules", permission: "schedule:view" },
{
label: "系统管理",
icon: Settings,
page: "system",
permission: "system:view",
children: [
{ label: "用户管理", icon: Users, page: "system", sub: "users", permission: "system:user:view" },
{ label: "项目管理", icon: Folder, page: "system", sub: "projects", permission: "system:project:view" },
],
},
];
frontend/app/routes/platform.tsx
+90 -78
View File
@@ -1,7 +1,9 @@
import { useEffect, useState } from "react";
import { Outlet, useLocation, useNavigate } from "react-router";
import { Home, Code, Calendar, Settings, Users, Folder, PanelLeft, ChevronRight } from "lucide-react";
import { PanelLeft, ChevronRight } from "lucide-react";
import { navigation } from "./platform.navigation";
import type { ActivePage, NavigationItem } from "./platform.navigation";
import { Collapsible } from "@base-ui/react/collapsible";
import type { Route } from "./+types/platform";
@@ -23,7 +25,7 @@ import {
SidebarTrigger,
SidebarInset,
} from "~/components/ui/sidebar";
import { useApi, useAuth } from "~/context/AuthContext";
import { useApi, useAuth, usePermission } from "~/context/AuthContext";
import { useEditSessionLifecycle } from "~/features/platform/hooks/useEditSessionLifecycle";
import {
bindScriptWorkspaceApi,
@@ -36,8 +38,6 @@ import { useUiStore } from "~/features/platform/state/uiStore";
import { bindAdminApi } from "~/features/admin/state/adminStore";
import { bindSchedulesApi } from "~/features/schedules/state/helpers";
type ActivePage = "home" | "scripts" | "schedules" | "system";
function pageFromPath(pathname: string): ActivePage {
const first = pathname.replace(/^\/+/, "").split("/")[0];
if (first === "workbench" || first === "") return "home";
@@ -59,6 +59,80 @@ export function meta({}: Route.MetaArgs) {
];
}
function NavRow({
item,
can,
activePage,
pathname,
onNavigate,
}: {
item: NavigationItem;
can: (code: string) => boolean;
activePage: ActivePage;
pathname: string;
onNavigate: (sub?: "users" | "projects") => void;
}) {
const itemIcon = <item.icon />;
const childActivePath = (child: NavigationItem) =>
child.activePath ?? pathForPage(child.page);
if (item.children && item.children.length > 0) {
return (
<Collapsible.Root
defaultOpen={activePage === item.page}
className="group/collapsible"
>
<SidebarMenuItem>
<Collapsible.Trigger
render={
<SidebarMenuButton
className="py-3 text-sm"
isActive={activePage === item.page}
data-active={activePage === item.page ? "true" : undefined}
/>
}
>
{itemIcon}
<span>{item.label}</span>
<ChevronRight className="ml-auto size-4 transition-transform group-data-[open]/collapsible:rotate-90" />
</Collapsible.Trigger>
<Collapsible.Panel>
<SidebarMenuSub className="gap-1.5 mt-2">
{item.children
.filter((child) => !child.permission || can(child.permission))
.map((child) => (
<SidebarMenuSubItem key={child.label}>
<SidebarMenuSubButton
className="h-9 text-xs"
onClick={() => onNavigate(child.sub)}
isActive={pathname === childActivePath(child)}
>
<child.icon /> <span>{child.label}</span>
</SidebarMenuSubButton>
</SidebarMenuSubItem>
))}
</SidebarMenuSub>
</Collapsible.Panel>
</SidebarMenuItem>
</Collapsible.Root>
);
}
return (
<SidebarMenuItem>
<SidebarMenuButton
className="py-3 text-sm"
onClick={() => onNavigate()}
isActive={activePage === item.page}
data-active={activePage === item.page ? "true" : undefined}
>
{itemIcon}
<span>{item.label}</span>
</SidebarMenuButton>
</SidebarMenuItem>
);
}
export default function PlatformLayout() {
const { currentWorkspace } = useAuth();
if (!currentWorkspace) {
@@ -98,7 +172,7 @@ function AuthenticatedLayout() {
const [sidebarCollapsed, setSidebarCollapsed] = useState(false);
const isSystemAdmin = user?.is_system_admin ?? false;
const can = usePermission;
// 绑定 api 到 script workspace store。
// 必须放在 render body(同步),不能用 useEffect([api]) + cleanup —
@@ -168,80 +242,18 @@ function AuthenticatedLayout() {
</SidebarHeader>
<SidebarContent>
<SidebarMenu className="gap-1.5">
<SidebarMenuItem>
<SidebarMenuButton
className="py-3 text-sm"
onClick={() => guardedNavigate("home")}
isActive={activePage === "home"}
data-active={activePage === "home" ? "true" : undefined}
>
<Home /> <span>工作台</span>
</SidebarMenuButton>
</SidebarMenuItem>
<SidebarMenuItem>
<SidebarMenuButton
className="py-3 text-sm"
onClick={() => guardedNavigate("scripts")}
isActive={activePage === "scripts"}
data-active={activePage === "scripts" ? "true" : undefined}
>
<Code /> <span>构建脚本</span>
</SidebarMenuButton>
</SidebarMenuItem>
<SidebarMenuItem>
<SidebarMenuButton
className="py-3 text-sm"
onClick={() => guardedNavigate("schedules")}
isActive={activePage === "schedules"}
data-active={activePage === "schedules" ? "true" : undefined}
>
<Calendar /> <span>调度配置</span>
</SidebarMenuButton>
</SidebarMenuItem>
{isSystemAdmin && (
<Collapsible.Root
defaultOpen={activePage === "system"}
className="group/collapsible"
>
<SidebarMenuItem>
<Collapsible.Trigger
render={
<SidebarMenuButton
className="py-3 text-sm"
isActive={activePage === "system"}
data-active={activePage === "system" ? "true" : undefined}
{navigation
.filter((item) => !item.permission || can(item.permission))
.map((item) => (
<NavRow
key={item.page}
item={item}
can={can}
activePage={activePage}
pathname={location.pathname}
onNavigate={(sub) => guardedNavigate(item.page, sub)}
/>
}
>
<Settings />
<span>系统管理</span>
<ChevronRight className="ml-auto size-4 transition-transform group-data-[open]/collapsible:rotate-90" />
</Collapsible.Trigger>
<Collapsible.Panel>
<SidebarMenuSub className="gap-1.5 mt-2">
<SidebarMenuSubItem>
<SidebarMenuSubButton
className="h-9 text-xs"
onClick={() => guardedNavigate("system", "users")}
isActive={location.pathname === "/system/users"}
>
<Users /> <span>用户管理</span>
</SidebarMenuSubButton>
</SidebarMenuSubItem>
<SidebarMenuSubItem>
<SidebarMenuSubButton
className="h-9 text-xs"
onClick={() => guardedNavigate("system", "projects")}
isActive={location.pathname === "/system/projects"}
>
<Folder /> <span>项目管理</span>
</SidebarMenuSubButton>
</SidebarMenuSubItem>
</SidebarMenuSub>
</Collapsible.Panel>
</SidebarMenuItem>
</Collapsible.Root>
)}
))}
</SidebarMenu>
</SidebarContent>
<SidebarFooter>
migrations/data/README.md
-70
View File
@@ -1,70 +0,0 @@
# Data Migrations
该目录保存从旧版 JSON 状态文件迁移到 MySQL 的一次性工具。
数据迁移工具必须满足：
- 默认只预检，必须显式传入 `--apply` 才能写数据库；
- 单个事务提交，失败时完整回滚；
- 可以安全重复执行，不产生重复数据；
- 输出源文件摘要、源数据数量和实际变更数量；
- 不修改或删除旧版 JSON 源文件。
## system.json
```powershell
$env:DATABASE_URL = "mysql+asyncmy://<user>:<password>@<host>:3306/<database>?charset=utf8mb4"
python -m migrations.data.migrate_system_json --source "<path>/system.json"
python -m migrations.data.migrate_system_json --source "<path>/system.json" --apply
```
新迁移用户使用不可登录的占位密码哈希。后续接入认证时，必须通过密码初始化、
管理员重置或外部身份认证启用登录，不能把旧版无密码账号视为已有凭据。
## WORKSPACE_DEFINITIONS
旧版 Workspace 定义位于 `文件1/server.py` 的
`WORKSPACE_DEFINITIONS` 常量中。迁移工具通过 Python AST 仅读取这一静态常量，
不执行旧服务代码：
```powershell
python -m migrations.data.migrate_legacy_workspaces --source "<path>/server.py"
python -m migrations.data.migrate_legacy_workspaces --source "<path>/server.py" --apply
```
- 创建者取每个 Workspace 成员中的第一个管理员；
- 成员角色沿用第 9 步迁入的平台注册角色；
- 活跃目录统一为 `file:///workspace/workspaces/{workspace_code}`；
- 制品前缀统一为 `workspaces/{workspace_id}`；
- 旧审计记录仅在用户唯一属于一个 Workspace 时补齐归属。
## 第 9 小步执行记录
- 执行日期：2026-07-24
- 源文件：`文件1/platform_data/system.json`
- SHA-256：`ef4ee0e92f4a3679c17f23aff6066688fa9231db4477208e36fcbd343cd446a7`
- 迁移结果：2 个角色、13 个权限、21 条角色权限、4 个用户、26 条审计记录
- 角色权限：`admin=13`，`developer=8`
- 中文字段：UTF-8 校验通过，问号乱码记录为 0
- 幂等验证：第二次执行插入和更新均为 0，跳过已有审计记录 26 条
- 旧版源文件：未修改、未删除
## 第 10 小步执行记录
- 执行日期：2026-07-24
- 源文件及常量：`文件1/server.py` / `WORKSPACE_DEFINITIONS`
- SHA-256：`d7ec05f1ab7b2cdeb78f6293de40fbe23fa74b6a7fe4ccb03a9cc64e4cc1f464`
- 迁移结果：2 个 Workspace、4 条成员关系
- 成员分布：`model-dev` 2 人，`risk-validation` 2 人
- 审计归属：补齐 26 条，其中 `model-dev` 24 条、`risk-validation` 2 条
- 中文字段：UTF-8 校验通过，异常记录为 0
- 幂等验证：第二次执行及镜像内 dry-run 的变更数均为 0
- 镜像验证：仅挂载旧版 `server.py` 时，迁移工具可独立读取并完成校验
- 旧版源文件：未修改、未删除
## 第 11、12 小步数据处理决定
根据实施确认，第 11、12 小步不迁移旧版资源、脚本或稳定版本数据。新实现直接
使用 MySQL、Workspace 文件目录和对象存储（s3 模式连 S3-兼容服务，local 模式
走 `/data/storage` 共享卷），从空的 `data_resources`、`scripts` 及
`versions` 表开始运行。功能验收产生的临时对象和数据库记录均已清理。
migrations/data/migrate_legacy_workspaces.py
-365
View File
@@ -1,365 +0,0 @@
from __future__ import annotations
import argparse
import ast
import asyncio
import hashlib
import json
import os
from pathlib import Path
from typing import Any
from urllib.parse import quote
from common.db import create_database_engine, create_session_factory
from common.db.models import (
Roles,
Users,
WorkspaceMembers,
Workspaces,
)
from pydantic import BaseModel, ConfigDict, TypeAdapter
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from migrations.data.migrate_system_json import (
deterministic_legacy_ulid,
set_changed,
)
class LegacyWorkspace(BaseModel):
model_config = ConfigDict(extra="forbid")
id: str
name: str
description: str | None = None
userIds: list[str]
portOffset: int
WORKSPACE_LIST = TypeAdapter(list[LegacyWorkspace])
def require_unique(values: list[str], label: str) -> None:
duplicates = sorted(
value for value in set(values) if values.count(value) > 1
)
if duplicates:
raise ValueError(f"duplicate {label}: {duplicates}")
def extract_workspace_definitions(
source_path: Path,
) -> tuple[list[LegacyWorkspace], str]:
raw = source_path.read_bytes()
module = ast.parse(
raw.decode("utf-8-sig"),
filename=str(source_path),
)
definition: Any | None = None
for statement in module.body:
if not isinstance(statement, ast.Assign):
continue
if any(
isinstance(target, ast.Name)
and target.id == "WORKSPACE_DEFINITIONS"
for target in statement.targets
):
definition = ast.literal_eval(statement.value)
break
if definition is None:
raise ValueError("WORKSPACE_DEFINITIONS was not found")
workspaces = WORKSPACE_LIST.validate_python(definition)
require_unique([item.id for item in workspaces], "workspace ids")
for workspace in workspaces:
require_unique(
workspace.userIds,
f"members of workspace {workspace.id}",
)
if not workspace.userIds:
raise ValueError(
f"workspace {workspace.id!r} has no members"
)
return workspaces, hashlib.sha256(raw).hexdigest()
def new_stats() -> dict[str, int]:
return {
"workspaces_inserted": 0,
"workspaces_updated": 0,
"workspace_members_inserted": 0,
"workspace_members_updated": 0,
}
async def load_users_and_roles(
session: AsyncSession,
) -> tuple[dict[str, Users], dict[str, str]]:
users = {
item.username: item
for item in (
await session.scalars(select(Users).order_by(Users.username))
).all()
}
roles_by_id = {
item.role_id: item.role_code
for item in (
await session.scalars(select(Roles).order_by(Roles.role_code))
).all()
}
role_codes_by_username = {
username: roles_by_id[user.platform_role_id]
for username, user in users.items()
if user.platform_role_id in roles_by_id
}
return users, role_codes_by_username
def validate_members(
workspaces: list[LegacyWorkspace],
users: dict[str, Users],
role_codes_by_username: dict[str, str],
) -> None:
source_members = {
username
for workspace in workspaces
for username in workspace.userIds
}
missing_users = sorted(source_members - set(users))
if missing_users:
raise ValueError(
f"workspace members were not migrated as users: {missing_users}"
)
users_without_roles = sorted(
source_members - set(role_codes_by_username)
)
if users_without_roles:
raise ValueError(
"workspace members have no platform role: "
f"{users_without_roles}"
)
def workspace_creator(
workspace: LegacyWorkspace,
users: dict[str, Users],
role_codes_by_username: dict[str, str],
) -> Users:
admin_username = next(
(
username
for username in workspace.userIds
if role_codes_by_username[username] == "admin"
),
workspace.userIds[0],
)
return users[admin_username]
async def migrate_workspaces(
session: AsyncSession,
source: list[LegacyWorkspace],
users: dict[str, Users],
role_codes_by_username: dict[str, str],
stats: dict[str, int],
) -> dict[str, str]:
existing = {
item.workspace_code: item
for item in (
await session.scalars(
select(Workspaces).order_by(Workspaces.workspace_code)
)
).all()
}
workspace_ids: dict[str, str] = {}
for legacy in source:
workspace = existing.get(legacy.id)
if workspace is None:
workspace_id = deterministic_legacy_ulid(
"workspace", legacy.id
)
creator = workspace_creator(
legacy,
users,
role_codes_by_username,
)
workspace = Workspaces(
workspace_id=workspace_id,
workspace_code=legacy.id,
workspace_name=legacy.name,
description=legacy.description,
active_root_uri=(
"file:///workspace/workspaces/"
f"{quote(legacy.id, safe='')}"
),
quota_bytes=0,
used_bytes=0,
status="active",
created_by=creator.user_id,
artifact_bucket="model-platform",
artifact_prefix=f"workspaces/{workspace_id}",
)
session.add(workspace)
stats["workspaces_inserted"] += 1
else:
values = {
"workspace_name": legacy.name,
"description": legacy.description,
"active_root_uri": (
"file:///workspace/workspaces/"
f"{quote(legacy.id, safe='')}"
),
"artifact_bucket": "model-platform",
"artifact_prefix": (
f"workspaces/{workspace.workspace_id}"
),
}
if set_changed(workspace, values):
stats["workspaces_updated"] += 1
workspace_ids[legacy.id] = workspace.workspace_id
return workspace_ids
async def migrate_members(
session: AsyncSession,
source: list[LegacyWorkspace],
workspace_ids: dict[str, str],
users: dict[str, Users],
stats: dict[str, int],
) -> None:
existing = {
(item.workspace_id, item.user_id): item
for item in (
await session.scalars(select(WorkspaceMembers))
).all()
}
for legacy in source:
workspace_id = workspace_ids[legacy.id]
for username in legacy.userIds:
user = users[username]
pair = (workspace_id, user.user_id)
member = existing.get(pair)
values = {
"role_id": user.platform_role_id,
"member_status": "active",
}
if member is None:
member = WorkspaceMembers(
workspace_id=workspace_id,
user_id=user.user_id,
**values,
)
session.add(member)
existing[pair] = member
stats["workspace_members_inserted"] += 1
elif set_changed(member, values):
stats["workspace_members_updated"] += 1
async def run_migration(
database_url: str,
source: list[LegacyWorkspace],
*,
apply_changes: bool,
) -> dict[str, int]:
engine = create_database_engine(database_url)
factory = create_session_factory(engine)
stats = new_stats()
try:
async with factory() as session:
try:
users, role_codes_by_username = (
await load_users_and_roles(session)
)
validate_members(
source,
users,
role_codes_by_username,
)
workspace_ids = await migrate_workspaces(
session,
source,
users,
role_codes_by_username,
stats,
)
await migrate_members(
session,
source,
workspace_ids,
users,
stats,
)
if apply_changes:
await session.commit()
else:
await session.rollback()
except Exception:
await session.rollback()
raise
finally:
await engine.dispose()
return stats
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description=(
"Migrate legacy WORKSPACE_DEFINITIONS into MySQL."
)
)
parser.add_argument(
"--source",
required=True,
type=Path,
help="Path to the legacy server.py file.",
)
parser.add_argument(
"--apply",
action="store_true",
help="Commit changes. Without this flag the transaction is rolled back.",
)
return parser.parse_args()
async def async_main() -> None:
args = parse_args()
source_path = args.source.resolve(strict=True)
workspaces, source_sha256 = extract_workspace_definitions(source_path)
stats = await run_migration(
os.environ["DATABASE_URL"],
workspaces,
apply_changes=args.apply,
)
result = {
"mode": "apply" if args.apply else "dry-run",
"source": str(source_path),
"source_symbol": "WORKSPACE_DEFINITIONS",
"source_sha256": source_sha256,
"source_counts": {
"workspaces": len(workspaces),
"workspace_members": sum(
len(workspace.userIds) for workspace in workspaces
),
},
"changes": stats,
}
print(json.dumps(result, ensure_ascii=False, indent=2))
def main() -> None:
asyncio.run(async_main())
if __name__ == "__main__":
main()
migrations/data/migrate_system_json.py
-406
View File
@@ -1,406 +0,0 @@
from __future__ import annotations
import argparse
import asyncio
import hashlib
import json
import os
from pathlib import Path
from typing import Any
from urllib.parse import quote
from common.db import create_database_engine, create_session_factory
from common.db.models import (
Permissions,
RolePermissions,
Roles,
Users,
)
from pydantic import BaseModel, ConfigDict
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
CROCKFORD_BASE32 = "0123456789ABCDEFGHJKMNPQRSTVWXYZ"
LEGACY_PASSWORD_HASH = "!legacy-account-without-password!"
PERMISSION_NAMES = {
"dashboard.view": "查看工作台",
"script.build": "构建脚本",
"script.public.manage": "管理公共脚本",
"schedule.own": "管理本人调度",
"schedule.all": "管理全部调度",
"experiment.own": "管理本人实验",
"experiment.all": "管理全部实验",
"resource.personal": "管理个人资源",
"resource.public.upload": "上传公共资源",
"resource.public.manage": "管理公共资源",
"system.view": "查看系统管理",
"system.manage": "管理系统配置",
}
class LegacyUser(BaseModel):
model_config = ConfigDict(extra="forbid")
id: str
name: str
role: str
roleKey: str
avatar: str | None = None
class LegacyRole(BaseModel):
model_config = ConfigDict(extra="forbid")
key: str
name: str
description: str | None = None
permissions: list[str]
class LegacySystem(BaseModel):
"""Legacy platform_data/system.json shape.
``extra='ignore'`` so legacy files that include other top-level
keys (e.g. historical audit-log payloads) can still be parsed —
only the fields this script actually consumes are listed below.
"""
model_config = ConfigDict(extra="ignore")
users: list[LegacyUser]
roles: list[LegacyRole]
def deterministic_legacy_ulid(entity_type: str, legacy_key: str) -> str:
"""Create a stable ULID-compatible ID with an epoch timestamp prefix."""
digest = hashlib.sha256(
f"model-platform-v1:{entity_type}:{legacy_key}".encode()
).digest()
value = int.from_bytes(b"\x00" * 6 + digest[:10], byteorder="big")
encoded = ["0"] * 26
for index in range(25, -1, -1):
encoded[index] = CROCKFORD_BASE32[value & 31]
value >>= 5
return "".join(encoded)
def require_unique(values: list[str], label: str) -> None:
duplicates = sorted(
value for value in set(values) if values.count(value) > 1
)
if duplicates:
raise ValueError(f"duplicate {label}: {duplicates}")
def validate_source(source: LegacySystem) -> None:
role_codes = [role.key for role in source.roles]
user_codes = [user.id for user in source.users]
require_unique(role_codes, "role keys")
require_unique(user_codes, "user ids")
role_code_set = set(role_codes)
unknown_roles = sorted(
user.roleKey
for user in source.users
if user.roleKey not in role_code_set
)
if unknown_roles:
raise ValueError(f"users reference unknown roles: {unknown_roles}")
for role in source.roles:
require_unique(role.permissions, f"permissions of role {role.key}")
for permission_code in role.permissions:
if "." not in permission_code:
raise ValueError(
f"invalid permission code {permission_code!r}"
)
def load_source(path: Path) -> tuple[LegacySystem, str]:
raw = path.read_bytes()
source = LegacySystem.model_validate_json(raw)
validate_source(source)
return source, hashlib.sha256(raw).hexdigest()
def set_changed(instance: Any, values: dict[str, Any]) -> bool:
changed = False
for attribute, value in values.items():
if getattr(instance, attribute) != value:
setattr(instance, attribute, value)
changed = True
return changed
def new_stats() -> dict[str, int]:
return {
"roles_inserted": 0,
"roles_updated": 0,
"permissions_inserted": 0,
"permissions_updated": 0,
"role_permissions_inserted": 0,
"users_inserted": 0,
"users_updated": 0,
}
async def migrate_roles(
session: AsyncSession,
source: LegacySystem,
stats: dict[str, int],
) -> dict[str, str]:
existing = {
item.role_code: item
for item in (
await session.scalars(select(Roles).order_by(Roles.role_code))
).all()
}
role_ids: dict[str, str] = {}
for legacy in source.roles:
values = {
"role_name": legacy.name,
"role_scope": "platform",
"is_builtin": 1,
"description": legacy.description,
}
role = existing.get(legacy.key)
if role is None:
role = Roles(
role_id=deterministic_legacy_ulid("role", legacy.key),
role_code=legacy.key,
**values,
)
session.add(role)
stats["roles_inserted"] += 1
elif set_changed(role, values):
stats["roles_updated"] += 1
role_ids[legacy.key] = role.role_id
return role_ids
async def migrate_permissions(
session: AsyncSession,
source: LegacySystem,
stats: dict[str, int],
) -> dict[str, str]:
permission_codes = sorted(
{
permission_code
for role in source.roles
for permission_code in role.permissions
}
)
existing = {
item.permission_code: item
for item in (
await session.scalars(
select(Permissions).order_by(Permissions.permission_code)
)
).all()
}
permission_ids: dict[str, str] = {}
for permission_code in permission_codes:
values = {
"permission_name": PERMISSION_NAMES.get(
permission_code, permission_code
),
"module_code": permission_code.split(".", 1)[0],
"description": f"由旧版 system.json 迁移：{permission_code}",
}
permission = existing.get(permission_code)
if permission is None:
permission = Permissions(
permission_id=deterministic_legacy_ulid(
"permission", permission_code
),
permission_code=permission_code,
**values,
)
session.add(permission)
stats["permissions_inserted"] += 1
elif set_changed(permission, values):
stats["permissions_updated"] += 1
permission_ids[permission_code] = permission.permission_id
return permission_ids
async def migrate_role_permissions(
session: AsyncSession,
source: LegacySystem,
role_ids: dict[str, str],
permission_ids: dict[str, str],
stats: dict[str, int],
) -> None:
existing = set(
(
await session.execute(
select(
RolePermissions.role_id,
RolePermissions.permission_id,
)
)
).all()
)
for role in source.roles:
for permission_code in role.permissions:
pair = (
role_ids[role.key],
permission_ids[permission_code],
)
if pair not in existing:
session.add(
RolePermissions(
role_id=pair[0],
permission_id=pair[1],
)
)
existing.add(pair)
stats["role_permissions_inserted"] += 1
async def migrate_users(
session: AsyncSession,
source: LegacySystem,
role_ids: dict[str, str],
stats: dict[str, int],
) -> dict[str, str]:
existing = {
item.username: item
for item in (
await session.scalars(select(Users).order_by(Users.username))
).all()
}
user_ids: dict[str, str] = {}
for legacy in source.users:
values = {
"display_name": legacy.name,
"platform_role_id": role_ids[legacy.roleKey],
"avatar_uri": (
f"initial://{quote(legacy.avatar)}"
if legacy.avatar
else None
),
}
user = existing.get(legacy.id)
if user is None:
user = Users(
user_id=deterministic_legacy_ulid("user", legacy.id),
username=legacy.id,
password_hash=LEGACY_PASSWORD_HASH,
status="active",
**values,
)
session.add(user)
stats["users_inserted"] += 1
elif set_changed(user, values):
stats["users_updated"] += 1
user_ids[legacy.id] = user.user_id
return user_ids
async def run_migration(
database_url: str,
source: LegacySystem,
*,
apply_changes: bool,
) -> dict[str, int]:
engine = create_database_engine(database_url)
factory = create_session_factory(engine)
stats = new_stats()
try:
async with factory() as session:
try:
role_ids = await migrate_roles(session, source, stats)
permission_ids = await migrate_permissions(
session, source, stats
)
await migrate_role_permissions(
session,
source,
role_ids,
permission_ids,
stats,
)
await migrate_users(
session, source, role_ids, stats
)
if apply_changes:
await session.commit()
else:
await session.rollback()
except Exception:
await session.rollback()
raise
finally:
await engine.dispose()
return stats
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Migrate legacy system.json records into MySQL."
)
parser.add_argument(
"--source",
required=True,
type=Path,
help="Path to the legacy platform_data/system.json file.",
)
parser.add_argument(
"--apply",
action="store_true",
help="Commit changes. Without this flag the transaction is rolled back.",
)
return parser.parse_args()
async def async_main() -> None:
args = parse_args()
source_path = args.source.resolve(strict=True)
source, source_sha256 = load_source(source_path)
database_url = os.environ["DATABASE_URL"]
stats = await run_migration(
database_url,
source,
apply_changes=args.apply,
)
result = {
"mode": "apply" if args.apply else "dry-run",
"source": str(source_path),
"source_sha256": source_sha256,
"source_counts": {
"roles": len(source.roles),
"permissions": len(
{
permission
for role in source.roles
for permission in role.permissions
}
),
"role_permissions": sum(
len(role.permissions) for role in source.roles
),
"users": len(source.users),
},
"changes": stats,
}
print(json.dumps(result, ensure_ascii=False, indent=2))
def main() -> None:
asyncio.run(async_main())
if __name__ == "__main__":
main()
migrations/versions/e1f2a3b4c5d6_rebuild_baseline.py
+11 -23
View File
@@ -36,28 +36,19 @@ DEVELOPER_ROLE_ID = "0000000000000000000000000B"
PERMISSIONS: list[tuple[str, str, str]] = [
# (permission_code, permission_name, module_code)
("dashboard.view", "查看工作台", "dashboard"),
("script.build", "构建脚本", "script"),
("script.public.manage", "管理公共脚本", "script"),
("schedule.own", "管理本人调度", "schedule"),
("schedule.all", "管理全部调度", "schedule"),
("experiment.own", "管理本人实验", "experiment"),
("experiment.all", "管理全部实验", "experiment"),
("resource.personal", "管理个人资源", "resource"),
("resource.public.upload", "上传公共资源", "resource"),
("resource.public.manage", "管理公共资源", "resource"),
("system.view", "查看系统管理", "system"),
("system.manage", "管理系统配置", "system"),
("dashboard:view", "查看工作台", "dashboard"),
("script:view", "查看构建脚本", "script"),
("schedule:view", "查看调度配置", "schedule"),
("system:view", "查看系统管理", "system"),
("system:user:view", "查看用户管理", "system"),
("system:project:view", "查看项目管理", "system"),
]
# developer gets *.own + personal resource only; no system.*, no *.all.
# developer gets menu-view permissions only; no system:* (admin-only).
DEVELOPER_PERMISSION_CODES: list[str] = [
"dashboard.view",
"script.build",
"script.public.manage",
"schedule.own",
"experiment.own",
"resource.personal",
"dashboard:view",
"script:view",
"schedule:view",
]
ADMIN_PERMISSION_CODES: list[str] = [code for code, _, _ in PERMISSIONS]
@@ -334,7 +325,6 @@ def upgrade() -> None:
sa.Column('created_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3)'), nullable=False),
sa.Column('updated_at', mysql.DATETIME(fsp=3), server_default=sa.text('CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)'), nullable=False),
sa.Column('owner_user_id', mysql.CHAR(length=26), nullable=True),
sa.Column('parent_object_id', mysql.CHAR(length=26), nullable=True),
sa.Column('relative_path', sa.String(length=1024), nullable=True, comment='Workspace 相对路径'),
sa.Column('path_hash', sa.BINARY(length=32), nullable=True, comment='SHA-256(relative_path)，由应用写入'),
sa.Column('bucket_name', sa.String(length=128), nullable=True),
@@ -349,12 +339,11 @@ def upgrade() -> None:
sa.Column('deleted_at', mysql.DATETIME(fsp=3), nullable=True),
sa.Column('trash_key', sa.String(length=1100), nullable=True, comment="Path inside the trash bucket where the soft-deleted bytes live. Format: '{source_bucket}/{object_key}' so a restore is a same-key copy back to the source bucket. NULL while the row is still available."),
sa.PrimaryKeyConstraint('storage_object_id'),
comment='Workspace 文件和 RustFS 对象的统一元数据'
comment='Workspace 文件和 RustFS 对象的统一元数据;目录树走 materialized path (relative_path),不要 join 邻接表列——已删除。'
)
op.create_index('fk_storage_created_by', 'storage_objects', ['created_by'], unique=False)
op.create_index('idx_storage_content_hash', 'storage_objects', ['content_hash'], unique=False)
op.create_index('idx_storage_owner', 'storage_objects', ['owner_user_id', 'object_status'], unique=False)
op.create_index('idx_storage_parent', 'storage_objects', ['parent_object_id'], unique=False)
op.create_index('idx_storage_workspace_path', 'storage_objects', ['workspace_id', 'storage_backend', 'path_hash'], unique=False)
op.create_index('idx_storage_workspace_usage', 'storage_objects', ['workspace_id', 'usage_type', 'object_status'], unique=False)
op.create_index('uk_storage_bucket_key_active', 'storage_objects', ['storage_backend', 'bucket_name', 'object_key_hash_active'], unique=True)
@@ -693,7 +682,6 @@ def downgrade() -> None:
op.drop_index('idx_storage_workspace_usage', table_name='storage_objects')
op.drop_index('idx_storage_workspace_relative_path', table_name='storage_objects')
op.drop_index('idx_storage_workspace_path', table_name='storage_objects')
op.drop_index('idx_storage_parent', table_name='storage_objects')
op.drop_index('idx_storage_owner', table_name='storage_objects')
op.drop_index('idx_storage_content_hash', table_name='storage_objects')
op.drop_index('fk_storage_created_by', table_name='storage_objects')
migrations/versions/f7a8b9c0d1e2_drop_storage_parent.py
-56
View File
@@ -1,56 +0,0 @@
"""drop dead storage_objects.parent_object_id column and idx_storage_parent
Revision ID: f7a8b9c0d1e2
Revises: e1f2a3b4c5d6
Create Date: 2026-08-21
Removes a never-written column and its orphaned index. Tree structure is
maintained entirely via relative_path (materialized path); see
backend/src/backend/scripts.py (list_workspace_tree / list_workspace_directories).
MySQL 8.0 does not support DROP INDEX IF EXISTS / DROP COLUMN IF EXISTS,
so the calls below are unconditional.
"""
from collections.abc import Sequence
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import mysql
# revision identifiers, used by Alembic.
revision: str = "f7a8b9c0d1e2"
down_revision: str | Sequence[str] | None = "e1f2a3b4c5d6"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def upgrade() -> None:
"""Drop the dead index, then the dead column."""
op.drop_index("idx_storage_parent", table_name="storage_objects")
op.drop_column("storage_objects", "parent_object_id")
# Refresh the table comment so the warning reaches the DB, not just the ORM.
op.execute(
"ALTER TABLE storage_objects "
"COMMENT = 'Workspace 文件和 RustFS 对象的统一元数据;"
"目录树走 materialized path (relative_path),"
"不要 join 邻接表列——已删除。'"
)
def downgrade() -> None:
"""Recreate the column and index for rollback."""
op.add_column(
"storage_objects",
sa.Column("parent_object_id", mysql.CHAR(length=26), nullable=True),
)
op.create_index(
"idx_storage_parent",
"storage_objects",
["parent_object_id"],
unique=False,
)
op.execute(
"ALTER TABLE storage_objects "
"COMMENT = 'Workspace 文件和 RustFS 对象的统一元数据'"
)