Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: cloud only and subcontracting #843

Merged
merged 10 commits into from
Mar 3, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/test-when-pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ jobs:
pip install -r requirements.txt
pip install -r requirements-media.txt
pip install -r requirements-dev.txt
pip install -r requirements-dashboard.txt

- name: Run Tests (First Attempt)
id: test_first_try
Expand Down
64 changes: 64 additions & 0 deletions docs/插件化设计.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# 插件化设计

考虑到训练性能影响、包大小等因素,自[#780](https://github.com/SwanHubX/SwanLab/issues/780)起,swanlab将云端和本地版分开,在
`mode="cloud"`
时,不再运行本地版服务。本地版服务的主要将交由[SwanLab-Dashboard](https://github.com/SwanHubX/SwanLab-Dashboard)分包处理。

> NOTE: 由于维护精力问题,本地版服务进入维护模式,不再更新新功能,仅维护bug和适配新版本swanlab的接口变化。
>
> 好消息是,当[#780](https://github.com/SwanHubX/SwanLab/issues/780)
> 完成的同时,我们将提供私有化部署版,代替[SwanLab-Dashboard](https://github.com/SwanHubX/SwanLab-Dashboard)的绝大多数使用场景。

---

SwanLab-SDK可以被抽象为一个解码器,使用者通过传入不同的数据以及配置,完成不同可视化场景的预处理。
以此为基础,swanlab在处理的时候提供了不同的事件,通过事件回调(同步)的方式完成各种需求——因此,所谓的插件本质上是一个事件回调处理器。有关事件的详细信息,可参阅[回调函数](https://github.com/SwanHubX/SwanLab-Toolkit/wiki/第3部分:回调函数)

在swanlab源码当中,有一个继承自[SwanKitCallback](https://github.com/SwanHubX/SwanLab-Toolkit/blob/b914037d471628e2d3194d2d4bb4d9f3f3a7fb9c/swankit/callback/__init__.py#L17C7-L17C22)
的helper,用于聚合所有的插件回调函数。当某一个事件回调被触发时,helper会按照插件的传入次序依次**同步**调用插件的回调函数。

> 从上述设计可以看出我们不会再关心回调处理函数的返回值是什么,因为我们认为插件的回调函数是无状态的,不应该有返回值。

## 🙋 如何自定义一个插件(回调处理器)

swanlab一切的起点都是`swanlab.init`,此函数允许传入`callbacks`参数,用于向helper注册插件回调函数。我们约定所有的回调处理器都应该继承自
`SwanKitCallback`,类似下面这样:

```python
from swankit.callback import SwanKitCallback


class CustomCallback(SwanKitCallback):
def __str__(self):
# 用于在swanlab.init的时候打印插件的名称,并作为插件的唯一标识
return "CustomCallback"

def on_stop(self, error: str = None):
print("Experiment stopped.")
```

然后在`swanlab.init`的时候传入`callbacks`参数:

```python
import swanlab
from swankit.callback import SwanKitCallback


class CustomCallback(SwanKitCallback):
def __str__(self):
# 用于在swanlab.init的时候打印插件的名称,并作为插件的唯一标识
return "CustomCallback"

def on_stop(self, error: str = None):
print("Experiment stopped.")


swanlab.init(callback=CustomCallback())
```

这样,在实验结束后,swanlab会调用`CustomCallback`的`on_stop`方法,在终端打印`Experiment stopped`。

## ⚠️ 分包后的单元测试

尽管`swanboard`函数变成一个可选依赖,但是单元测试时依旧需要`swanboard`的支持。即运行单元测试之前,需确保
`pip install swanboard`
2 changes: 1 addition & 1 deletion docs/登录认证.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@
> [#792](https://github.com/SwanHubX/SwanLab/issues/792) [#797](https://github.com/SwanHubX/SwanLab/issues/797)

对于云端版而言,swanlab需要用户提供api key以完成登录认证,完成云端的鉴权。用户可以在swanlab的 [官网](https://swanlab.cn)
上注册账号,然后在[个人中心](https://swanlab.cn/space/~/settings)获取api key。本部分主要讲解sdk中登录认证的代码逻辑。
上注册账号,然后在[个人中心](https://swanlab.cn/space/~/settings)获取api key。
2 changes: 2 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ filename = "requirements.txt"

[tool.hatch.metadata.hooks.requirements_txt.optional-dependencies]
media = ["requirements-media.txt"] # pip install "swanlab[meida]"
dashboard = ["requirements-dashboard.txt"] # pip install "swanlab[dashboard]"

[tool.hatch.metadata.hooks.fancy-pypi-readme] # 动态设置readme
content-type = "text/markdown"
Expand Down Expand Up @@ -91,6 +92,7 @@ include = [
"/README.md", # 包含readme
"/requirements.txt", # 包含依赖
"/requirements-media.txt", # 包含可选依赖
"/requirements-dashboard.txt", # 包含可选依赖
]

[tool.hatch.build.targets.wheel]
Expand Down
1 change: 1 addition & 0 deletions requirements-dashboard.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
swanboard==0.1.7b1
1 change: 0 additions & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
swankit==0.1.3
swanboard==0.1.7b1
cos-python-sdk-v5
urllib3>=1.26.0
requests>=2.25.0
Expand Down
17 changes: 7 additions & 10 deletions swanlab/api/upload/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,12 @@
@Description:
上传相关接口
"""
from ..http import get_http, sync_error_handler
from .model import ColumnModel, MediaModel, ScalarModel, FileModel
from typing import List

from swanlab.error import ApiError
from swanlab.log import swanlog
from .model import ColumnModel, MediaModel, ScalarModel, FileModel
from ..http import get_http, sync_error_handler

house_url = '/house/metrics'

Expand All @@ -21,12 +22,7 @@ def create_data(metrics: List[dict], metrics_type: str) -> dict:
携带上传日志的指标信息
"""
http = get_http()
return {
"projectId": http.proj_id,
"experimentId": http.exp_id,
"type": metrics_type,
"metrics": metrics
}
return {"projectId": http.proj_id, "experimentId": http.exp_id, "type": metrics_type, "metrics": metrics}


@sync_error_handler
Expand Down Expand Up @@ -71,7 +67,7 @@ def upload_scalar_metrics(scalar_metrics: List[ScalarModel]):
_valid_files = {
'config.yaml': ['config', 'yaml'],
'requirements.txt': ['requirements', 'txt'],
'swanlab-metadata.json': ['metadata', 'json']
'swanlab-metadata.json': ['metadata', 'json'],
}
"""
支持上传的文件列表,filename: key
Expand Down Expand Up @@ -117,5 +113,6 @@ def upload_column(columns: List[ColumnModel]):
"upload_column",
"ScalarModel",
"MediaModel",
"ColumnModel"
"ColumnModel",
"FileModel",
]
19 changes: 13 additions & 6 deletions swanlab/cli/commands/dashboard/watch.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,14 @@
@Description:
watch命令
"""
from swanlab.env import get_swanlog_dir, SwanLabEnv
from swanlab.log import swanlog
from swanboard import SwanBoardRun
import click
import os
import sys
import socket
import sys

import click

from swanlab.env import get_swanlog_dir, SwanLabEnv
from swanlab.log import swanlog


def get_free_port(address='0.0.0.0', default_port=5092) -> int:
Expand Down Expand Up @@ -88,7 +89,7 @@ def get_free_port(address='0.0.0.0', default_port=5092) -> int:
readable=True,
),
help="Specify the folder to store Swanlog, which is by default the folder where Swanlab Watch is run."
"The option will be deprecated in the future, you can just use `swanlab watch <LOG PATH>` to specify the path."
"The option will be deprecated in the future, you can just use `swanlab watch <LOG PATH>` to specify the path.",
)
@click.option(
"--log-level",
Expand All @@ -101,6 +102,12 @@ def watch(path: str, host: str, port: int, logdir: str, log_level: str):
"""
Run this commands to turn on the swanlab service.
"""
try:
# noinspection PyPackageRequirements
from swanboard import SwanBoardRun
except ModuleNotFoundError:
click.echo("Please install the swanboard package: `pip install swanlab[dashboard]`")
return sys.exit(1)
swanlog.level = log_level
# ----- 校验path,path如果被输入,已经由上层校验已存在,可读,是一个文件夹 -----
if logdir is not None:
Expand Down
Loading