-
-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add scaleway provider #56
Open
raskyld
wants to merge
10
commits into
woodpecker-ci:main
Choose a base branch
from
scaleway:feat/alpha-support
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+572
−0
Open
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
2f6c6f4
feat: WIP: Scaleway provider
raskyld 80b3f1e
clean: config scheme
raskyld f026317
feat: add provider logic
raskyld ae2c686
feat: add scw to setupProvider()
raskyld 4cae2e4
fix(chore): fix typo
raskyld 4ec4a7a
fix: typo & halt before delete
raskyld 37ff856
fix: bad volume params
raskyld 1bad108
address review
raskyld a9b3feb
bump deps after rebase
raskyld edb6c1a
some lint and cleaning
raskyld File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| package scaleway | ||
|
|
||
| import ( | ||
| "errors" | ||
|
|
||
| "github.com/cenkalti/backoff/v4" | ||
| "github.com/scaleway/scaleway-sdk-go/api/instance/v1" | ||
| "github.com/scaleway/scaleway-sdk-go/scw" | ||
| ) | ||
|
|
||
| // Config is the Scaleway Provider specific configuration | ||
| // | ||
| // This is decoupled from the CLI interface for future-proofing reasons. | ||
| // Please, see ProviderFlags for information on how to configure the provider from the | ||
| // CLI or environment variables. | ||
| type Config struct { | ||
| // ApiToken of Scaleway IAM | ||
| // | ||
| // Creating a standalone IAM Applications is recommended to segregate | ||
| // permissions. | ||
| SecretKey string `json:"secret_key"` | ||
| AccessKey string `json:"access_key"` | ||
| DefaultProjectID string `json:"default_project_id"` | ||
| ClientRetry backoff.BackOff | ||
| InstancePool map[string]InstancePool `json:"instance_pool"` | ||
| } | ||
|
|
||
| // Locality defines a geographical area | ||
| // | ||
| // Scaleway Cloud has multiple Region that are made of several Zones. | ||
| // Exactly one of Zones or Region SHOULD be set, | ||
| // if both are set, use Region and ignore Zones | ||
| type Locality struct { | ||
| Zones []scw.Zone `json:"zones,omitempty"` | ||
| Region *scw.Region `json:"region,omitempty"` | ||
| } | ||
|
|
||
| // InstancePool is used as a template to spawn your instances | ||
| type InstancePool struct { | ||
| // Locality where your instances should live | ||
| // The Provider will try to spread your | ||
| // instances evenly among Locality.Zones if possible | ||
| Locality Locality `json:"locality"` | ||
| // ProjectID where resources should be applied | ||
| ProjectID *string `json:"project_id,omitempty"` | ||
| // Prefix is added before each instance name | ||
| Prefix string `json:"prefix"` | ||
| // Tags added to the placement group and its instances | ||
| Tags []string `json:"tags"` | ||
| // DynamicIPRequired: define if a dynamic IPv4 is required for the Instance. | ||
| DynamicIPRequired *bool `json:"dynamic_ip_required,omitempty"` | ||
| // RoutedIPEnabled: if true, configure the Instance, so it uses the new routed IP mode. | ||
| RoutedIPEnabled *bool `json:"routed_ip_enabled,omitempty"` | ||
| // CommercialType: define the Instance commercial type (i.e. GP1-S). | ||
| CommercialType string `json:"commercial_type,omitempty"` | ||
| // Image: instance image ID or label. | ||
| Image string `json:"image,omitempty"` | ||
| // EnableIPv6: true if IPv6 is enabled on the server. | ||
| EnableIPv6 bool `json:"enable_ipv6,omitempty"` | ||
| // PublicIPs to attach to your instance indexed per instance.IPType | ||
| PublicIPs map[instance.IPType]int `json:"public_ips,omitempty"` | ||
| // SecurityGroups to use per zone | ||
| SecurityGroups map[scw.Zone]string `json:"security_groups,omitempty"` | ||
| // Storage of the block storage associated with your Instances | ||
| // It should be a multiple of 512 bytes, in future version we could give | ||
| // more customisation over the volumes used by the agents | ||
| Storage scw.Size `json:"storage"` | ||
| } | ||
|
|
||
| func (l Locality) ResolveZones() ([]scw.Zone, error) { | ||
| if l.Region != nil { | ||
| if !l.Region.Exists() { | ||
| return nil, errors.New("you specified an invalid region: " + l.Region.String()) | ||
| } | ||
|
|
||
| return l.Region.GetZones(), nil | ||
| } | ||
|
|
||
| zones := l.Zones | ||
| if len(zones) == 0 { | ||
| return nil, errors.New("you need to specify a valid locality") | ||
| } | ||
|
|
||
| for _, zone := range zones { | ||
| if !zone.Exists() { | ||
| return nil, errors.New("you specified a non-existing zone: " + zone.String()) | ||
| } | ||
| } | ||
|
|
||
| return zones, nil | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| // Package scaleway implements a way to use the Scaleway Cloud Provider for your | ||
xoxys marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| // Woodpecker CIs. | ||
| // | ||
| // This package contains subpackage per Scaleway Instance API version. | ||
| // | ||
| // # Limitations | ||
| // | ||
| // - As of now, we can only deploy instances in single-zones. | ||
| // | ||
| // Authors: | ||
| // - Enzo "raskyld" Nocera <[email protected]> [@[email protected]] | ||
| // | ||
| // [@[email protected]]: https://social.vivaldi.net/@raskyld | ||
| package scaleway | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| package scaleway | ||
|
|
||
| import ( | ||
| "log/slog" | ||
|
|
||
| "github.com/scaleway/scaleway-sdk-go/api/instance/v1" | ||
| "github.com/scaleway/scaleway-sdk-go/scw" | ||
| ) | ||
|
|
||
| type InstanceAlreadyExistsError struct { | ||
| inst *instance.Server | ||
| } | ||
|
|
||
| type InstanceDoesNotExists struct { | ||
| InstanceName string | ||
| Project string | ||
| Zones []scw.Zone | ||
| } | ||
|
|
||
| func (i InstanceAlreadyExistsError) Error() string { | ||
| return "instance already exists" | ||
| } | ||
|
|
||
| func (i InstanceAlreadyExistsError) LogValue() slog.Value { | ||
| return slog.GroupValue(slog.String("err", i.Error()), | ||
| slog.Group("instance", slog.String("id", i.inst.ID), slog.String("name", i.inst.Name), | ||
| slog.String("zone", i.inst.Zone.String()), slog.String("project", i.inst.Project))) | ||
| } | ||
|
|
||
| func (i InstanceDoesNotExists) Error() string { | ||
| return "instance does not exist" | ||
| } | ||
|
|
||
| func (i InstanceDoesNotExists) LogValue() slog.Value { | ||
| zones := make([]string, 0, len(i.Zones)) | ||
| for _, zone := range i.Zones { | ||
| zones = append(zones, zone.String()) | ||
| } | ||
|
|
||
| return slog.GroupValue(slog.String("name", i.InstanceName), slog.String("project", i.Project), slog.Any("zones", zones)) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,147 @@ | ||
| package scaleway | ||
|
|
||
| import ( | ||
| "errors" | ||
| "os" | ||
|
|
||
| "github.com/scaleway/scaleway-sdk-go/scw" | ||
| "github.com/urfave/cli/v2" | ||
| ) | ||
|
|
||
| const ( | ||
| DefaultPool = "default" | ||
| DefaultAgentStorageGB = 25 | ||
|
|
||
| category = "Scaleway" | ||
| flagPrefix = "scw" | ||
| envPrefix = "WOODPECKER_SCW" | ||
| ) | ||
|
|
||
| var ProviderFlags = []cli.Flag{ | ||
| &cli.StringFlag{ | ||
| Name: flagPrefix + "-access-key", | ||
| Usage: "Scaleway IAM API Token Access Key", | ||
| EnvVars: []string{envPrefix + "_ACCESS_KEY"}, | ||
| // NB(raskyld): We should recommend the usage of file-system to users | ||
| // Most container runtimes support mounting secrets into the fs | ||
| // natively. | ||
| FilePath: os.Getenv(envPrefix + "_ACCESS_KEY_FILE"), | ||
| Category: category, | ||
| }, | ||
| &cli.StringFlag{ | ||
| Name: flagPrefix + "-secret-key", | ||
| Usage: "Scaleway IAM API Token Secret Key", | ||
| EnvVars: []string{envPrefix + "_SECRET_KEY"}, | ||
| // NB(raskyld): We should recommend the usage of file-system to users | ||
| // Most container runtimes support mounting secrets into the fs | ||
| // natively. | ||
| FilePath: os.Getenv(envPrefix + "_SECRET_KEY_FILE"), | ||
| Category: category, | ||
| }, | ||
| // TODO(raskyld): implement multi-AZ | ||
| &cli.StringFlag{ | ||
| Name: flagPrefix + "-zone", | ||
| Usage: "Scaleway Zone where to spawn instances", | ||
| EnvVars: []string{envPrefix + "_ZONE"}, | ||
| Category: category, | ||
| Value: scw.ZoneFrPar2.String(), | ||
| }, | ||
| &cli.StringFlag{ | ||
| Name: flagPrefix + "-instance-type", | ||
| Usage: "Scaleway Instance type to spawn", | ||
| EnvVars: []string{envPrefix + "_INSTANCE_TYPE"}, | ||
| Category: category, | ||
| }, | ||
| &cli.StringSliceFlag{ | ||
| Name: flagPrefix + "-tags", | ||
| Usage: "Comma separated list of tags to uniquely identify the instances spawned", | ||
| EnvVars: []string{envPrefix + "_TAGS"}, | ||
| Category: category, | ||
| }, | ||
| &cli.StringFlag{ | ||
| Name: flagPrefix + "-project", | ||
| Usage: "Scaleway Project ID in which to spawn the instances", | ||
| EnvVars: []string{envPrefix + "_PROJECT"}, | ||
| Category: category, | ||
| }, | ||
| &cli.StringFlag{ | ||
| Name: flagPrefix + "-prefix", | ||
| Usage: "Prefix prepended before any Scaleway resource name", | ||
| EnvVars: []string{envPrefix + "_PREFIX"}, | ||
| Category: category, | ||
| Value: "wip-woodpecker-ci-autoscaler", | ||
| }, | ||
| &cli.BoolFlag{ | ||
| Name: flagPrefix + "-enable-ipv6", | ||
| Usage: "Enable IPv6 for the instances", | ||
| EnvVars: []string{envPrefix + "_ENABLE_IPV6"}, | ||
| Category: category, | ||
| }, | ||
| &cli.StringFlag{ | ||
| Name: flagPrefix + "-image", | ||
| Usage: "The base image for your instance", | ||
| EnvVars: []string{envPrefix + "_IMAGE"}, | ||
| Category: category, | ||
| Value: "ubuntu_jammy", | ||
| }, | ||
| &cli.Uint64Flag{ | ||
| Name: flagPrefix + "-storage-size", | ||
| Usage: "How much storage to provision for your agents in GB", | ||
| EnvVars: []string{envPrefix + "_STORAGE_SIZE"}, | ||
| Category: category, | ||
| Value: DefaultAgentStorageGB, | ||
| }, | ||
| } | ||
|
|
||
| func FromCLI(c *cli.Context) (Config, error) { | ||
| if !c.IsSet(flagPrefix + "-instance-type") { | ||
| return Config{}, errors.New("you must specify an instance type") | ||
| } | ||
|
|
||
| if !c.IsSet(flagPrefix + "-tags") { | ||
| return Config{}, errors.New("you must specify tags to apply to your resources") | ||
| } | ||
|
|
||
| if !c.IsSet(flagPrefix + "-project") { | ||
| return Config{}, errors.New("you must specify in which project resources should be spawned") | ||
| } | ||
|
|
||
| if !c.IsSet(flagPrefix + "-secret-key") { | ||
| return Config{}, errors.New("you must specify a secret key") | ||
| } | ||
|
|
||
| if !c.IsSet(flagPrefix + "-access-key") { | ||
| return Config{}, errors.New("you must specify an access key") | ||
| } | ||
|
|
||
| zone := scw.Zone(c.String(flagPrefix + "-zone")) | ||
| if !zone.Exists() { | ||
| return Config{}, errors.New(zone.String() + " is not a valid zone") | ||
| } | ||
|
|
||
| cfg := Config{ | ||
| SecretKey: c.String(flagPrefix + "-secret-key"), | ||
| AccessKey: c.String(flagPrefix + "-access-key"), | ||
| DefaultProjectID: c.String(flagPrefix + "-project"), | ||
| } | ||
|
|
||
| cfg.InstancePool = map[string]InstancePool{ | ||
| DefaultPool: { | ||
| Locality: Locality{ | ||
| Zones: []scw.Zone{zone}, | ||
| }, | ||
| ProjectID: scw.StringPtr(c.String(flagPrefix + "-project")), | ||
| Prefix: c.String(flagPrefix + "-prefix"), | ||
| Tags: c.StringSlice(flagPrefix + "-tags"), | ||
| // We do not need stables IP for our JIT runners | ||
| DynamicIPRequired: scw.BoolPtr(true), | ||
| CommercialType: c.String(flagPrefix + "-instance-type"), | ||
| Image: c.String(flagPrefix + "-image"), | ||
| EnableIPv6: c.Bool(flagPrefix + "-enable-ipv6"), | ||
| //nolint:gomnd | ||
| Storage: scw.Size(c.Uint64(flagPrefix+"-storage-size") * 1e9), | ||
| }, | ||
| } | ||
|
|
||
| return cfg, nil | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was there a specific reason to use another structure to do the config setup for this provider? I would prefer to keep the setup for providers equal. Everything done by
scaleway.FromCLI(ctx)should be handled byscaleway.Newsimilar to e.g. https://github.com/woodpecker-ci/autoscaler/blob/main/providers/hetznercloud/provider.go#L43There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I initially decoupled the parsing from CLI and the config structure to allow using declarative file (e.g. a yaml to config the autoscaler) instead of the CLI. But it seems like we won't support configuration from file so I can remove that indeed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you have some time to change it that would be great. If not we can also merge the current state and change it later.