Terraform Prefect ECS Module

This module deploys a Prefect worker onto ECS Fargate using Terraform.

Requirements

You will need an AWS user with the following permissions. This will allow Terraform to create and manage the resources required to create the Prefect worker in ECS. Some example policies that provide these permissions are mentioned as a starting point, but we recommend providing more restricted access.

• ec2:CreateVpc (provided by AmazonVPCFullACcess policy)
• ecs:CreateCluster (provided by AmazonECS_FullAccess policy)
• secretsmanager:CreateSecret (provided by SecretsManagerReadWrite policy)
• iam:CreateRole (provided by IAMFullAcess)
• logs:createLogGroup (provided by CloudWatchLogsFullAccess policy)
• ecr:CreateRepository (provided by AmazonEC2ContainerRegistryFullAccess policy)

Next, you will need:

• The Terraform CLI
• The Prefect CLI
• Your Prefect account ID
• Your Prefect workspace ID
• Your Prefect API key

Usage

Configure Terraform

The following is an example directory structure to use this module:

.
├── main.tf
├── terraform.tfvars
└── variables.tf

// variables.tf
variable prefect_api_key {}

// terraform.tfvars
prefect_api_key = "pnu_xxxxx"

// main.tf

provider "aws" {
  region = "us-east-1"
}

module "prefect_ecs_worker" {
  source = "prefecthq/ecs-worker/prefect"

  name                  = "dev"

  vpc_id                = "vpc-acfc2092275244ca8"
  worker_subnets        = [
    "subnet-014aa5f348034e45b",
    "subnet-df23ae9eab1f49af9"
  ]

  prefect_api_key       = var.prefect_api_key
  prefect_account_id    = "6e02a1db-07de-4760-a15d-60d8fe0b04e1"
  prefect_workspace_id  = "54cdfc71-9f13-41ba-9492-e1cf24eed185"

  worker_work_pool_name = "my-ecs-pool"
}

See the Inputs section below for more options.

See the full example, which makes use of an AWS-provided module to create most of the AWS resources automatically.

You can run terraform init followed by terraform apply to create the resources.

Configure the Prefect work pool

Once terraform apply has completed successfully, within a few minutes you should see a new ECS work pool in the UI.

Click the three dots to open the context menu, and select Edit.

Provide the following values for the work pool:

Field	Required	Notes	Example
Execution role ARN	Yes	This is specified in the task definition resource, but is still needed in the work pool settings.	arn:aws:iam::123456789:role/prefect-worker-execution-role-<name>
VPC ID	Yes	Required when using the awsvpc network mode.	vpc-123abc456def
Cluster	No	If not set, uses the default cluster.	arn:aws:ecs:us-east-1:123456789:cluster/prefect-worker-<name>
Image	No	Image setting is retrieved from the deployment configuration, but a default can be provided here.	123456.dkr.ecr.us-east-1.amazonaws.com/<image_name>:latest
Task role ARN	No	Defaults to the task role on the service, but can be overridden here.	arn:aws:iam::123456789:role/prefect-worker-task-role-<name>

More details on these settings are available on the Edit page of the work pool.

This configuration can also be provided in the base job template. For more information, see work pools.

Additionally, work pools and the associated base job templates can be managed with the Prefect Terraform provider. See the work_pool resource documentation for more information.

Once complete, you will see a new work pool available in Prefect. You can then use this work pool for your deployments. See the deployments documentation for more information.

Reference

The Terraform docs below can be generated with the following command:

make docs

Further reading

• Prefect ECS guide
• Amazon ECS networking best practices
• Troubleshoot ECS pulling secrets

Requirements

Name	Version
aws	~> 5.0

Providers

Name	Version
aws	~> 5.0

Modules

No modules.

Resources

Name	Type
aws_cloudwatch_log_group.prefect_worker_log_group	resource
aws_ecs_cluster.prefect_worker_cluster	resource
aws_ecs_cluster_capacity_providers.prefect_worker_cluster_capacity_providers	resource
aws_ecs_service.prefect_worker_service	resource
aws_ecs_task_definition.prefect_worker_task_definition	resource
aws_iam_role.prefect_worker_execution_role	resource
aws_iam_role.prefect_worker_task_role	resource
aws_iam_role_policy.allow_create_log_group	resource
aws_iam_role_policy.prefect_worker_allow_ecs_task	resource
aws_iam_role_policy.ssm_allow_read_prefect_api_key	resource
aws_iam_role_policy_attachment.prefect_worker_ecs_policy	resource
aws_secretsmanager_secret.prefect_api_key	resource
aws_secretsmanager_secret_version.prefect_api_key_version	resource
aws_security_group.prefect_worker	resource
aws_security_group_rule.network_outbound	resource
aws_region.current	data source

Inputs

Name	Description	Type	Default	Required
name	Unique name for this worker deployment	string	n/a	yes
prefect_account_id	Prefect Cloud account ID	string	n/a	yes
prefect_api_key	Prefect Cloud API key	string	n/a	yes
prefect_workspace_id	Prefect Cloud workspace ID	string	n/a	yes
vpc_id	VPC ID in which to create all resources	string	n/a	yes
worker_subnets	Subnet(s) to use for the worker	list(string)	n/a	yes
worker_work_pool_name	Work pool that the worker should poll	string	n/a	yes
secrets_manager_recovery_in_days	Deletion delay for AWS Secrets Manager upon resource destruction	number	30	no
worker_cpu	CPU units to allocate to the worker	number	1024	no
worker_desired_count	Number of workers to run	number	1	no
worker_extra_pip_packages	Packages to install on the worker assuming image is based on prefecthq/prefect	string	"prefect-aws s3fs"	no
worker_image	Container image for the worker. This could be the name of an image in a public repo or an ECR ARN	string	"prefecthq/prefect:3-python3.11"	no
worker_log_retention_in_days	Number of days to retain worker logs	number	30	no
worker_memory	Memory units to allocate to the worker	number	2048	no
worker_task_role_arn	Optional task role ARN to pass to the worker. If not defined, a task role will be created	string	null	no
worker_type	Prefect worker type that gets passed into the Prefect worker start command	string	"ecs"	no

Outputs

Name	Description
prefect_worker_cluster_name	n/a
prefect_worker_execution_role_arn	n/a
prefect_worker_security_group	n/a
prefect_worker_service_id	n/a
prefect_worker_task_role_arn	n/a