Prefect Work Pool Configuration

Overview

The blog-data-pool Prefect ECS work pool is used by the blog_data pipeline for all scheduled and ad‑hoc runs. It must exist and have a correct ECS base job template or deployments will fail.

Today the work pool is created and kept up to date by the Prefect deployer ECS task, not by Terraform:

CircleCI runs the deploy-to-prefect job.
That job launches an internal ECS task that runs deploy_blog_data.py from this repository (infra/platform/docker/prefect-deployer/deploy_blog_data.py).
The script calls prefect work-pool inspect blog-data-pool and, if missing, creates it with the canonical ECS base job template embedded in the script.

For normal usage you do not need to edit the work pool manually. Treat the deployer script as the source of truth for the base job template. Changes should be made in code and rolled out via CI.

Required configuration (base job template)

Regardless of how the pool is created, the base job template must include:

Execution role ARN – allows ECS to pull images from ECR and access secrets.
Task role ARN – allows the running task to access AWS services (S3, Secrets Manager, etc.).
Security groups – network security configuration for the tasks.
Subnets – private subnets for task execution.
Cluster name – the ECS cluster to run tasks in.
Image variable – required image variable so deployments can choose the Docker image tag.

The values for these fields come from the production Terraform stack in this repository. See infra/platform/infra/envs/prod (and its outputs) for the up-to-date values.

Manual configuration (rarely needed)

⚠️ Warning: In normal operation the work pool is managed by the Prefect deployer script. Manual changes should only be used for debugging or one-off recovery (for example, in a dev environment). In production, any manual changes may be overwritten the next time the deployer runs.

For reference only, the sections below show how to configure the work pool by hand so you can compare the values against what the deployer is applying.

Method 1: Prefect UI

Navigate to https://pipelines.rocketclub.online
Go to Work Pools → blog-data-pool
Click Edit

Under Base Job Template, configure:

{
  "job_configuration": {
    "cluster": "arn:aws:ecs:eu-west-2:421115711209:cluster/rocket-club-prod-ecs",
    "task_definition": {
      "executionRoleArn": "arn:aws:iam::421115711209:role/blog-data-ecs-task-execution-role",
      "taskRoleArn": "arn:aws:iam::421115711209:role/blog-data-ecs-task-role",
      "networkMode": "awsvpc",
      "requiresCompatibilities": ["FARGATE"],
      "containerDefinitions": [
        {
          "name": "prefect",
          "image": "{{ image }}"
        }
      ]
    },
    "task_run_request": {
      "cluster": "arn:aws:ecs:eu-west-2:421115711209:cluster/rocket-club-prod-ecs",
      "launchType": "FARGATE"
    },
    "networkConfiguration": {
      "awsvpcConfiguration": {
        "subnets": ["subnet-087582825f3eeb00f", "subnet-0554b8e00e684b94a"],
        "securityGroups": ["sg-0c6cdbc8071f0052d"],
        "assignPublicIp": "DISABLED"
      }
    },
    "configure_cloudwatch_logs": true
  },
  "variables": {
    "type": "object",
    "properties": {
      "image": {
        "type": "string",
        "title": "Image",
        "description": "Docker image to use for flow execution"
      },
      "cluster": {
        "type": "string",
        "default": "arn:aws:ecs:eu-west-2:421115711209:cluster/rocket-club-prod-ecs"
      },
      "aws_credentials": {
        "type": "object",
        "default": {
          "region_name": "eu-west-2"
        }
      },
      "configure_cloudwatch_logs": {
        "type": "boolean",
        "default": true
      }
    },
    "required": ["image"]
  }
}

Click Save

Method 2: Prefect CLI

# Set Prefect API connection
export PREFECT_API_URL="https://pipelines.rocketclub.online/api"

# Update work pool with base job template
prefect work-pool update blog-data-pool \
  --base-job-template '{
    "job_configuration": {
      "cluster": "arn:aws:ecs:eu-west-2:421115711209:cluster/rocket-club-prod-ecs",
      "task_definition": {
        "executionRoleArn": "arn:aws:iam::421115711209:role/prod-ecsTaskExecutionRole",
        "taskRoleArn": "arn:aws:iam::421115711209:role/prod-ecsTaskRole",
        "networkMode": "awsvpc",
        "requiresCompatibilities": ["FARGATE"],
        "containerDefinitions": [{
          "name": "prefect",
          "image": "{{ image }}"
        }]
      },
      "task_run_request": {
        "cluster": "arn:aws:ecs:eu-west-2:421115711209:cluster/rocket-club-prod-ecs",
        "launchType": "FARGATE"
      },
      "networkConfiguration": {
        "awsvpcConfiguration": {
          "subnets": ["subnet-087582825f3eeb00f", "subnet-0554b8e00e684b94a"],
          "securityGroups": ["sg-0c6cdbc8071f0052d"],
          "assignPublicIp": "DISABLED"
        }
      },
      "configure_cloudwatch_logs": true
    },
    "variables": {
      "type": "object",
      "properties": {
        "image": {
          "type": "string",
          "title": "Image",
          "description": "Docker image to use for flow execution"
        },
        "cluster": {
          "type": "string",
          "default": "arn:aws:ecs:eu-west-2:421115711209:cluster/rocket-club-prod-ecs"
        },
        "aws_credentials": {
          "type": "object",
          "default": {
            "region_name": "eu-west-2"
          }
        },
        "configure_cloudwatch_logs": {
          "type": "boolean",
          "default": true
        }
      },
      "required": ["image"]
    }
  }'

Method 3: Create a new work pool

If the work pool doesn't exist yet and you need to create it by hand (for example, in a non-production environment), you can use the Prefect CLI directly:

export PREFECT_API_URL="https://pipelines.rocketclub.online/api"

prefect work-pool create blog-data-pool \
  --type ecs \
  --base-job-template '{
    "job_configuration": {
      "cluster": "arn:aws:ecs:eu-west-2:421115711209:cluster/rocket-club-prod-ecs",
      "task_definition": {
        "executionRoleArn": "arn:aws:iam::421115711209:role/prod-ecsTaskExecutionRole",
        "taskRoleArn": "arn:aws:iam::421115711209:role/prod-ecsTaskRole",
        "networkMode": "awsvpc",
        "requiresCompatibilities": ["FARGATE"],
        "containerDefinitions": [{
          "name": "prefect",
          "image": "{{ image }}"
        }]
      },
      "task_run_request": {
        "cluster": "arn:aws:ecs:eu-west-2:421115711209:cluster/rocket-club-prod-ecs",
        "launchType": "FARGATE"
      },
      "networkConfiguration": {
        "awsvpcConfiguration": {
          "subnets": ["subnet-087582825f3eeb00f", "subnet-0554b8e00e684b94a"],
          "securityGroups": ["sg-0c6cdbc8071f0052d"],
          "assignPublicIp": "DISABLED"
        }
      },
      "configure_cloudwatch_logs": true
    },
    "variables": {
      "type": "object",
      "properties": {
        "image": {
          "type": "string",
          "title": "Image",
          "description": "Docker image to use for flow execution"
        },
        "cluster": {
          "type": "string",
          "default": "arn:aws:ecs:eu-west-2:421115711209:cluster/rocket-club-prod-ecs"
        },
        "aws_credentials": {
          "type": "object",
          "default": {
            "region_name": "eu-west-2"
          }
        },
        "configure_cloudwatch_logs": {
          "type": "boolean",
          "default": true
        }
      },
      "required": ["image"]
    }
  }'

Verification

Check Work Pool Configuration

export PREFECT_API_URL="https://pipelines.rocketclub.online/api"

prefect work-pool inspect blog-data-pool

Look for the base job template to confirm it includes:

executionRoleArn
taskRoleArn
networkConfiguration with subnets and security groups
containerDefinitions with image: "{{ image }}"
variables.properties.image with required: ["image"]

Test Flow Run

Trigger a test flow run:

prefect deployment run data-extraction/data-extraction

Monitor the logs - you should no longer see the "Fargate requires task definition to have execution role ARN" error.

Security Groups and Network Configuration

ECS Tasks Security Group

The sg-0c6cdbc8071f0052d security group allows:

Outbound HTTPS (443) - For pulling ECR images, accessing AWS APIs
Outbound DNS (53) - For DNS resolution
Outbound Neo4j (7687) - For database access
Inbound from Prefect Server - For communication with Prefect

Private Subnets

Tasks run in private subnets with NAT gateway for outbound internet access:

subnet-087582825f3eeb00f (eu-west-2a)
subnet-0554b8e00e684b94a (eu-west-2b)

IAM Roles

Execution Role (`blog-data-ecs-task-execution-role`)

Allows ECS to:

Pull images from ECR
Write logs to CloudWatch
Read secrets from Secrets Manager

Task Role (`blog-data-ecs-task-role`)

Allows the running task to:

Assume the pipeline role for S3 access
Access other AWS services as needed

Troubleshooting

Error: "Fargate requires task definition to have execution role ARN"

Cause: Work pool base job template is missing executionRoleArn

Fix: Update work pool configuration using Method 1 or 2 above

Error: "Task failed to start"

Cause: Network configuration or security group issues

Fix: Verify subnets and security groups in work pool match Terraform outputs

Error: "Unable to pull image"

Cause: Execution role lacks ECR permissions

Fix: Verify execution role has ECR pull permissions (should be configured in Terraform)

Infrastructure Overview (source of truth for VPC, ECS, and Terraform)
Deployment Checklist