CircleCI Setup Guide for Prefect Deployment

Overview

This guide walks through setting up CircleCI to deploy your Prefect flows to your self-hosted Prefect server through the param-gated deploy-data-platform pipeline — the migrated successor to the external blog_data repo's deploy pipeline (see ADR-0005). The pipeline runs only when its pipeline parameter is set, not automatically on push.

Architecture

Trigger deploy-data-platform pipeline parameter
    ↓
CircleCI deploy-data-platform workflow
    ↓
1. Build data-platform Docker image → Push to ECR (blog-data:<SHA>)
    ↓
2. Build Prefect deployer image + terraform-prod apply (roll the deployer task definition)
    ↓
3. deploy-to-prefect → run the Prefect deployer ECS task (deploys from baked-in data/platform)
    ↓
Prefect Server (https://pipelines.rocketclub.online)
    ↓
4. Redeploy the four blog-data-* Lambdas
    ↓
ECS Worker Pool → Run Flow Tasks

Prerequisites

Before setting up CircleCI, ensure you have:

Yes Terraform Infrastructure Deployed
- Prefect server running at https://pipelines.rocketclub.online
- ECR repository created (blog-data)
- ECS cluster and work pool configured
- IAM roles and permissions set up
Yes Prefect Server Accessible
- Can access Prefect UI at https://pipelines.rocketclub.online
- Have admin credentials (username:password)
Yes CircleCI Project Connected
- GitHub repository connected to CircleCI
- Project visible in CircleCI dashboard

Step 1: CircleCI and Prefect Prerequisites

Before configuring anything in this repo, ensure that:

Terraform for the prod environment has been applied from this repository (infra/platform/infra/envs/prod, typically via the CircleCI terraform-prod workflow).
The shared CircleCI context aws-blog-infra-prod exists and is attached to the relevant workflows (it provides short‑lived AWS credentials via STS).
You can reach the Prefect UI at https://pipelines.rocketclub.online.

CircleCI jobs in this repo do not require their own long‑lived IAM user or Prefect API credentials; they assume OrganizationAccountAccessRole via STS through the shared context and trigger a Prefect deployer task inside the VPC.

Step 2: CircleCI Credentials

The .circleci/config.yml in this repo assumes the aws-blog-infra-prod context is attached to the deploy-data-platform workflow. That context is responsible for providing short‑lived AWS credentials via STS; do not set project‑level AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY values for this repo.

Optionally, if you want the deploy-to-prefect job to run a Prefect smoke test, add PREFECT_API_URL=https://pipelines.rocketclub.online/api to that same context.

Step 3: Verify CircleCI Configuration

The CircleCI configuration is already set up in .circleci/config.yml. It includes:

Workflow Overview

The deploy-data-platform workflow (param-gated) runs, in order:

build-data-platform-image
- Builds data/platform/Dockerfile with the monorepo short-SHA tag
- Pushes image to ECR blog-data:<SHA>
- Skips if image already exists (idempotent)
build-prefect-deployer-image → terraform-prod-plan → approval → terraform-prod-apply
- Rebuilds the deployer image (which bakes data/platform) and rolls the ECS task definition forward to it
deploy-to-prefect (runs the Prefect deployer ECS task)
- aws ecs run-tasks the prod-prefect-deployer task, overriding PREFECT_IMAGE_REFERENCE with the freshly built flow image
- The deployer runs prefect deploy --all from the baked-in data/platform, tails CloudWatch logs, and fails on non-zero exit
deploy-flight-lambdas / deploy-ork-lambdas
- Redeploy the four blog-data-* Lambdas from the monorepo

Key Configuration Points

Trigger: Param-gated (deploy-data-platform), not on push to a branch
Image Tagging: Uses the monorepo commit SHA for immutable tags
Prefect Connection: Uses an internal Prefect deployer task running inside the VPC; CircleCI does not talk directly to Prefect
Work Pool: Deploys to blog-data-pool (defined in data/platform/prefect.yaml)

Step 4: Create Prefect Work Pool (If Not Exists)

Before deploying flows, ensure the work pool exists in your Prefect server:

Option A: Via Prefect UI

Go to https://pipelines.rocketclub.online
Log in with your credentials
Navigate to Work Pools
Click "Create Work Pool"
Configure:
- Name: blog-data-pool
- Type: ECS
- Queue: default

Option B: Via Prefect CLI

# Configure Prefect CLI to use self-hosted server (optional; typically from within the VPC or an admin machine)
export PREFECT_API_URL="https://pipelines.rocketclub.online/api"

# Create work pool
prefect work-pool create blog-data-pool --type ecs

# Verify work pool exists
prefect work-pool ls

Step 5: Test the Deployment

5.1 Trigger a Deployment

Trigger the deploy-data-platform pipeline (it does not run on push) — via the CircleCI UI's "Trigger Pipeline" with the parameter set, or the API:

curl -X POST https://circleci.com/api/v2/project/gh/rlhatcher/rocket-club/pipeline \
  -H "Circle-Token: $CIRCLECI_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"branch":"main","parameters":{"deploy-data-platform":true}}'

5.2 Monitor CircleCI Build

Go to CircleCI dashboard
Find your deploy-data-platform workflow run
Monitor the jobs:
- build-data-platform-image
- terraform-prod-plan → approval → terraform-prod-apply
- deploy-to-prefect
- deploy-flight-lambdas / deploy-ork-lambdas

5.3 Verify in Prefect UI

Go to https://pipelines.rocketclub.online
Navigate to Deployments
Verify data-pipeline-daily deployment exists
Check that the version matches your git commit hash

5.4 Verify in ECR

# List images in ECR
aws ecr describe-images \
  --repository-name blog-data \
  --region eu-west-2 \
  --query 'imageDetails[*].[imageTags[0],imagePushedAt]' \
  --output table

Step 6: Verify Deployment Configuration

Check that data/platform/prefect.yaml is correctly configured:

deployments:
  - name: data-pipeline-daily
    entrypoint: flows/main_pipeline.py:data_pipeline_flow
    work_pool:
      name: blog-data-pool
      work_queue_name: default
      job_variables:
        image: '{{ $PREFECT_IMAGE_REFERENCE }}'

The image field uses the $PREFECT_IMAGE_REFERENCE environment variable set by CircleCI.

Troubleshooting

Issue: "Failed to connect to Prefect server"

Solution:

Verify PREFECT_API_URL is correct (should end with /api) if you are using the Prefect CLI
Check that your machine or runner can reach https://pipelines.rocketclub.online/api
Confirm the Prefect API ECS service is healthy in AWS

Issue: "Work pool 'blog-data-pool' not found"

Solution:

Create the work pool using Prefect UI or CLI (see Step 4)
Verify work pool name matches exactly in data/platform/prefect.yaml

Issue: "Permission denied when pushing to ECR"

Solution:

Verify the aws-blog-infra-prod context is attached and that OrganizationAccountAccessRole has ECR permissions
Ensure AWS_REGION matches your ECR repository region

Issue: "Image already exists in ECR"

Note: This is expected behavior when re-running CI on the same commit. The build job will skip and exit successfully.

Issue: "Deployment not showing in Prefect UI"

Solution:

Check CircleCI logs for deployment errors
Verify Prefect server is accessible from CircleCI
Check that prefect deploy --all completed successfully
Verify network connectivity to https://pipelines.rocketclub.online

Next Steps

After successful deployment:

Monitor Flow Runs: Check Prefect UI for scheduled runs
Test Manual Runs: Trigger a flow run manually from Prefect UI
Set Up Alerts: Configure Prefect notifications for flow failures
Review Logs: Check ECS task logs for flow execution details

Additional Resources

Summary Checklist

Terraform infrastructure deployed
Prefect server accessible at https://pipelines.rocketclub.online
CircleCI environment variables configured
Work pool blog-data-pool created in Prefect
Test deployment successful
Flows visible in Prefect UI
Docker image in ECR