CircleCI Setup Guide for Prefect Deployment

Overview

This guide walks through setting up CircleCI to automatically deploy your Prefect flows to your self-hosted Prefect server whenever you push to the main, pipeline, or ecs branches.

Architecture

GitHub Push (main/pipeline/ecs)
    ↓
CircleCI Workflow
    ↓
1. Build Docker Image → Push to ECR
    ↓
2. Deploy Flows → Self-Hosted Prefect Server
    ↓
Prefect Server (https://pipelines.rocketclub.online)
    ↓
ECS Worker Pool → Run Flow Tasks

Prerequisites

Before setting up CircleCI, ensure you have:

Yes Terraform Infrastructure Deployed
- Prefect server running at https://pipelines.rocketclub.online
- ECR repository created (blog-data)
- ECS cluster and work pool configured
- IAM roles and permissions set up
Yes Prefect Server Accessible
- Can access Prefect UI at https://pipelines.rocketclub.online
- Have admin credentials (username:password)
Yes CircleCI Project Connected
- GitHub repository connected to CircleCI
- Project visible in CircleCI dashboard

Step 1: CircleCI and Prefect Prerequisites

Before configuring anything in this repo, ensure that:

Terraform for the prod environment has been applied from the blog_infra repo (infra/envs/prod).
The shared CircleCI context aws-blog-infra-prod has been created and wired up as described in the blog_infra docs.
You can reach the Prefect UI at https://pipelines.rocketclub.online.

CircleCI jobs in this repo do not require their own long‑lived IAM user or Prefect API credentials; they assume OrganizationAccountAccessRole via STS through the shared context and trigger a Prefect deployer task inside the VPC.

Step 2: CircleCI Credentials

The .circleci/config.yml in this repo assumes the aws-blog-infra-prod context is attached to the deploy-pipeline workflow. That context is responsible for providing short‑lived AWS credentials via STS; do not set project‑level AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY values for this repo.

Optionally, if you want the deploy-to-prefect job to run a Prefect smoke test, add PREFECT_API_URL=https://pipelines.rocketclub.online/api to that same context.

Step 3: Verify CircleCI Configuration

The CircleCI configuration is already set up in .circleci/config.yml. It includes:

Workflow Overview

build-and-push-image (runs on: main, pipeline, ecs branches)
- Checks out code
- Builds Docker image with commit hash tag
- Pushes image to ECR
- Skips if image already exists (idempotent)
deploy-to-prefect (runs after build-and-push-image)
- Installs dependencies
- Configures Prefect to use self-hosted server
- Deploys all flows defined in prefect.yaml
- Verifies deployment

Key Configuration Points

Branches: Only runs on main, pipeline, and ecs branches
Image Tagging: Uses $CIRCLE_SHA1 (git commit hash) for immutable tags
Prefect Connection: Uses an internal Prefect deployer task running inside the VPC; CircleCI does not talk directly to Prefect
Work Pool: Deploys to blog-data-pool (defined in prefect.yaml)

Step 4: Create Prefect Work Pool (If Not Exists)

Before deploying flows, ensure the work pool exists in your Prefect server:

Option A: Via Prefect UI

Go to https://pipelines.rocketclub.online
Log in with your credentials
Navigate to Work Pools
Click "Create Work Pool"
Configure:
- Name: blog-data-pool
- Type: ECS
- Queue: default

Option B: Via Prefect CLI

# Configure Prefect CLI to use self-hosted server (optional; typically from within the VPC or an admin machine)
export PREFECT_API_URL="https://pipelines.rocketclub.online/api"

# Create work pool
prefect work-pool create blog-data-pool --type ecs

# Verify work pool exists
prefect work-pool ls

Step 5: Test the Deployment

5.1 Trigger a Deployment

Push a commit to one of the deployment branches:

# Make a small change
echo "# Test deployment" >> README.md
git add README.md
git commit -m "test: trigger CircleCI deployment"

# Push to pipeline branch (for testing)
git push origin main:pipeline

5.2 Monitor CircleCI Build

Go to CircleCI dashboard
Find your workflow run
Monitor the two jobs:
- build-and-push-image
- deploy-to-prefect

5.3 Verify in Prefect UI

Go to https://pipelines.rocketclub.online
Navigate to Deployments
Verify data-pipeline-daily deployment exists
Check that the version matches your git commit hash

5.4 Verify in ECR

# List images in ECR
aws ecr describe-images \
  --repository-name blog-data \
  --region eu-west-2 \
  --query 'imageDetails[*].[imageTags[0],imagePushedAt]' \
  --output table

Step 6: Verify Deployment Configuration

Check that your prefect.yaml is correctly configured:

deployments:
  - name: data-pipeline-daily
    entrypoint: flows/main_pipeline.py:data_pipeline_flow
    work_pool:
      name: blog-data-pool
      work_queue_name: default
      job_variables:
        image: '{{ $PREFECT_IMAGE_REFERENCE }}'

The image field uses the $PREFECT_IMAGE_REFERENCE environment variable set by CircleCI.

Troubleshooting

Issue: "Failed to connect to Prefect server"

Solution:

Verify PREFECT_API_URL is correct (should end with /api) if you are using the Prefect CLI
Check that your machine or runner can reach https://pipelines.rocketclub.online/api
Confirm the Prefect API ECS service is healthy in AWS

Issue: "Work pool 'blog-data-pool' not found"

Solution:

Create the work pool using Prefect UI or CLI (see Step 4)
Verify work pool name matches exactly in prefect.yaml

Issue: "Permission denied when pushing to ECR"

Solution:

Verify the aws-blog-infra-prod context is attached and that OrganizationAccountAccessRole has ECR permissions
Ensure AWS_REGION matches your ECR repository region

Issue: "Image already exists in ECR"

Note: This is expected behavior when re-running CI on the same commit. The build job will skip and exit successfully.

Issue: "Deployment not showing in Prefect UI"

Solution:

Check CircleCI logs for deployment errors
Verify Prefect server is accessible from CircleCI
Check that prefect deploy --all completed successfully
Verify network connectivity to https://prefect.rocketclub.online

Next Steps

After successful deployment:

Monitor Flow Runs: Check Prefect UI for scheduled runs
Test Manual Runs: Trigger a flow run manually from Prefect UI
Set Up Alerts: Configure Prefect notifications for flow failures
Review Logs: Check ECS task logs for flow execution details

Additional Resources

Summary Checklist

Terraform infrastructure deployed
Prefect server accessible at https://prefect.rocketclub.online
CircleCI environment variables configured
Work pool blog-data-pool created in Prefect
Test deployment successful
Flows visible in Prefect UI
Docker image in ECR