Quick Start: Deploy Prefect Flows with CircleCI

This is the fastest way to get your Prefect flows deployed to your self-hosted Prefect server using CircleCI.

Prerequisites

Before you start, ensure:

• Yes Terraform infrastructure is deployed (from the blog_infra repository)
• Yes Prefect server is running at https://pipelines.rocketclub.online
• Yes GitHub repository is connected to CircleCI
• Yes AWS CLI is configured
• Yes Python 3.7+ installed

5-Minute Setup

1. Ensure blog_infra infrastructure and CircleCI context

• Terraform infrastructure is deployed from the blog_infra repository (infra/envs/prod).
• The shared CircleCI context aws-blog-infra-prod exists and is attached to this repo's deploy-pipeline workflow.
• Prefect server is running at https://pipelines.rocketclub.online.

2. Create Prefect Work Pool (if needed)

# Set environment variables
export PREFECT_API_URL="https://pipelines.rocketclub.online/api"

# Create work pool
prefect work-pool create blog-data-pool --type ecs

# Verify
prefect work-pool ls

3. Test Deployment

# Push to pipeline branch to test
git push origin main:pipeline

# Or push to main for production
git push origin main

4. Verify Deployment

In CircleCI:

1. Go to CircleCI Pipelines
2. Watch the workflow run
3. Verify both jobs complete successfully

In Prefect UI:

1. Go to https://pipelines.rocketclub.online
2. Navigate to Deployments
3. Verify 4 deployments exist:
   • data-pipeline-daily (scheduled)
   • data-extraction (on-demand)
   • data-cleaning (on-demand)
   • graph-load (on-demand)

In AWS ECR:

aws ecr describe-images \
  --repository-name blog-data \
  --region eu-west-2 \
  --query 'imageDetails[0].[imageTags[0],imagePushedAt]' \
  --output table

What Gets Deployed

Scheduled Flow

data-pipeline-daily

• Runs on the schedule defined in prefect.yaml (currently 0 2 */7 * * in America/New_York)
• Executes: extract → clean → load
• Automatically triggered by Prefect according to that schedule

On-Demand Flows

data-extraction

• Extract data from web sources
• Trigger manually from Prefect UI

data-cleaning

• Clean and transform raw data
• Trigger manually from Prefect UI

graph-load

• Load clean data into Neo4j
• Trigger manually from Prefect UI

Triggering Manual Runs

Via Prefect UI

1. Go to https://pipelines.rocketclub.online
2. Navigate to Deployments
3. Click on a deployment
4. Click "Run"
5. Monitor in Flow Runs

Via Prefect CLI

# Set environment variables
export PREFECT_API_URL="https://pipelines.rocketclub.online/api"

# Trigger a deployment
prefect deployment run data-extraction
prefect deployment run data-cleaning
prefect deployment run graph-load
prefect deployment run data-pipeline-daily

Troubleshooting

Script fails to get AWS credentials

Solution:

cd terraform
terraform apply  # Ensure infrastructure is deployed
terraform output  # Verify outputs are available

Work pool creation fails

Solution:

# Verify Prefect server is accessible
curl -I https://pipelines.rocketclub.online/api/health

# Check connectivity
export PREFECT_API_URL="https://pipelines.rocketclub.online/api"
prefect version

CircleCI build fails

Check:

1. Environment variables are set correctly in CircleCI
2. AWS credentials have ECR permissions
3. Prefect server is accessible from CircleCI

View logs: CircleCI Pipelines

Deployment not showing in Prefect UI

Check:

1. CircleCI deploy-to-prefect job completed successfully
2. Prefect server is accessible
3. Work pool blog-data-pool exists

Verify:

export PREFECT_API_URL="https://pipelines.rocketclub.online/api"
prefect deployment ls

Next Steps

After successful deployment:

1. Monitor Scheduled Runs

• Check Prefect UI daily for pipeline runs
• Review flow run logs for errors
• Set up notifications for failures

2. Optimize Resources

# Check ECS task resource usage
aws ecs describe-tasks \
  --cluster rocket-club-prod-ecs \
  --region eu-west-2

# Adjust task CPU/memory in Terraform if needed

3. Set Up Alerts

Configure Prefect notifications:

1. Go to Settings → Notifications
2. Add notification for flow failures
3. Configure email/Slack/webhook

4. Review Costs

# Check ECR storage usage
aws ecr describe-repositories \
  --repository-names blog-data \
  --region eu-west-2

# Clean up old images if needed

Manual Alternative

If you prefer manual setup, see:

• Detailed Setup: CIRCLECI_SETUP.md
• Deployment Guide: CIRCLECI_DEPLOYMENT.md
• Checklist: DEPLOYMENT_CHECKLIST.md

Support

For issues:

1. Check documentation in docs/
2. Review CircleCI logs
3. Check Prefect server logs
4. Review ECS task logs in CloudWatch

Summary

Yes 5-minute setup with automated script

Yes 4 deployments (1 scheduled + 3 on-demand)

Yes Automatic deployment on push to main/pipeline/ecs

Yes Self-hosted Prefect at https://pipelines.rocketclub.online

🚀 Ready to go!