Deployment Guide

Note: Production infrastructure (VPC, ECS, Aurora, S3, IAM, Prefect API, etc.) is defined and deployed from this repository via Terraform (see infra/platform/infra/envs/prod). This document focuses on the application-side pieces for the data platform (data/platform).

How deployment works

Application deployment for the data platform (data/platform) is handled entirely by CircleCI and a Prefect deployer ECS task defined in Terraform in this repository, through the param-gated deploy-data-platform pipeline (the migrated successor to the external blog_data repo's deploy pipeline — see ADR-0005):

1. You trigger the deploy-data-platform pipeline parameter (it does not run on push).
2. CircleCI builds the data-platform Docker image from data/platform/Dockerfile and pushes it to the blog-data ECR repository tagged with the monorepo commit SHA.
3. It rebuilds the Prefect deployer image (which bakes data/platform) and rolls the deployer ECS task definition forward via a prod Terraform apply.
4. CircleCI runs the deploy-to-prefect job, which:
   • Assumes an AWS IAM role using STS.
   • Starts a one-off Prefect deployer ECS task (defined in Terraform).
   • That task runs prefect deploy --all from the baked-in data/platform (no blog_data clone) against the self-hosted Prefect API at https://pipelines.rocketclub.online, overriding PREFECT_IMAGE_REFERENCE with the freshly built flow-execution image.
5. The Prefect deployer script ensures the blog-data-pool ECS work pool exists with the correct base job template and updates all deployments using the new image tag.
6. Finally, the pipeline redeploys the four blog-data-* Lambdas (deploy-flight-lambdas / deploy-ork-lambdas) from the monorepo, so one trigger covers the whole data-platform deploy.

Infrastructure (VPC, ECS services, ALB/CloudFront, Aurora, etc.) is provisioned and updated via Terraform in this repo; see the infrastructure docs for details.

Visual overview

For a high-level view of how the Prefect worker pool and CI/CD fit together, see the generated diagrams (served from /diagrams/):

• Prefect runtime:

  

• CI & Prefect deployer:

  

These diagrams are generated from code using data/platform/generate_architecture_diagram.py. See Pipelines Architecture for generation instructions.

Day-to-day deployment

For practical deployment tasks, use the following documents:

• CircleCI Setup Guide
• CircleCI Deployment Guide
• Quick Start Deployment
• Deployment Checklist
• Prefect Work Pool Configuration

This page is intentionally high-level – treat code and data/platform/prefect.yaml as the source of truth for deployments.