Cloudinary Configuration Guide

Last Updated: 2025-10-30
Status: Production
Related Diagram: terraform/diagram_7_image_processing.png

Overview

This document provides detailed configuration information for Cloudinary CDN integration in the blog-data pipeline. It covers account setup, credential management, transformation settings, and integration with AWS services.

Account Information

Cloudinary Account

Cloud Name: ronaldhatcher
Account Type: Free Tier
Region: Auto (Global CDN)
Dashboard: https://cloudinary.com/console

Free Tier Limits

Storage: 25 GB
Bandwidth: 25 GB/month
Transformations: 25,000/month
API Calls: Unlimited (rate-limited)

Current Usage: Monitor at https://cloudinary.com/console/usage

Upgrade Path

If limits are exceeded:

Plus Plan: $99/month (100 GB storage, 100 GB bandwidth)
Advanced Plan: $249/month (500 GB storage, 500 GB bandwidth)
Custom Plan: Contact sales for enterprise needs

Credentials Management

AWS Secrets Manager

Secret Name: blog-data/cloudinary/credentials
Region: eu-west-2
ARN: arn:aws:secretsmanager:eu-west-2:*:secret:blog-data/cloudinary/credentials-*

Secret Structure:

{
  "cloud_name": "your-cloud-name",
  "api_key": "YOUR_CLOUDINARY_API_KEY",
  "api_secret": "YOUR_CLOUDINARY_API_SECRET"
}

Creating/Updating the Secret

Initial Creation:

aws secretsmanager create-secret \
  --name blog-data/cloudinary/credentials \
  --description "Cloudinary API credentials for image upload" \
  --secret-string '{
    "cloud_name": "your-cloud-name",
    "api_key": "YOUR_CLOUDINARY_API_KEY",
    "api_secret": "YOUR_CLOUDINARY_API_SECRET"
  }' \
  --region eu-west-2

Updating Credentials:

aws secretsmanager update-secret \
  --secret-id blog-data/cloudinary/credentials \
  --secret-string '{
    "cloud_name": "your-cloud-name",
    "api_key": "YOUR_NEW_API_KEY",
    "api_secret": "YOUR_NEW_API_SECRET"
  }' \
  --region eu-west-2

Retrieving Credentials:

aws secretsmanager get-secret-value \
  --secret-id blog-data/cloudinary/credentials \
  --region eu-west-2 \
  --query SecretString \
  --output text | jq .

IAM Permissions

ECS Task Execution Role Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": [
        "arn:aws:secretsmanager:eu-west-2:*:secret:blog-data/cloudinary/credentials-*"
      ]
    }
  ]
}

Terraform Configuration:

resource "aws_iam_role_policy" "ecs_secrets_access" {
  name = "cloudinary-secrets-access"
  role = aws_iam_role.ecs_task_execution_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "secretsmanager:GetSecretValue"
        ]
        Resource = [
          "arn:aws:secretsmanager:eu-west-2:*:secret:blog-data/cloudinary/credentials-*"
        ]
      }
    ]
  })
}

Security Best Practices

Never commit credentials to code
- Use Secrets Manager for all environments
- Never use environment variables in code
Rotate credentials regularly
- Generate new API key/secret in Cloudinary dashboard
- Update Secrets Manager
- No downtime required (ECS tasks fetch on startup)
Limit IAM permissions
- Only grant GetSecretValue (not PutSecretValue or DeleteSecret)
- Scope to specific secret ARN
Monitor access
- Enable CloudTrail logging for Secrets Manager
- Alert on unusual access patterns

Folder Structure

Organization

Images are organized by product type and manufacturer:

cloudinary://ronaldhatcher/
└── kits/
    ├── estes/
    │   ├── 1234.jpg
    │   ├── 5678.jpg
    │   └── ...
    ├── loc/
    │   ├── 9012.jpg
    │   ├── 3456.jpg
    │   └── ...
    └── rocketarium/
        ├── 7890.jpg
        └── ...

Naming Convention

Public ID Format: {product_id}

Examples:

Product ID: 1234 → Public ID: 1234
Product ID: alpha-iii → Public ID: alpha-iii

Full URL:

https://res.cloudinary.com/ronaldhatcher/image/upload/kits/{manufacturer}/{product_id}.{ext}

Folder Configuration in Code

# In src/cloudinary_uploader.py
folder = f"kits/{manufacturer}"
public_id = product_id

result = cloudinary.uploader.upload(
    image_data,
    folder=folder,
    public_id=public_id,
    resource_type="image"
)

Transformations

Automatic Transformations

Cloudinary automatically applies these transformations when requested:

1. Format Optimization (`f_auto`)

Automatically selects the best format:

WebP for Chrome, Edge, Firefox
AVIF for browsers that support it
JPEG/PNG fallback for older browsers

URL:

https://res.cloudinary.com/ronaldhatcher/image/upload/f_auto/kits/estes/1234.jpg

2. Quality Optimization (`q_auto`)

Automatically adjusts quality based on content:

High quality for images with fine details
Lower quality for images with solid colors
Typically 40-80% file size reduction

URL:

https://res.cloudinary.com/ronaldhatcher/image/upload/q_auto/kits/estes/1234.jpg

3. Responsive Sizing

Generate multiple sizes for different devices:

Thumbnail (300x300):

https://res.cloudinary.com/ronaldhatcher/image/upload/w_300,h_300,c_fill/kits/estes/1234.jpg

Medium (600x600):

https://res.cloudinary.com/ronaldhatcher/image/upload/w_600,h_600,c_fit/kits/estes/1234.jpg

Large (1200x1200):

https://res.cloudinary.com/ronaldhatcher/image/upload/w_1200,h_1200,c_limit/kits/estes/1234.jpg

Transformation Parameters

Common Parameters:

f_auto - Automatic format selection
q_auto - Automatic quality optimization
w_XXX - Width in pixels
h_XXX - Height in pixels
c_fill - Crop to fill dimensions
c_fit - Fit within dimensions (maintain aspect ratio)
c_limit - Limit to maximum dimensions
dpr_2.0 - Device pixel ratio (for retina displays)

Example Combined:

https://res.cloudinary.com/ronaldhatcher/image/upload/f_auto,q_auto,w_600,h_600,c_fit,dpr_2.0/kits/estes/1234.jpg

Transformation Presets

Recommended Presets:

Product Thumbnail:
- f_auto,q_auto,w_300,h_300,c_fill
- Use for: Product listings, search results
Product Detail:
- f_auto,q_auto,w_800,h_800,c_fit
- Use for: Product detail pages
Hero Image:
- f_auto,q_auto,w_1200,h_600,c_fill
- Use for: Banner images, featured products
Mobile Optimized:
- f_auto,q_auto,w_400,h_400,c_fit,dpr_2.0
- Use for: Mobile devices with retina displays

Upload Configuration

Upload Parameters

Default Upload Settings:

result = cloudinary.uploader.upload(
    image_data,
    folder=f"kits/{manufacturer}",
    public_id=product_id,
    resource_type="image",
    overwrite=True,
    invalidate=True,
    timeout=30
)

Parameter Explanations:

folder: Organizes images by manufacturer
public_id: Unique identifier for the image
resource_type: Type of asset (image, video, raw)
overwrite: Replace existing image with same public_id
invalidate: Clear CDN cache when overwriting
timeout: Upload timeout in seconds

Upload Options

Additional Options:

# Add tags for organization
tags=["rocket-kit", manufacturer, "product"]

# Set access control
type="upload"  # or "private" for restricted access

# Add metadata
context={"product_id": product_id, "manufacturer": manufacturer}

# Set eager transformations (pre-generate)
eager=[
    {"width": 300, "height": 300, "crop": "fill"},
    {"width": 600, "height": 600, "crop": "fit"}
]

Error Handling

Retry Logic:

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=10)
)
def upload_with_retry(image_data, public_id, folder):
    return cloudinary.uploader.upload(
        image_data,
        folder=folder,
        public_id=public_id,
        resource_type="image"
    )

CDN Configuration

Global Distribution

Cloudinary uses a global CDN with edge locations worldwide:

North America: 20+ locations
Europe: 15+ locations
Asia Pacific: 10+ locations
South America: 5+ locations

Benefits:

Low latency worldwide
Automatic failover
DDoS protection
SSL/TLS encryption

Caching

Default Cache Settings:

Browser cache: 1 year (31536000 seconds)
CDN cache: Permanent (until invalidated)

Cache Invalidation:

# Automatic on overwrite with invalidate=True
cloudinary.uploader.upload(
    image_data,
    public_id=product_id,
    overwrite=True,
    invalidate=True  # Clears CDN cache
)

# Manual invalidation
cloudinary.api.delete_resources([public_id], invalidate=True)

HTTPS/SSL

All Cloudinary URLs use HTTPS by default:

https://res.cloudinary.com/ronaldhatcher/...

Certificate: Managed by Cloudinary (auto-renewed)

Integration with AWS

ECS Task Configuration

Environment Variables (NOT USED):

# DO NOT USE - credentials should come from Secrets Manager
environment:
  - name: CLOUDINARY_CLOUD_NAME
    value: ronaldhatcher # No Bad practice

Secrets Manager Integration (RECOMMENDED):

# In src/cloudinary_uploader.py
def get_cloudinary_credentials_from_secrets_manager():
    """Fetch credentials from AWS Secrets Manager."""
    secret_name = "blog-data/cloudinary/credentials"
    region_name = "eu-west-2"

    client = boto3.client('secretsmanager', region_name=region_name)
    secret = client.get_secret_value(SecretId=secret_name)
    return json.loads(secret['SecretString'])

S3 Integration

Storing Cloudinary URLs:

# In CSV stored in S3
df['cloudinary_images'] = cloudinary_url

# Upload to S3
s3_client.put_object(
    Bucket='blog-data-raw',
    Key=f'kits/{manufacturer}_kits.csv',
    Body=df.to_csv(index=False)
)

CSV Structure:

product_id,name,manufacturer,image,cloudinary_images
1234,Alpha III,Estes,https://vendor.com/img.jpg,https://res.cloudinary.com/.../1234.jpg

Monitoring and Logging

Cloudinary Dashboard

Usage Monitoring:

Storage: Current usage vs. limit
Bandwidth: Monthly usage vs. limit
Transformations: Monthly count vs. limit
API Calls: Rate and errors

Access: https://cloudinary.com/console/usage

Application Logging

Upload Success:

INFO: Uploading image for product 1234 to Cloudinary
INFO: Successfully uploaded image: https://res.cloudinary.com/.../1234.jpg
INFO: Upload took 1.23 seconds

Upload Failure:

ERROR: Failed to upload image for product 1234: Connection timeout
ERROR: Cloudinary API error: Invalid credentials

Metrics

Tracked in ExtractionMetrics:

metrics.images_uploaded += 1
metrics.images_failed += 1
metrics.upload_duration += elapsed_time

Troubleshooting

Common Issues

1. Invalid Credentials

Symptom: cloudinary.exceptions.AuthorizationRequired

Solution:

# Verify secret exists and is correct
aws secretsmanager get-secret-value \
  --secret-id blog-data/cloudinary/credentials \
  --region eu-west-2

# Check ECS task role has permission
aws iam get-role-policy \
  --role-name blog-data-ecs-task-execution-role \
  --policy-name cloudinary-secrets-access

2. Upload Timeout

Symptom: requests.exceptions.Timeout

Solution:

Increase timeout: timeout=60
Check network connectivity from ECS
Verify Cloudinary API status

3. Quota Exceeded

Symptom: cloudinary.exceptions.Error: Quota exceeded

Solution:

Check usage in Cloudinary dashboard
Upgrade plan if needed
Optimize uploads (reduce frequency, smaller images)

4. Image Not Found

Symptom: 404 on Cloudinary URL

Solution:

Verify public_id is correct
Check folder path matches upload
Ensure image was successfully uploaded (check logs)

Cost Optimization

Best Practices

Use Transformations Wisely
- Cache transformed URLs in frontend
- Don't generate unique transformations for each request
- Use presets for common sizes
Optimize Upload Frequency
- Only upload when image changes
- Use overwrite=True to replace, not duplicate
- Batch uploads when possible
Monitor Usage
- Set up alerts for 80% quota usage
- Review bandwidth usage monthly
- Identify and optimize high-traffic images
Leverage CDN Caching
- Use long cache times (1 year default)
- Invalidate only when necessary
- Use versioned URLs for cache busting

Cost Comparison

Cloudinary vs. S3-only:

Aspect	Cloudinary	S3 Only
Storage	$0 (free tier)	~$0.023/GB
Bandwidth	$0 (free tier)	~$0.09/GB
Transformations	Automatic	Manual (Lambda)
CDN	Included	CloudFront extra
Optimization	Automatic	Manual
Total (25GB)	$0	~$3/month

Verdict: Cloudinary free tier is more cost-effective for current usage.

IMAGE_PROCESSING.md - Complete image processing guide
Architecture Diagram 7 - Visual flow
Cloudinary Documentation - Official docs
Cloudinary Python SDK - SDK reference

Appendix

Cloudinary API Reference

Upload API:

cloudinary.uploader.upload(file, **options)

Admin API:

cloudinary.api.resources(**options)  # List resources
cloudinary.api.resource(public_id)   # Get resource details
cloudinary.api.delete_resources([public_ids])  # Delete resources

URL Generation:

cloudinary.CloudinaryImage(public_id).build_url(**options)

Useful Links

Dashboard: https://cloudinary.com/console
API Reference: https://cloudinary.com/documentation/image_upload_api_reference
Transformation Reference: https://cloudinary.com/documentation/image_transformation_reference
Python SDK: https://github.com/cloudinary/pycloudinary

Cloudinary Configuration Guide

Overview

Account Information

Cloudinary Account

Free Tier Limits

Upgrade Path

Credentials Management

AWS Secrets Manager

Creating/Updating the Secret

IAM Permissions

Security Best Practices

Folder Structure

Organization

Naming Convention

Folder Configuration in Code

Transformations

Automatic Transformations

1. Format Optimization (f_auto)

2. Quality Optimization (q_auto)

3. Responsive Sizing

Transformation Parameters

Transformation Presets

Upload Configuration

Upload Parameters

Upload Options

Error Handling

CDN Configuration

Global Distribution

Caching

HTTPS/SSL

Integration with AWS

ECS Task Configuration

S3 Integration

Monitoring and Logging

Cloudinary Dashboard

Application Logging

Metrics

Troubleshooting

Common Issues

1. Invalid Credentials

2. Upload Timeout

3. Quota Exceeded

4. Image Not Found

Cost Optimization

Best Practices

Cost Comparison

Related Documentation

Appendix

Cloudinary API Reference

Useful Links

1. Format Optimization (`f_auto`)

2. Quality Optimization (`q_auto`)