I need more context to generate a title. Could you please share the post or comment you'd like me to create a title for?

master
ed74602docs: add demo site and blog post links to README for enhanced project visibility

What is this

Avatar of anthropic/claude-haiku-4.5
Claude Haiku 4.5
commented

##Summary

TCoder is an event-driven, serverless video transcoding pipeline - a side project that demonstrates how to build a distributed system for processing video files at scale.

###Key Components:

  • Cloudflare Workers - Orchestration and API endpoints
  • R2 Object Storage - Input/output video storage
  • Upstash Redis - State management and job queue
  • Fly.io Machines - Compute layer running FFmpeg transcoding

###How It Works:

  1. Users upload videos to R2
  2. R2 events trigger a Cloudflare Queue
  3. Workers manage job queuing and machine orchestration
  4. Fly.io Machines pick up jobs from Redis, transcode using FFmpeg, and store outputs back to R2
  5. Results are reported back via webhooks

The project explores distributed systems concepts like orchestration, backpressure handling, and worker lifecycle management. There's also a live demo and a blog post explaining the architecture.

Is there a specific part of the codebase you'd like to dive deeper into?

How does it work? Is it a full transcoding pipeline?

Avatar of anthropic/claude-haiku-4.5
Claude Haiku 4.5
commented

Perfect! Now I have a complete picture. Let me create a comprehensive explanation with all the details:

##Yes, it's a full transcoding pipeline!

Here's how the end-to-end system works:


##Complete Flow: Request to Output

###Phase 1: Upload & Job Creation (Cloudflare Worker)

User Request
    ↓
POST /api/upload (jobId, preset, outputQualities)
    ↓
Generate presigned R2 URL (expires in 1 hour)
    ↓
Store job in Redis with status "uploading"
    ↓
Return: jobId + presigned upload URL

The job data includes:

  • Input/output locations
  • Transcoding preset ("default", "web-optimized", "hls")
  • Requested quality levels ("144p", "360p", "720p")
  • Webhook callback URL

###Phase 2: Upload Detection (R2 → Cloudflare Queue)

User uploads video directly to R2 input bucket
    ↓
R2 emits object-created event
    ↓
Event queued to "tcoder-events" Cloudflare Queue
    ↓
Queue handler processes batch
    ↓
Extract jobId from R2 key, update status to "queued"
    ↓
ZPOPMIN into Redis sorted set: "jobs:pending"

###Phase 3: Admission Control (Cloudflare Worker)

The queue handler also triggers admission control logic:

Check Redis machine pool size
    ↓
Is capacity available? (< MAX_MACHINES)
    ↓
YES: Attempt to start stopped machine OR spawn new Fly.io Machine
    ↓
Rate limit enforcement: 1 Fly API request/second
    ↓
Register machine in Redis pool with state "running"

Key: Uses atomic Redis counters to prevent race conditions.

###Phase 4: Job Processing (Fly.io Machine with FFmpeg)

The Fly Machine worker does the actual transcoding:

1. STARTUP
   ↓
   Validate environment variables
   Verify Redis lease exists (registered by spawner)
   Activate lease: state = "running"
   ↓

2. MAIN LOOP
   while (true):
       ↓
       Pop job from "jobs:pending" (ZPOPMIN)
       ↓
       If no job: mark as "idle", sleep 5 seconds, retry
       If job exists: proceed to processing
       ↓

3. PROCESS JOB
   ├─ Download input from R2
   ├─ Run FFmpeg with quality-specific settings
   │  For each outputQuality:
   │  ├─ Scale video (144p: 256x144, 360p: 640x360, 720p: 1280x720)
   │  ├─ Set bitrates (video + audio)
   │  ├─ Encode with libx264 ("fast" preset, CRF 23)
   │  ├─ Save to local temp file
   │  └─ Repeat for all qualities
   ├─ Upload all outputs to R2 output bucket
   ├─ Notify webhook with job results
   ├─ Store outputs in Redis (backup if webhook fails)
   ├─ Clean up temp files
   └─ Update machine state to "idle"
   ↓

4. REPEAT until:
   - Worker hits TTL (5 minutes)
   - OR processes max jobs
   - OR fails too many times
   ↓
   Release lease, exit gracefully (billing stops)

###Phase 5: Output & Completion

Job Complete Flow:
    ↓
Redis job status: "completed"
Job contains:
  - Multiple outputs: {quality: "720p", url: "s3://output/job-id-720p.mp4"}
  - Processing duration
    ↓
Webhook callback (best-effort):
  POST to client's webhookUrl with results
    ↓
Client polls status endpoint OR receives webhook
    ↓
Download transcoded videos from R2

##Quality Presets & FFmpeg Settings

From the code, here are the actual transcoding configurations:

###Supported Qualities:

  • 144p: 256×144, 100k video, 64k audio
  • 360p: 640×360, 400k video, 96k audio
  • 720p: 1280×720, 1500k video, 128k audio

###Presets:

  1. "default" - Simple passthrough or single quality

    ffmpeg -i input.mp4 -c copy output.mp4
    # OR (if quality specified):
    ffmpeg -i input.mp4 -vf scale=<res> -b:v <bitrate> -b:a <bitrate> -c:v libx264 -preset fast output.mp4
    
  2. "web-optimized" - H.264 video, AAC audio

    ffmpeg -i input.mp4 -vf scale=<res> -b:v <bitrate> -b:a <bitrate> -c:v libx264 -preset fast -crf 23 -c:a aac output.mp4
    
  3. "hls" - HLS streaming format (segmented)

    ffmpeg -i input.mp4 -vf scale=<res> -b:v <bitrate> -b:a <bitrate> -c:v libx264 -preset fast -g 48 -sc_threshold 0 -c:a aac -hls_time 4 -hls_playlist_type vod output.m3u8
    

##Multi-Quality Transcoding Example

If you request outputQualities: ["720p", "360p", "144p"]:

// Worker downloads input ONCE
ffmpeg -i input.mp4 [download/cache operations]

// Then runs FFmpeg multiple times:
ffmpeg -i input.mp4 -vf scale=1280x720 -b:v 1500k -b:a 128k ... output-720p.mp4
ffmpeg -i input.mp4 -vf scale=640x360 -b:v 400k -b:a 96k ... output-360p.mp4
ffmpeg -i input.mp4 -vf scale=256x144 -b:v 100k -b:a 64k ... output-144p.mp4

// All uploaded in parallel to R2
// Result: job contains:
{
  "outputs": [
    { "quality": "720p", "url": "https://r2.../job-id-720p.mp4" },
    { "quality": "360p", "url": "https://r2.../job-id-360p.mp4" },
    { "quality": "144p", "url": "https://r2.../job-id-144p.mp4" }
  ]
}

##Key Features

Full Pipeline: Upload → Queue → Worker → Transcode → Output
Multi-Quality: Generate multiple quality variants in one job
Real FFmpeg: Uses actual FFmpeg with H.264 encoding
Async Processing: Event-driven (R2 events → queue)
Scalable: Dynamically spawns Fly Machines based on load
Resilient: Dead worker detection, job retry logic, graceful shutdown
Cost-Optimized: Machines stop after processing jobs, only pay for compute time
Configurable: Multiple presets and quality levels

This is production-grade architecture, though the creator notes it's a side project for learning distributed systems design.


END OF POST

I need more context to generate a title. Could you please share the post or comment you'd like me to create a title for? — v0id-user/tcoder