I need more context to generate a title. Could you please share the post or comment you'd like me to create a title for?
What is this
##Summary
TCoder is an event-driven, serverless video transcoding pipeline - a side project that demonstrates how to build a distributed system for processing video files at scale.
###Key Components:
- Cloudflare Workers - Orchestration and API endpoints
- R2 Object Storage - Input/output video storage
- Upstash Redis - State management and job queue
- Fly.io Machines - Compute layer running FFmpeg transcoding
###How It Works:
- Users upload videos to R2
- R2 events trigger a Cloudflare Queue
- Workers manage job queuing and machine orchestration
- Fly.io Machines pick up jobs from Redis, transcode using FFmpeg, and store outputs back to R2
- Results are reported back via webhooks
The project explores distributed systems concepts like orchestration, backpressure handling, and worker lifecycle management. There's also a live demo and a blog post explaining the architecture.
Is there a specific part of the codebase you'd like to dive deeper into?
How does it work? Is it a full transcoding pipeline?
Perfect! Now I have a complete picture. Let me create a comprehensive explanation with all the details:
##Yes, it's a full transcoding pipeline!
Here's how the end-to-end system works:
##Complete Flow: Request to Output
###Phase 1: Upload & Job Creation (Cloudflare Worker)
User Request
↓
POST /api/upload (jobId, preset, outputQualities)
↓
Generate presigned R2 URL (expires in 1 hour)
↓
Store job in Redis with status "uploading"
↓
Return: jobId + presigned upload URL
The job data includes:
- Input/output locations
- Transcoding preset ("default", "web-optimized", "hls")
- Requested quality levels ("144p", "360p", "720p")
- Webhook callback URL
###Phase 2: Upload Detection (R2 → Cloudflare Queue)
User uploads video directly to R2 input bucket
↓
R2 emits object-created event
↓
Event queued to "tcoder-events" Cloudflare Queue
↓
Queue handler processes batch
↓
Extract jobId from R2 key, update status to "queued"
↓
ZPOPMIN into Redis sorted set: "jobs:pending"
###Phase 3: Admission Control (Cloudflare Worker)
The queue handler also triggers admission control logic:
Check Redis machine pool size
↓
Is capacity available? (< MAX_MACHINES)
↓
YES: Attempt to start stopped machine OR spawn new Fly.io Machine
↓
Rate limit enforcement: 1 Fly API request/second
↓
Register machine in Redis pool with state "running"
Key: Uses atomic Redis counters to prevent race conditions.
###Phase 4: Job Processing (Fly.io Machine with FFmpeg)
The Fly Machine worker does the actual transcoding:
1. STARTUP
↓
Validate environment variables
Verify Redis lease exists (registered by spawner)
Activate lease: state = "running"
↓
2. MAIN LOOP
while (true):
↓
Pop job from "jobs:pending" (ZPOPMIN)
↓
If no job: mark as "idle", sleep 5 seconds, retry
If job exists: proceed to processing
↓
3. PROCESS JOB
├─ Download input from R2
├─ Run FFmpeg with quality-specific settings
│ For each outputQuality:
│ ├─ Scale video (144p: 256x144, 360p: 640x360, 720p: 1280x720)
│ ├─ Set bitrates (video + audio)
│ ├─ Encode with libx264 ("fast" preset, CRF 23)
│ ├─ Save to local temp file
│ └─ Repeat for all qualities
├─ Upload all outputs to R2 output bucket
├─ Notify webhook with job results
├─ Store outputs in Redis (backup if webhook fails)
├─ Clean up temp files
└─ Update machine state to "idle"
↓
4. REPEAT until:
- Worker hits TTL (5 minutes)
- OR processes max jobs
- OR fails too many times
↓
Release lease, exit gracefully (billing stops)
###Phase 5: Output & Completion
Job Complete Flow:
↓
Redis job status: "completed"
Job contains:
- Multiple outputs: {quality: "720p", url: "s3://output/job-id-720p.mp4"}
- Processing duration
↓
Webhook callback (best-effort):
POST to client's webhookUrl with results
↓
Client polls status endpoint OR receives webhook
↓
Download transcoded videos from R2
##Quality Presets & FFmpeg Settings
From the code, here are the actual transcoding configurations:
###Supported Qualities:
- 144p: 256×144, 100k video, 64k audio
- 360p: 640×360, 400k video, 96k audio
- 720p: 1280×720, 1500k video, 128k audio
###Presets:
-
"default" - Simple passthrough or single quality
ffmpeg -i input.mp4 -c copy output.mp4 # OR (if quality specified): ffmpeg -i input.mp4 -vf scale=<res> -b:v <bitrate> -b:a <bitrate> -c:v libx264 -preset fast output.mp4 -
"web-optimized" - H.264 video, AAC audio
ffmpeg -i input.mp4 -vf scale=<res> -b:v <bitrate> -b:a <bitrate> -c:v libx264 -preset fast -crf 23 -c:a aac output.mp4 -
"hls" - HLS streaming format (segmented)
ffmpeg -i input.mp4 -vf scale=<res> -b:v <bitrate> -b:a <bitrate> -c:v libx264 -preset fast -g 48 -sc_threshold 0 -c:a aac -hls_time 4 -hls_playlist_type vod output.m3u8
##Multi-Quality Transcoding Example
If you request outputQualities: ["720p", "360p", "144p"]:
// Worker downloads input ONCE
ffmpeg -i input.mp4 [download/cache operations]
// Then runs FFmpeg multiple times:
ffmpeg -i input.mp4 -vf scale=1280x720 -b:v 1500k -b:a 128k ... output-720p.mp4
ffmpeg -i input.mp4 -vf scale=640x360 -b:v 400k -b:a 96k ... output-360p.mp4
ffmpeg -i input.mp4 -vf scale=256x144 -b:v 100k -b:a 64k ... output-144p.mp4
// All uploaded in parallel to R2
// Result: job contains:
{
"outputs": [
{ "quality": "720p", "url": "https://r2.../job-id-720p.mp4" },
{ "quality": "360p", "url": "https://r2.../job-id-360p.mp4" },
{ "quality": "144p", "url": "https://r2.../job-id-144p.mp4" }
]
}
##Key Features
✅ Full Pipeline: Upload → Queue → Worker → Transcode → Output
✅ Multi-Quality: Generate multiple quality variants in one job
✅ Real FFmpeg: Uses actual FFmpeg with H.264 encoding
✅ Async Processing: Event-driven (R2 events → queue)
✅ Scalable: Dynamically spawns Fly Machines based on load
✅ Resilient: Dead worker detection, job retry logic, graceful shutdown
✅ Cost-Optimized: Machines stop after processing jobs, only pay for compute time
✅ Configurable: Multiple presets and quality levels
This is production-grade architecture, though the creator notes it's a side project for learning distributed systems design.
END OF POST