Ultimate Video Onboarding Pipeline – YAML & SQL Triggers on Supabase

Every video platform faces the same problem: users upload raw media, and you need to deliver it in multiple formats, resolutions, and with thumbnails. Doing this manually is a nightmare. Doing it with a serverless, event‑driven architecture is the solution — especially when you combine PostgreSQL triggers, pgmq, Supabase Storage, and ffmpeglab.

This guide shows you how to build the ultimate video onboarding pipeline using a declarative YAML template and a transpiler that generates the complete SQL migration. You get:

• Video thumbnails at multiple sizes (160×90, 320×180, 640×360)
• Video transcoding to multiple resolutions (480p, 720p, 1080p)
• Image thumbnails (320×320)
• Real‑time notifications via pg_notify
• Per‑user isolation with RLS policies
• Durable job queuing with pgmq
• A visual SVG graph of your pipeline
• Per‑run grouping – all outputs for a single upload are stored under a unique runId folder.

The Gap: Manual Media Processing Is Broken

Most media processing pipelines require manual steps: upload a file, trigger a script, wait for processing, then manually move the file. This is slow, error‑prone, and doesn't scale.

What if the pipeline could be fully automated — triggered by the upload itself, processing in the background, and notifying the user when complete? And what if you could define that pipeline in a declarative YAML file that you can version, share, and reuse?

This guide shows you exactly how to build that pipeline using a YAML‑driven approach.

Architecture Overview

The pipeline consists of:

• Supabase Storage — two buckets: video-uploads (private) and video-processed (public). RLS policies isolate user data.
• PostgreSQL triggers — one trigger per processing step. Each fires on INSERT into storage.objects when a file appears in a specific bucket.
• pgmq — message queue for job processing.
• ffmpeglab-runner — executes FFmpeg commands to generate thumbnails and resized media.
• pg_notify — real‑time status updates.
• YAML transpiler — reads a declarative YAML file and generates the complete SQL migration, plus an optional SVG graph.

Important: This pipeline uses the existing render and logpiece tables from the FFmpegLab server. It does not create new tables — it only adds the pipeline components.

What the Pipeline Delivers

Prerequisites

• A Supabase project (cloud or self‑hosted).
• ffmpeglab-server and ffmpeglab-runner deployed (see setup guide).
• The render and logpiece tables must already exist (created by the FFmpegLab server migrations).
• Access to your Supabase database (psql or the Supabase SQL Editor).
• Deno installed to run the transpiler.

The YAML‑Driven Approach

While you can write the SQL directly, the recommended way is to use the YAML transpiler. This gives you:

• Declarative pipeline definition – define steps, triggers, and buckets in clean YAML.
• Automatic SQL generation – the transpiler produces the exact PostgreSQL migration.
• Visual pipeline graph – generate an SVG diagram of your pipeline with --svg.
• Reusable templates – share and version your pipeline definitions.

The transpiler is a single TypeScript file that reads your YAML and generates the SQL migration. It runs with Deno and has zero external dependencies (except yaml for parsing).

The YAML Template

Create a file called video-pipeline.yaml with the following content. It defines the buckets, RLS policies, and each processing step. The runId section configures how the per‑run ID is generated — in this case, deterministically from the input file name.

name: "Ultimate Video Onboarding Pipeline"
pipelineId: "video-pipeline"
runId:
  mode: "deterministic"
  template: "{baseFilename}"
description: "Parallel video & image processing: thumbnails, transcodes, all artifacts kept"
version: "1.0.0"

editor:
  compressionLevel: 23
  preset: "medium"
  aspectRatio: "16:9"
  framerate: 30
  opacity: 1.0
  output: "mp4"

storage:
  output_bucket: "video-processed"
  buckets:
    - name: "video-uploads"
      public: false
      allowed_mime_types:
        - "video/mp4"
        - "video/quicktime"
        - "video/webm"
        - "image/jpeg"
        - "image/png"
        - "image/webp"
        - "image/gif"
    - name: "video-processed"
      public: true
      allowed_mime_types:
        - "image/jpeg"
        - "image/png"
        - "video/mp4"

  rls_policies:
    - name: "Users can upload to their own video folder"
      operation: "INSERT"
      role: "authenticated"
      condition: |
        bucket_id = 'video-uploads' AND
        (storage.foldername(name))[1] = auth.uid()::text

    - name: "Users can read their own processed video"
      operation: "SELECT"
      role: "authenticated"
      condition: |
        bucket_id = 'video-processed' AND
        (storage.foldername(name))[1] = auth.uid()::text

    - name: "Public read access to processed video"
      operation: "SELECT"
      role: "anon"
      condition: |
        bucket_id = 'video-processed'

    - name: "Service role can manage processed video"
      operation: "ALL"
      role: "service_role"
      condition: |
        bucket_id = 'video-processed'

steps:
  # ---- Video thumbnails ----
  - id: "thumbnail_160x90"
    trigger:
      name: "handle_thumbnail_160x90"
      event: "INSERT"
      table: "storage.objects"
      condition: |
        NEW.bucket_id = 'video-uploads' AND
        NEW.name NOT LIKE '%.emptyFolderPlaceholder' AND
        (NEW.metadata->>'mimetype' LIKE 'video/%')
    command: -i $MEDIA_1 -vf thumbnail,scale=160:90 -frames:v 1 -f image2 -y $OUTPUT_PATH
    inputs: ["INPUT_FILE"]
    outputs: ["OUTPUT_FILE"]
    output_path: "{{userId}}/{{pipelineId}}/{{runId}}/thumbnails/160x90.jpg"
    editor:
      output: "jpg"
      preset: "fast"
      selectedCode: "custom"
      width: 160
      height: 90
    keep: true

  - id: "thumbnail_320x180"
    trigger:
      name: "handle_thumbnail_320x180"
      event: "INSERT"
      table: "storage.objects"
      condition: |
        NEW.bucket_id = 'video-uploads' AND
        NEW.name NOT LIKE '%.emptyFolderPlaceholder' AND
        (NEW.metadata->>'mimetype' LIKE 'video/%')
    command: -i $MEDIA_1 -vf thumbnail,scale=320:180 -frames:v 1 -f image2 -y $OUTPUT_PATH
    inputs: ["INPUT_FILE"]
    outputs: ["OUTPUT_FILE"]
    output_path: "{{userId}}/{{pipelineId}}/{{runId}}/thumbnails/320x180.jpg"
    editor:
      output: "jpg"
      preset: "fast"
      selectedCode: "custom"
      width: 320
      height: 180
    keep: true

  - id: "thumbnail_640x360"
    trigger:
      name: "handle_thumbnail_640x360"
      event: "INSERT"
      table: "storage.objects"
      condition: |
        NEW.bucket_id = 'video-uploads' AND
        NEW.name NOT LIKE '%.emptyFolderPlaceholder' AND
        (NEW.metadata->>'mimetype' LIKE 'video/%')
    command: -i $MEDIA_1 -vf thumbnail,scale=640:360 -frames:v 1 -f image2 -y $OUTPUT_PATH
    inputs: ["INPUT_FILE"]
    outputs: ["OUTPUT_FILE"]
    output_path: "{{userId}}/{{pipelineId}}/{{runId}}/thumbnails/640x360.jpg"
    editor:
      output: "jpg"
      preset: "fast"
      selectedCode: "custom"
      width: 640
      height: 360
    keep: true

  # ---- Video transcodes ----
  - id: "transcode_480p"
    trigger:
      name: "handle_transcode_480p"
      event: "INSERT"
      table: "storage.objects"
      condition: |
        NEW.bucket_id = 'video-uploads' AND
        NEW.name NOT LIKE '%.emptyFolderPlaceholder' AND
        (NEW.metadata->>'mimetype' LIKE 'video/%')
    command: -i $MEDIA_1 -c:v libx264 -crf 23 -preset medium -vf scale=-2:480 -c:a aac -b:a 128k -movflags +faststart -f mp4 -y $OUTPUT_PATH
    inputs: ["INPUT_FILE"]
    outputs: ["OUTPUT_FILE"]
    output_path: "{{userId}}/{{pipelineId}}/{{runId}}/videos/480p.mp4"
    editor:
      output: "mp4"
      preset: "medium"
      selectedCode: "custom"
      width: 854
      height: 480
      compressionLevel: 23
    keep: true

  - id: "transcode_720p"
    trigger:
      name: "handle_transcode_720p"
      event: "INSERT"
      table: "storage.objects"
      condition: |
        NEW.bucket_id = 'video-uploads' AND
        NEW.name NOT LIKE '%.emptyFolderPlaceholder' AND
        (NEW.metadata->>'mimetype' LIKE 'video/%')
    command: -i $MEDIA_1 -c:v libx264 -crf 23 -preset medium -vf scale=-2:720 -c:a aac -b:a 128k -movflags +faststart -f mp4 -y $OUTPUT_PATH
    inputs: ["INPUT_FILE"]
    outputs: ["OUTPUT_FILE"]
    output_path: "{{userId}}/{{pipelineId}}/{{runId}}/videos/720p.mp4"
    editor:
      output: "mp4"
      preset: "medium"
      selectedCode: "custom"
      width: 1280
      height: 720
      compressionLevel: 23
    keep: true

  - id: "transcode_1080p"
    trigger:
      name: "handle_transcode_1080p"
      event: "INSERT"
      table: "storage.objects"
      condition: |
        NEW.bucket_id = 'video-uploads' AND
        NEW.name NOT LIKE '%.emptyFolderPlaceholder' AND
        (NEW.metadata->>'mimetype' LIKE 'video/%')
    command: -i $MEDIA_1 -c:v libx264 -crf 23 -preset medium -vf scale=-2:1080 -c:a aac -b:a 128k -movflags +faststart -f mp4 -y $OUTPUT_PATH
    inputs: ["INPUT_FILE"]
    outputs: ["OUTPUT_FILE"]
    output_path: "{{userId}}/{{pipelineId}}/{{runId}}/videos/1080p.mp4"
    editor:
      output: "mp4"
      preset: "medium"
      selectedCode: "custom"
      width: 1920
      height: 1080
      compressionLevel: 23
    keep: true

  # ---- Image thumbnail ----
  - id: "image_thumbnail_320x320"
    trigger:
      name: "handle_image_thumbnail"
      event: "INSERT"
      table: "storage.objects"
      condition: |
        NEW.bucket_id = 'video-uploads' AND
        NEW.name NOT LIKE '%.emptyFolderPlaceholder' AND
        (NEW.metadata->>'mimetype' LIKE 'image/%')
    command: -i $MEDIA_1 -vf scale=320:320:force_original_aspect_ratio=decrease,pad=320:320:(ow-iw)/2:(oh-ih)/2 -q:v 85 -f image2 -y $OUTPUT_PATH
    inputs: ["INPUT_FILE"]
    outputs: ["OUTPUT_FILE"]
    output_path: "{{userId}}/{{pipelineId}}/{{runId}}/thumbnails/320x320.jpg"
    editor:
      output: "jpg"
      preset: "fast"
      selectedCode: "custom"
      width: 320
      height: 320
    keep: true

render:
  project_name: "video-processing"
  status: "queued"
  public: false

The keep: true flag tells the transpiler to send the output directly to the final bucket (video-processed). Steps without keep (or with keep: false) use next_bucket for chaining. The runId is computed deterministically from the input file name (using mode: "deterministic" and template: "{baseFilename}"). This ensures all parallel steps compute the same run ID, grouping all outputs for a single upload under one folder.

Running the Transpiler

Download the transpiler and the SVG generator:

Run the transpiler to generate the migration files:

Add the --svg flag to also generate a visual graph of your pipeline:

The output will be:

Apply the migration to your Supabase database:

Visualising the Pipeline

The generated SVG gives you a clear overview of your pipeline. Steps marked with KEEP are green – their outputs are permanently stored in the final bucket. Edges are labelled with the destination bucket (video-processed), making it easy to see where the processed files end up.

In the graph above, all steps trigger on the same video-uploads bucket and output directly to video-processed. This parallel execution reduces latency and keeps the pipeline simple. The edge labels now correctly show video-processed as the destination bucket.

Processing Logic Explained

When a file is uploaded to video-uploads/{userId}/, the pipeline:

1. Identifies the media type — video or image (via MIME type in the trigger condition).
2. Fires all matching triggers — video steps fire for video files, the image step fires for image files.
3. Computes the run ID — using the file name (deterministic mode), all steps get the same runId.
4. Builds the exact FFmpeg commands — each step has its own command defined in the YAML.
5. Creates a render job in the existing render table with the commands in the data column.
6. Pushes a job to the render queue with the commands payload.
7. Sends a notification via pg_notify.
8. The ffmpeglab-runner picks up the job, resolves the $MEDIA_1 and $OUTPUT_PATH placeholders, and executes the commands.

Because all steps share the same runId, all outputs are written to video-processed/{userId}/video-pipeline/{runId}/thumbnails/ and .../videos/, keeping everything together.

Exact FFmpeg Commands

The YAML steps define the following FFmpeg commands using placeholders:

• $MEDIA_1 — The path to the downloaded input file (resolved by the runner).
• $OUTPUT_PATH — The temporary path for the output file (resolved by the runner).

1. Video Thumbnails

2. Video Transcoding

3. Image Thumbnail

Configure ffmpeglab-runner

The runner needs to be configured to poll the render queue and execute the provided FFmpeg commands. The transpiler uses the existing render queue.

Monitor the Pipeline

You can monitor the pipeline using SQL queries and notifications.

Customising the Pipeline

Add a New Thumbnail Size

Add a new step with the desired dimensions:

Add a New Resolution

Sequential Pipelines (e.g., Audio)

For pipelines that must run in order, use next_bucket and omit keep (or set keep: false).

Frequently Asked Questions (FAQ)

Final Word

You now have the Ultimate Video Onboarding Pipeline — a fully automated, event‑driven media processing system that turns a single upload into a complete media package. With PostgreSQL triggers, pgmq, and Supabase Storage, you get:

• Automatic thumbnails for videos and images
• Multiple video resolutions for different devices
• Real‑time notifications for your users
• Per‑user isolation with RLS policies
• Durable job queuing with pgmq
• Full observability with monitoring views and logs
• Exact FFmpeg commands ready to use
• A declarative YAML definition that you can version and reuse
• Per‑run grouping via deterministic runId – all outputs for one upload stay together.

The pipeline is production‑ready, scalable, and configurable. It uses the existing render and logpiece tables from the FFmpegLab server, so there are no table conflicts — just pure, automated media processing.