Color tile captions

Add Grant's colour-tile captions to a talking-head video in Tella: one spoken word at a time in black type on a square-cornered tile that cycles lime, red, blue and pink, low in the frame, while the words that carry the point are pulled out as scaled-up keyword tiles across the top, behind the speaker, sized so a row fills the frame width and landing on the spoken word. A few keywords get a highlight that acts out the word: letters rising, a wipe, a typewriter, a blur-in, a strikethrough, a dashed outline, a shimmer, viewfinder brackets with a flash, or a moving shader fill. Built as editable text overlays by a script that measures the words with the font, plus small Tella Motion overlays for the drawn highlights. Use when asked for this caption or subtitle style, colourful word-by-word captions, coloured block or tile subtitles, captions with big or scaled-up keywords, keyword highlights, or captions like the Tella Skills launch video.

Use in Tella

Use it in Tella

  1. Open any video in Tella, then open the AI chat.
  2. Paste the skill:
    tellahq/user-skills/color-tile-captions-fgde86fj8u
  3. Then ask for what you want.

Keep a copy

Add this skill to your library to use it across videos and make it your own.

Instructions

What the AI chat follows when the skill is pasted, from its SKILL.md.

Colour-tile captions with scaled-up keywords and highlights

Word-by-word captions made of text overlays, not the video's own subtitles. Three layers:

  1. Small captions. One spoken word at a time, black on a bright square-cornered tile, centred low in the frame. The tile colour changes with every word.
  2. Keywords. The words that carry the point leave the small captions and appear as big tiles across the top of the frame, behind the speaker's head, each landing on its spoken word and holding after it.
  3. Highlights. A few keywords act out what they say (a strikethrough on "mistakes", a shader inside "shaders").

references/reference.mp4 is the original video (43 s, 360 px wide). Before the first run, pull a few frames from it with ffmpeg and look at them (view_image), e.g. at 1.7 s, 7 s, 15.8 s, 17.5 s, 24.6 s, 26.7 s and 31.5 s.

The look, in numbers

Sizes are for a 1080 px wide canvas; the script scales them by the canvas's shorter side.

Small captionKeyword
FontABC Oracle Medium (500), else Inter 600same
Size56 px (5.2% of the shorter side)up to 165 px (15.3%), reduced until the widest row fits
Text#000000#000000
Tilelime #B8FF00, red #FF2A2A, blue #2B8CFF, pink #FF2BD6; never the same colour twice runningsame palette; tiles on screen together all differ
ShapebackgroundShape: regular, cornerRadius: 0, backgroundPadding: 0.1same, backgroundPadding: 0.05
Placecentred, tile centre at 83.3% of the heighttop of the first row at 3.3% of the height; a row is centred, tiles 15 px apart, 50 px clear of each side; a second row sits 18 px under the first
Depthfrontbehind_presenter
Inscale from 0.9 at full opacity, 90 ms, slow_downscale from 0.8 and fade, 160 ms, slow_down
Outnone (the next word replaces it)scale to 0.85 and fade, 120 ms, accelerate; none when the clip's cut ends it
Timingfrom the word's start to the next word's start, at least 100 mseach word from its own start; the whole group leaves together 450 ms after its last word ends, or when the next keyword group starts, or at the clip's end

Words keep the transcript's capitals. Trailing commas and full stops are dropped; ? and ! stay. A keyword's word never also shows as a small caption, and small captions keep running underneath while a keyword holds.

Steps

  1. Read the video. get_timeline for the canvas size, the clips and their durations, and each clip's existing text overlays; get_transcript (words view) for every clip with speech. Do this after the cut is final: every time comes from the kept words. Skip clips with no words and clips converted to Tella Motion (text overlays don't show there). If a clip already has caption tiles from an earlier run, remove them before adding a new set.

  2. Get the font, twice. The overlays need the font's reference and the script needs the same font's file to measure words.

    • Look for ABC Oracle Medium with list_library (type: 'font'). If it is there, use {"fontId": "mi_…"} with weight 500 and download the item's file to the sandbox.
    • Otherwise use {"fontFamily": "Inter"} with weight 600 (or the font the user names) and say so. Catalog families are on Google Fonts: curl -sfL -o /workspace/Inter.ttf "https://github.com/google/fonts/raw/main/ofl/inter/Inter%5Bopsz%2Cwght%5D.ttf" (other families under ofl/<family>/). The script sets a variable font's weight axis from fontWeight.
  3. Choose the keywords. About one word in four. Don't ask the user to pick; choose, and list them in the reply so they can swap any.

    • The title or topic in the hook ("Tella" + "Skills").
    • A whole short question or punchline as one row ("What is a skill?", "Have fun!").
    • Every item of a spoken list, each as its own group in turn ("silences", "mistakes", "best shots"; "animations", "transitions", "text effects", "shaders").
    • The phrase a clip ends on ("creative things?", "saving hours", "big / and broad").
    • Never a function word on its own. At most 4 words in a row and 2 rows in a group. One group on screen at a time: the next group replaces the one before.
    • Two-word keywords go side by side in one row; a phrase that builds ("big … and broad") takes a second row.
    • Size is automatic. Give a less important phrase maxFontSize 100 to 125, and use fontSize or topPct (down to about 1) when the tile would cover too much of the speaker's face.
  4. Choose the highlights. Read references/highlights.md. Pick a highlight only where the word names something that can be shown, about one keyword in three at most, and no two Tella Motion highlights on screen together. Text-animation highlights are set in the spec (effect); drawn ones need hideText or hideBackground in the spec where the reference says so.

  5. Write the spec and run the script. scripts/build_captions.py documents the spec at its top: the canvas, the font file and reference, and per clip its clipId, durationMs, the transcript words exactly as get_transcript returned them (kept words only), and keywords as lists of word indexes per row.

    python3 $HOME/.agents/skills/color-tile-captions/scripts/build_captions.py /workspace/captions/spec.json --out /workspace/captions/out
    

    It writes one <clipId>.ops.json per clip (the operations array for apply_video_edits) and report.json with every keyword tile's box, tile rect, timing, colour and animation. Per clip options: captionCenterYPct (default 83.3; about 94.5 when a panel, b-roll or screen content fills the lower middle of the frame), captionStyle: "mono" (white on black tiles, for a clip where the colours fight the footage or when asked), textOverrides (show a word differently, e.g. a deliberate misspelling), and seed at the top level to reshuffle colours.

  6. Apply, one clip per call. apply_video_edits with that clip's operations. Keep the ids returned for the k<index> operations: the highlights need them.

  7. Build the drawn highlights as in references/highlights.md, from report.json. Shader sources are in shaders/.

  8. One caption layer. Render a frame with subtitles: true. If the video's own subtitles show as well, turn them off with update_video and say so.

  9. Check every clip with get_filmstrip at each keyword's mid-hold and at a few small captions, then fix what you find:

    • Every keyword reads. When the speaker's head hides more than a letter or two, lower maxFontSize, raise the row with topPct, or set that group's depth to front, and re-run that clip.
    • No tile shows a word on two lines and rows stay inside the frame. A wrapped or overflowing word means the font file does not match the font reference.
    • A keyword with a Tella Motion background shows its word for the whole hold, not only while it fades in and out.
    • Drawn highlights sit on their tiles, and move with them on the way in and out (step through the entrance at 50 ms).
  10. Report the counts (captions, keywords, highlights), the keywords you chose per clip, the font used, and anything you changed on the video's own subtitles.

Why the script

A text overlay's box is not fitted to its text: without dimensions it is 60% of the canvas wide and 25% tall, the tile is drawn around the text in the middle of that box, and a box that is too narrow wraps the word. So tiles can only be laid out edge to edge by measuring. The script uses what Tella's renderer draws, measured on rendered frames:

  • tile width = the text's advance width + 2 × (0.30 + backgroundPadding) × font size
  • tile height = (1.4 + 2 × backgroundPadding) × font size
  • the tile is centred in the overlay's box; the script makes each box 48 px wider than its tile on each side (40 px for small captions) and about 10 px taller top and bottom.

From the spec's word indexes it also works out every start and duration, the keyword font size that makes the widest row fit, row positions, colours, and the animations above.

Rules

  • Take every time from get_transcript read in the same turn. Re-run a clip after any cut in it.
  • To change one clip, remove that clip's caption overlays and apply its operations again rather than patching tiles one by one.
  • Keep the palette and the black type unless the user gives brand colours; then pass 3 to 5 bright colours as palette and keep the type black (or use white type only on dark tiles, by hand).
  • A Tella Motion graphic that replaces a tile's background goes under its text: after adding it, send the keyword's text overlay to the top of the stack (update_text_overlay zIndex: 999). Left the other way round, the word only shows for a few frames at the start and end.
  • Optional, and only when asked or when you are also laying out the video: in the hook, a hard-cut punch-in on each keyword (about 1.12×, then 1.25×), as in the reference's first two seconds.

Files

The other files in the skill, such as reference videos it copies from.