Analyze Script Workflow

End-to-end pipeline that transforms a user's script into a complete storyboard with images, motion video, and music. For what an edit makes stale afterwards, use the interactive Dependency graph.

High-Level Overview#

Timing source: Measured from local Cloudflare Workflows runs (Workerd via Miniflare) for a 9-scene run. As of #929, Phase 4 runs sequentially — frame images render first, then motion/music prompts — because the motion-prompt pass is conditioned on the actual rendered starting frame (passed to the LLM as a vision input). This trades the prior image∥prompt parallelism for image-grounded motion.

Triggering Flow#

The pipeline starts from server handlers in src/sequences/sequences.fn.ts:

1. createSequenceFn — Creates a new sequence record, then calls triggerWorkflow('/storyboard', input)
2. updateSequenceFn — If script, style, aspect ratio, or analysis model changed, triggers the same workflow
3. retryStoryboardFn — Retries a failed sequence (resets status to processing, re-triggers)

All three use triggerWorkflow() from src/platform/server/workflow/client.ts, which:

• Resolves the Cloudflare Workflows binding for the trigger path via TRIGGER_TO_BINDING (src/platform/server/workflow/trigger-bindings.ts) — e.g. /storyboard → STORYBOARD_WORKFLOW
• Calls binding.create({ id, params: body }) to start a durable workflow instance in-process (Workerd locally, same runtime as production — no HTTP webhook, no QStash). options.deduplicationId becomes the instance id, making a trigger idempotent
• Returns the workflow instance id, persisted as workflowRunId on the relevant DB row for tracking

Input shape (StoryboardWorkflowInput):

Storyboard Workflow#

File: src/sequences/server/workflows/storyboard-workflow.ts

The storyboard workflow (StoryboardWorkflow, a WorkflowEntrypoint extending OpenStoryWorkflowEntrypoint) validates data, generates a poster image, then delegates to the analyze-script workflow. Each unit of work runs inside step.do('name', …) so the Workflows engine checkpoints and auto-retries it.

Step: verify-clear-and-start-processing

1. Validates auth via validateSequenceAuth()
2. Loads sequence with getSequenceForUser() — checks script and style exist
3. Loads and parses the style config
4. Deletes all existing frames for the sequence
5. Sets sequence status to processing
6. Returns resolved models: analysisModelId, imageModel, videoModel

Step: generate-poster

• Generates a poster image from the script+title+style for the video player empty state
• Non-critical — failures are logged and swallowed

Steps: upload-poster → save-poster

• upload-poster streams the provider image into R2 (uploadPosterToStorage) so the stored URL is the origin-relative /r2/ path and never expires (#1117). Also non-critical: an upload failure falls back to the provider URL
• save-poster writes posterUrl on the sequence and emits generation.poster:ready with the URL

Then spawns the AnalyzeScriptWorkflow child via spawnAndAwaitChild(ANALYZE_SCRIPT_WORKFLOW, …) (src/platform/server/workflow/await-child.ts) and awaits its completion. The child runs as its own durable instance with its own per-step retry budget.

After the analyze-script workflow completes, marks status as completed and emits generation.complete.

Analyze Script Workflow — Phase-by-Phase#

File: src/sequences/server/workflows/analyze-script-workflow.ts

This is the core orchestration workflow. It runs durable units via step.do(), spawns child workflows via spawnAndAwaitChild(), and uses Promise.all() / Promise.allSettled() to fan child workflows out in parallel. It reads its input from event.payload and its instance id from event.instanceId.

Phase 1: Scene Splitting (Streaming LLM)#

Sub-workflow: sceneSplitWorkflow (src/sequences/server/workflows/scene-split-workflow.ts)

Uses streaming LLM output to create scene rows progressively as scenes arrive. Shots exist only once a scene's shot-list entry has landed (#1593); each shot then gets a preview image, copied into R2.

Steps:

Since #1035 the split runs as two parallel LLM calls (sibling step.dos via Promise.all), both over the same line-gutter copy of the script, and the LLM never re-emits script text. #1486 adds a third call after slices exist, so a scene can own 1..N shots; since #1585 that call also carries every spoken line, so originalScript.dialogue comes from it (the slice regex is only a streaming preview):

1. scene-splitting-stream — the scenes call, a boundary-annotation contract: the model returns boundaries[] = { hintLine, quote }, and boundary-split.ts resolves each verbatim quote to a raw offset (exact → normalized → fuzzy, monotonic cursor) and slices the ORIGINAL script — extracts are byte-verbatim adjacent substrings (concat(slices) === script, asserted). The stream is fed through createStreamingSceneParser():
   • On each finalized boundary: persists the scene row + split script version, emits generation.scene:new. No shot row (#1593): the rail shows the scene as "listing shots…" until its shot-list entry lands
   • On title detection: updates the sequence title, emits generation.updated
   • Excessive anchor repairs → one retry with feedback; a second degraded result keeps the first-pass LLM scenes. Dropped boundaries are logged and emitted as a non-fatal generation.error
   • A cut inside one location/beat is not a new scene (the ONE SHOT RULE is gone)
2. scene-bibles — the bibles call: { characterBible[], locationBible[], elementBible[] }. Location/element firstMention is { text, lineNumber } on the wire; the owning scene id is derived server-side from the gutter line. A character entry carries voiceOnly (#1585): a narrator or off-screen voice gets a row but no sheet, no talent match, and no place in the still prompt. After the join, scene continuity tags are canonicalized onto bible tags (tag-reconcile.ts) since two independent calls can disagree.
3. scene-shot-list-1..K — lists 1..N structured shots inside each resolved slice. The scenes go out in batches of SHOT_LIST_BATCH_SCENES (8), one step.do per batch, all batches concurrent (Promise.all): a feature-length paste is a dozen bounded calls, not one call at the model's output cap, and one failed batch replays alone. Each call streams (runStructuredCall + onAccumulated): the accumulated text is partial-parsed at most once per PARSE_COALESCE_CHARS, and every scene entry that has settled (settledPrefix — a later entry has started, or the stream ended) is landed on the spot via persistSceneShots: scene row upserted (stable id via orderIndex), its shots allocated (attachSceneShots) and upserted on (sceneId, shotNumber), tail rows trimmed with deleteFromShotNumber, generation.shot:created emitted per shot, and a preview fired per shot — its spec text when the scene has 2+ shots, else the slice (fire-and-forget via triggerWorkflow, deduplicated per instance + shot, so a step replay is idempotent; the image workflow copies the preview into R2 and records it as a kind: 'preview' variant). Mid-stream entries match on sceneNumber only; the validated final payload is the authority — attachShotLists throws on an omitted scene and lands anything the stream did not (a positional match, a provider that sent no partial chunks). The credit deduction sums the batches. Each call (shotListPassResultSchema, union-free), each shot carrying the dialogue spoken in it, speakers spelled as the cast list spells them. Length is per scene (#1593): a scene's running time is its script label (metadata.durationSeconds, from Scene N — Xs) and its shots divide it. Enhance writes scenes only — it no longer labels shots or knows the video model's clip grid (#1621), so the shot-list pass is the only place shot count/durations are decided. The prompt gives each scene a shots: budget: N to M = at least one shot per longest clip, at most one per shortest (minShotsForScene / maxShotsForScene; up to M when the floor is 1, exactly N when floor equals cap). allocateSceneShots then spreads the label over the returned shots with allocateClipDurations on the model grid (a lone shot takes the whole label; a residual the grid cannot reach lands on the last shot) and clamps every shot to the model's longest clip — a pass that sends too few shots leaves the scene short, never ending on a 14-minute "clip". No film-wide target enters the run — sequences.targetDurationSeconds steers Enhance and the credit estimate only. attachShotLists rebuilds each scene's originalScript.dialogue from the shots with every line stamped shotNumber; dialogueForShot hands each clip only its own lines. Failure — no payload, or a scene the pass omits — fails the run: a one-shot fallback would silently leave scenes on the regex preview, which is empty for prose. deriveShots assembles visual/motion prompts from scene continuity + shot specs for every clip of a 2+ shot scene (#1517, see Phases 3–4); a 1-shot scene still takes the existing LLM prompt path, byte-identical. Stage 1 still renders one video clip per shot (no in-clip packing).
4. reconcile-shots / persist-scenes — reconcile stamps the title + workflow and assembles the result (scenes, shotMapping concatenated from the batches); persist-scenes is the replay-safe re-upsert of the final scene set + shot links (idempotent on (sceneId, shotNumber)), trimming orphan tail rows if a re-analyze produced fewer scenes.
5. deduct-llm-credits-scene-splitting / deduct-llm-credits-scene-bibles / deduct-llm-credits-scene-shot-list — one credit deduction per LLM call

• Prompts: phase/scene-splitting-boundaries-chat + phase/scene-bibles-chat + phase/scene-shot-list-chat
• Variables: { aspectRatio, script, elements } (script is sanitized and line-guttered); shot-list gets { scenes, style, characters } (formatted slices with their duration: + shots: budget lines, director style, the cast list with voice-only entries marked)
• Response schemas: sceneSplitScenesResultSchema + sceneSplitBiblesResultSchema + shotListPassResultSchema — all under the ~3KB Anthropic strict-output grammar budget enforced by response-schema-budget.test.ts
• Output: { scenes[], title, shotMapping[], characterBible[], locationBible[], elementBible[] } — shotMapping maps each analysis shot (analysisSceneId + shotNumber) to shotId/frameId used throughout remaining phases. A 1-shot scene still has exactly one mapping row.

Phase 2: Casting Characters & Locations (Parallel Sub-Workflows)#

After scene splitting, two child workflows run in parallel via Promise.all over spawnAndAwaitChild(...):

Talent Matching Workflow (src/cast/server/workflows/talent-matching-workflow.ts):

Bibles already exist from Phase 1. This workflow only matches.

1. Uses input.characterBible from scene-split (no character-extraction LLM), minus voice-only entries (#1585): a narrator has no face to cast, so it is never offered to the matcher, and build-matches drops any match that names one
2. Talent matching (skipped if no suggestedTalentIds):
   • Loads talent records from DB by IDs
   • LLM matches characters to talent
   • Deduplicates matches (each talent/character used once), emits generation.talent:matched
3. Returns: { characterBible, matches: talentCharacterMatches }

Location Matching Workflow (src/cast/server/workflows/location-matching-workflow.ts):

1. Uses input.locationBible from scene-split (no location-extraction LLM)
2. Location matching (skipped if no suggestedLocationIds):
   • Loads library locations from DB by IDs
   • LLM matches locations to library entries (requires confidence >= 0.5)
   • Deduplicates matches, emits generation.location:matched
3. Returns: { locationBible, matches: libraryLocationMatches }

Phase 3: References & Prompts (Parallel Sub-Workflows)#

Three child workflows spawned in parallel via Promise.all over spawnAndAwaitChild(...):

Character Bible Workflow (src/cast/server/workflows/character-bible-workflow.ts):

• Creates the characters DB rows (upsert on (sequenceId, characterId); the Script stage already made them sheet-less)
• Generates a reference sheet image for each on-screen character (one CharacterSheetWorkflow child per character, in parallel); a failed child fails the whole run (#939)
• A voice-only character (#1585) gets no child: its row is created completed with no sheet version, it is left out of the billed sheet count, and it never reaches the still prompt or the reference images. The motion prompt still sees it, for delivery
• Uses talent match images as reference when available
• Uploads sheets to R2 storage

Location Bible Workflow (src/cast/server/workflows/location-bible-workflow.ts):

• Inserts location records into DB from location bible
• Generates establishing-shot reference images for each location (parallel)
• Uses library location reference images when matched
• Uploads to R2 storage, updates DB

Visual Prompt Workflow (src/lib/workflows/visual-prompt-workflow.ts):

• Delegates to visualPromptSceneWorkflow per 1-shot scene (parallel via spawnAndAwaitChild)
• Each such scene gets an LLM call that generates fullPrompt and negativePrompt. Scene continuity is authored in Phase 1 and is not re-emitted here.
• A 2+ shot scene is skipped here (#1517): every one of its clips — the head included — gets its start-frame prompt assembled by deriveShots (scene context + the shot's framing / start state), which analyze-script writes to frame_prompt_versions in the persist-derived-visual-prompts step at the top of Phase 4 (derivedShotForItem in shot-work-items.ts is the one predicate; it is null on the 1-shot path). The head's assembled prompt also stands in as that scene's visual grounding for the music prompt.
• Merges results back into scene objects

Phase 4: Frame Images, then Motion/Music Prompts (Sequential)#

As of #929, frame images render before motion/music prompts (the two ran in parallel previously). The motion-prompt pass is conditioned on the actual rendered starting frame — the per-scene motion workflow reads the frame's thumbnailUrl and passes it to the LLM as a vision input — so the still must exist first. The motion prompt then describes movement that continues that exact pose and composition, instead of guessing a plausible start from scene text alone. Music has no image dependency but rides along with motion in the same child workflow, so it inherits the wait (an accepted latency cost on the non-critical music artifact).

Frame Images Workflow (src/lib/workflows/frame-images-workflow.ts):

1. Builds per-scene character and location reference maps
2. For each scene, generates images with each selected model in parallel (one spawnAndAwaitChild per scene × model, gathered with Promise.allSettled):
   • Spawns the ImageWorkflow child (binding IMAGE_WORKFLOW) per scene per model
   • After each image completes, fires the shot-grid variant generation via triggerWorkflow('/variant-image', …) (fire-and-forget; its progress is tracked on frame.variantImageStatus)
3. Returns { imageUrls } — primary model's URL per scene. The primary still is persisted to frame.thumbnailUrl, which the motion-prompt pass reads next.

Motion + Music Prompts Workflow (src/motion/server/workflows/motion-music-prompts-workflow.ts):

1. Snap durations — Snaps scene durations to video model capabilities upfront so both motion prompts and music design see identical values
2. Parallel generation — Motion prompts and music design run simultaneously (parallel within this child; the child itself runs after frame images):
   • MotionPromptWorkflow — fans out one MotionPromptSceneWorkflow child per scene. Each per-scene call (motion-prompt-scene-workflow.ts) loads the rendered starting frame and, when the chosen analysis model accepts image input, attaches it to the LLM as a vision input (#929). The image's input-hash is folded into motionPromptInputHash, so re-rendering the still re-stales the motion prompt. When no image exists (render failed) or the model is text-only, it falls back to the text-only path. Only 1-shot scenes get a child (#1517): every clip of a 2+ shot scene takes its motion prompt from deriveShots (action + the single camera move + sound cue; reference-only prefixes the assembled framing, since no still fixes it) in the batch's derive-extra-shot-motion-prompts step, written to shot_prompt_versions with a derived-shot-motion input hash. Both derived writes emit the same generation.shot:updated the LLM children do.
   • MusicPromptWorkflow — Single LLM call classifying per-scene music requirements + generating unified prompt with tags
3. Merge — Combines motion prompts and music design into completeScenes[]
4. Returns: { completeScenes, musicPrompt, musicTags }

Phase 5: Motion + Music Generation (Conditional)#

Sub-workflow: motionBatchWorkflow (src/motion/server/workflows/motion-batch-workflow.ts)

Only runs if autoGenerateMotion is enabled, a video model is set, and images were generated. A single orchestrator handles:

1. Parallel generation — All frame motion child workflows + the optional music workflow spawned simultaneously (spawnAndAwaitChild under Promise.all)
2. Collect video URLs — Reads from DB (authoritative ordering by orderIndex)
3. Merge video — Concatenates all frame videos into one sequence video
4. Merge audio+video — If music was generated, muxes audio onto the merged video

Final: Return#

Returns the completeScenes array.

Data Flow: Scene Object Accumulation#

Each phase enriches the Scene object. The frame's metadata column is updated after visual prompts to persist intermediate results. Phase 1 creates frames progressively during streaming and triggers preview images for instant feedback.

Scene type fields (from src/shots/scene-analysis.schema.ts):

Real-Time Events#

Events emitted via Upstash Realtime on a per-sequence channel (getGenerationChannel(sequenceId)).

Error Handling#

Failure Handling (onFailure)#

Every workflow extends OpenStoryWorkflowEntrypoint (src/platform/server/workflow/base-workflow.ts), which wraps the workflow body: when runImpl throws, the base class builds a ScopedDb from the payload and invokes the subclass-supplied onFailure({ event, error, scopedDb }). The analyze-script onFailure:

1. Sanitizes the error via sanitizeFailResponse() — extracts inner errors from nested failure wrappers, maps known Cloudflare error codes (e.g., 1102 → "Worker exceeded memory limit"), and truncates messages over 500 characters
2. Updates sequence status to 'failed' with the error message
3. Emits generation.failed with the sanitized error

The base class deliberately skips onFailure when the engine aborts mid-run (a transient state the instance resumes from), so it doesn't mark user-facing rows failed for a retry that will succeed.

Child workflows (image, motion, music, character bible, location bible, talent matching, location matching, frame-images, motion-batch) each implement their own onFailure that updates the relevant record's status to 'failed'.

BytePlus ACR leases (#1361, #1531). MotionWorkflow and StudioGenerationWorkflow lease every still they register on Ark under an owner of motion:<instanceId> / studio:<instanceId> (assetLeaseOwner). Both release by that owner on success (step release-byteplus-asset-leases, whichever via the clip finally rendered on; a release that exhausts its retries is logged, never fails the rendered clip) and at the end of onFailure (inside the base class's retried emit-failure step), via scopedDb.bytePlusAssets.releaseOwner. That drops the run's own leases and any reservation it never finalized; another run's lease on the same still is untouched. A batch parent never releases for its children. Every claim and finalize renews all of the run's leases, so a still leased early cannot expire while a later still waits. A miss is a byteplus_assets row with NULL assetId (counts against capacity); another run for that still gets pending and the claim step retries. A full leased pool is NonRetryableError — it does not wait out the TTL. Abandoned reservations are takeable after the 45-minute TTL. Ingest per still is -url (fal key + fetchable URL) → -claim (the only retried-for-minutes step) → -evict (an already-deleted asset counts as done) → -slot → -wait → -create. Motion's parents (motion-batch, update-stale-shots) await a motion child for 90 minutes, and analyze-script awaits motion-batch for 120.

Retry Strategy#

Under Cloudflare Workflows, retries are configured per step.do() (and on the workflow class), not on the trigger — the legacy retries/retryDelay options on triggerWorkflow() are accepted for back-compat but are no-ops.

Per-scene fan-out (image, variant, motion) uses Promise.allSettled over spawnAndAwaitChild, so one scene's failure or timeout doesn't kill the rest of the batch — failures are collected and surfaced as a single error.

Cloudflare Workflows Durability#

• Each step.do() step is checkpointed by the Workflows engine. The workflow body replays from the top on every step callback; already-completed steps return their persisted result instead of re-executing, so on failure or restart execution effectively resumes from the last completed step. (This is why steps must be idempotent and why large blobs shouldn't be returned across a step boundary.)
• spawnAndAwaitChild() starts a child workflow instance (its own binding.create()) and awaits its result via a wake event (waitForEvent). The child is durable independently of the parent.
• No application-level concurrency gating — fal queues submissions server-side (IN_QUEUE doesn't count toward the cap, jobs are never rejected), and OpenRouter handles its own rate limits. (A past QStash-era attempt at gating via flowControl produced ghost slot leaks on cancel and PR-preview cross-contamination; see #725. Cloudflare Workflows likewise has no app-level gate.)

Key Files Reference#