AI Video Reference

Use the AI video hooks to submit long-running video jobs, poll for completion, and play back hosted MP4 results. Video generation is job-based so apps can survive reloads, slow model runs, and flaky networks.

Job-Based Video Generation

Video generation uses two hooks:

useSubmitVideoJob submits a request and returns a jobId.
useVideoJobStatus polls that job until it completes or fails.

`useSubmitVideoJob`

Hook Options

onError ((error: Error) => void): Optional error callback.

Returns

submitVideo (function): Submit a video generation job.
isSubmitting (boolean): Whether submission is in progress.
error (Error | null): Current submission error.
clearError (() => void): Clear the current error.

`submitVideo` Input

prompt (string, required): Description of the motion or scene.
image (File, { file: File }, or { url: string }): Source image to animate, or first frame when using lastFrame.
video (File, { file: File }, or { url: string }): Source video to edit.
lastFrame (File, { file: File }, or { url: string }): Final frame for interpolation; requires image.
referenceImages (FileList, File[], or an array of file or URL objects): Style or subject references.
extendVideo (File, { file: File }, or { url: string }): Video to continue.
aspectRatio ("16:9" or "9:16"): Common landscape or portrait output ratio.
durationSeconds (number): Duration for generation workflows. Do not use it with video or extendVideo.
quality ("low", "medium", or "high"): Optional quality tier. Omit it unless the user requests it.
generateAudio (boolean, default false): Whether to request generated audio.
model (string): Model ID for the request.

`useVideoJobStatus`

Arguments

jobId (string | null): Job to poll, or null to disable polling.
options (object): Optional polling configuration.

Status Options

pollInterval (number, default 10000): Polling interval in milliseconds.
enabled (boolean, default true): Whether polling is enabled.
onComplete ((result: { url: string }) => void): Called once when the final URL is available.
onError ((error: string) => void): Called once when the job fails.

Returns

status ("pending", "processing", "completed", "failed", or null): Current job status.
resultUrl (string | null): Completed hosted video URL.
error (string | null): Current job error.
createdAt (string | null): Job creation timestamp.
completedAt (string | null): Job completion timestamp.
refetch (() => void): Trigger an immediate status fetch.

Generation Workflows

Choose one workflow per submitVideo call:

Text to video: prompt.
Animate an image: prompt and image.
Interpolate between frames: prompt, image, and lastFrame.
Reference-guided generation: prompt and referenceImages.
Extend a video: prompt and extendVideo.
Continue through frame extraction: extract the last frame with useVideoFrame, then pass it as image.
Edit a video: prompt and video.

Use lastFrame only with image. Treat image, referenceImages, and extendVideo as alternative starting points. durationSeconds applies only to generation workflows and must not be used with video or extendVideo.

Persist and Poll Example

Import both hooks from the AI hook module:

import { useSubmitVideoJob, useVideoJobStatus } from '@/hooks/use-ai'
import { usePersistentItem } from '@/hooks/use-persistent-item'

export default function App() {
  const [prompt, setPrompt] = React.useState('')
  const [jobId, setJobId] = usePersistentItem<string | null>('videoJobId', null)
  const [videoUrl, setVideoUrl] = usePersistentItem<string | null>(
    'videoUrl',
    null,
  )
  const [videoError, setVideoError] = React.useState<string | null>(null)
  const { submitVideo, isSubmitting, error: submitError } = useSubmitVideoJob()

  const job = useVideoJobStatus(jobId, {
    onComplete: ({ url }) => {
      setVideoUrl(url)
      setJobId(null)
    },
    onError: (message) => {
      setVideoError(message)
      setJobId(null)
    },
  })

  const isGenerating = job.status === 'pending' || job.status === 'processing'

  const handleGenerate = async () => {
    if (!prompt.trim()) return
    setVideoError(null)

    const result = await submitVideo({
      prompt: prompt.trim(),
      aspectRatio: '16:9',
      durationSeconds: 4,
    })

    if (result) setJobId(result.jobId)
  }

  const errorMessage = submitError?.message || videoError || job.error

  return (
    <div>
      <input
        value={prompt}
        onChange={(event) => setPrompt(event.target.value)}
        placeholder="Describe the video you want..."
      />
      <button onClick={handleGenerate} disabled={isSubmitting || isGenerating}>
        {isSubmitting
          ? 'Submitting...'
          : isGenerating
            ? 'Generating...'
            : 'Generate video'}
      </button>
      {errorMessage ? <p className="text-destructive">{errorMessage}</p> : null}
      {videoUrl ? (
        <video src={videoUrl} controls className="w-full rounded-lg" />
      ) : null}
    </div>
  )
}

Video Utilities

The platform also includes client-side video helpers for workflows around generated videos. These hooks process media locally in the browser and return local File objects that work with useDownload, useFileUpload, or video generation inputs.

All three utilities accept hosted URLs and local media. URL sources must be same-origin or CORS-enabled; platform-hosted media works automatically. Only one stitch or audio-overlay operation can run at a time.

`useVideoFrame`

Extract a still frame from a video URL, Blob, File, or { url } object.

Hook Options

onError ((error: Error) => void): Optional custom error handler.
onFinish ((frame: File) => void): Called after successful extraction.

Returns

extractFrame (function): Extract a frame and resolve to a File or undefined.
isLoading (boolean): Whether extraction is in progress.
error (Error | null): Current extraction error.
clearError (() => void): Clear the current error.

`extractFrame` Input

source (string, Blob, File, or { url: string }, required): Video source.
time (number, "first", or "last"): Frame position.
format ("image/png" or "image/jpeg"): Output image format.
quality (number): Image quality.

import { useVideoFrame } from '@/hooks/use-video-frame'

const { extractFrame, isLoading, error, clearError } = useVideoFrame()

const frameFile = await extractFrame({
  source: previousVideoUrl,
  time: 'last',
  format: 'image/png',
})

time can be "first", "last", or a timestamp in seconds. Use the returned File directly as the image for the next submitVideo call.

`useVideoStitch`

Concatenate two or more clips into one MP4 file.

Hook Options

onError ((error: Error) => void): Optional custom error handler.
onFinish ((video: File) => void): Called after successful stitching.
onProgress ((progress: number) => void): Called with progress from 0 through 1.

Returns

stitch (function): Concatenate clips and resolve to an MP4 File or undefined.
isProcessing (boolean): Whether stitching is in progress.
progress (number): Progress from 0 through 1.
error (Error | null): Current stitching error.
cancel (() => void): Cancel the current operation.
clearError (() => void): Clear the current error.

`stitch` Input

sources (array of string, Blob, File, or { url: string }, required): Two or more clips in output order.
width (number): Output width; defaults to the first video's width.
height (number): Output height; defaults to the first video's height.
muted (boolean, default false): Whether to discard source audio.

import { useDownload } from '@/hooks/use-download'
import { useVideoStitch } from '@/hooks/use-video-stitch'

const { stitch, isProcessing, progress, cancel } = useVideoStitch()
const { download } = useDownload()

const result = await stitch({
  sources: [firstClipUrl, secondClipUrl],
  muted: false,
})

if (result) download(result)

All clips should use the same aspect ratio for best results. The returned File already includes the .mp4 extension, so pass it directly to download(result).

`useVideoAudioOverlay`

Replace or mix audio on top of a video.

Hook Options

onError ((error: Error) => void): Optional custom error handler.
onFinish ((video: File) => void): Called after a successful overlay.
onProgress ((progress: number) => void): Called with progress from 0 through 1.

Returns

overlay (function): Overlay audio and resolve to an MP4 File or undefined.
isProcessing (boolean): Whether processing is in progress.
progress (number): Progress from 0 through 1.
error (Error | null): Current overlay error.
cancel (() => void): Cancel the current operation.
clearError (() => void): Clear the current error.

`overlay` Input

video (string, Blob, File, or { url: string }, required): Video source.
audio (string, Blob, File, or { url: string }, required): Audio source.
audioVolume (number, default 1): Overlay volume from 0 through 1.
videoVolume (number, default 0): Original video volume from 0 through 1.
audioOffset (number, default 0): Seconds to skip at the start of the audio.
width (number): Output width; defaults to the video's width.
height (number): Output height; defaults to the video's height.

import { useVideoAudioOverlay } from '@/hooks/use-video-audio-overlay'

const { overlay, isProcessing, progress, cancel } = useVideoAudioOverlay()

const result = await overlay({
  video: videoUrl,
  audio: speechUrl,
  audioVolume: 1,
  videoVolume: 0,
})

videoVolume defaults to 0, so the overlay replaces the original audio. Set it above 0 to mix both tracks. Audio longer than the video is truncated; when it is shorter, the remainder of the video plays without overlay audio. The returned File already includes the .mp4 extension.

Model Selection

When no model is specified, the platform chooses an efficient default model for the user's plan. To expose model choice, pass a model ID and build selectors from listVideoModels():

import { listVideoModels, useSubmitVideoJob } from '@/hooks/use-ai'

const models = listVideoModels()
const [modelId, setModelId] = React.useState<string | undefined>(undefined)
const { submitVideo } = useSubmitVideoJob()

await submitVideo({
  prompt: 'A drone shot over mountains',
  model: modelId,
})

listVideoModels() returns model objects with id, displayName, provider, tier, description, and capabilities. Video model capabilities include:

image_input
reference_images
reference_videos
reference_audio
video_extension
video_editing
last_frame

Prompt-only text-to-video is implicit for video models and is not listed as a capability. Let the server validate exact combinations such as duration, aspect ratio, and incompatible inputs.

Best Practices

Persist the jobId until completion so polling survives reloads.
Use resultUrl for playback and persistence.
Treat job.status as the source of truth for loading UI.
Use onComplete and onError to copy terminal state into your own durable state.
Choose one video task per submission instead of mixing incompatible inputs.
Prefer frame extraction for cross-model continuation, then generate the next clip from the extracted File.
Use useVideoStitch to assemble multi-shot workflows into a downloadable file.
Use useVideoAudioOverlay for narration, background music, or replacing original audio.
Include subject, action, style, camera motion, composition, focus, and ambiance in video prompts.

AI Video Reference

Hook Options

Returns

submitVideo Input

Arguments

Status Options

Returns

useVideoFrame

Hook Options

Returns

extractFrame Input

useVideoStitch

Hook Options

Returns

stitch Input

useVideoAudioOverlay

Hook Options

Returns

overlay Input

`submitVideo` Input

`useVideoFrame`

`extractFrame` Input

`useVideoStitch`

`stitch` Input

`useVideoAudioOverlay`

`overlay` Input