Processing video with WebCodecs and @remotion/media-parser

parseMedia() is able to extract tracks and samples from audio and video in a format that is suitable for usage with WebCodecs APIs.

Minimal example

The following snippet is a basic, non-production ready example of how to use parseMedia() with WebCodecs.

Reading video frames
tsx
// ⚠️ Simple, but non-production ready example
import {parseMedia, MediaParserOnAudioTrack, MediaParserOnVideoTrack} from '@remotion/media-parser';
 
const result = await parseMedia({
  src: 'https://parser.media/video.mp4',
  onVideoTrack: ({track}) => {
    const decoder = new VideoDecoder({
      output: console.log,
      error: console.error,
    });
 
    decoder.configure(track);
 
    return (sample) => {
      decoder.decode(new EncodedVideoChunk(sample));
    };
  },
});

The importance of queueing

Consider the following video processing pipeline, which converts a video, similarly to what convertMedia() does.

The pipeline consists of different steps, each of which have different speeds.
For example, the parsing of the media is much faster than the decoding of the video.

This can lead to a "traffic jam" in front of the video decoder, where many samples are queueing quickly.
Since this will cause memory to build up, it will negatively impact the performance of the page.

It makes sense to connect the whole pipeline together and throttle each step to it does not perform faster than the next step.
In the example above, six queues are required to be managed, with each one of them malfunctioning causing memory to run away.

How Remotion helps you work with WebCodecs

Since handling queueing correctly is a challenging task, we offer helpers that help you work with WebCodecs.

Remotion's primitives are designed to offer you functionality to implement queueing in a sensible way:

• In parseMedia(), callbacks are asnychronous, and as long as they don't resolve, the parsing process will not continue.
• The helper functions createAudioDecoder() and createVideoDecoder() offer you a helper method waitForQueueToBeLessThan() which you can await.
• For processing video and audio frames as well as encoding video and audio, we don't offer helpers yet, but hope to do so soon.

Practical considerations

If you use parseMedia() with codecs, make the following considerations for your implementation.

Check if browser has VideoDecoder and AudioDecoder

As of May 2025, all major browsers support VideoDecoder and AudioDecoder except Safari, which has support for VideoDecoder only.
AudioDecoder is supported in Safari Technology Preview, meaning that soon, all major browsers will support WebCodecs fully.

Still, it is a good idea to check if the browser supports the decoders.
You can opt to skip a track by returning null from the callback.

Rejecting samples
tsx
import type {MediaParserOnAudioTrack, MediaParserOnVideoTrack} from '@remotion/media-parser';
 
const onVideoTrack: MediaParserOnVideoTrack = ({track}) => {
  if (typeof VideoDecoder === 'undefined') {
    return null;
  }
 
  const videoDecoder = new VideoDecoder({
    output: console.log,
    error: console.error,
  });
  // ...
};
 
const onAudioTrack: MediaParserOnAudioTrack = ({track}) => {
  if (typeof AudioDecoder === 'undefined') {
    return null;
  }
 
  const audioDecoder = new AudioDecoder({
    output: console.log,
    error: console.error,
  });
  // ...
};

Check if the browser supports the codec

Not all browsers support all codecs that parseMedia() emits.
The best way is to use AudioDecoder.isConfigSupported() and VideoDecoder.isConfigSupported() to check if the browser supports the codec.
These are async APIs, fortunately onAudioTrack and onVideoTrack allow async code as well.

Checking if the browser supports the codec
tsx
import type {MediaParserOnAudioTrack, MediaParserOnVideoTrack} from '@remotion/media-parser';
 
const onVideoTrack: MediaParserOnVideoTrack = async ({track}) => {
  const videoDecoder = new VideoDecoder({
    output: console.log,
    error: console.error,
  });
 
  const {supported} = await VideoDecoder.isConfigSupported(track);
  if (!supported) {
    return null;
  }
 
  // ...
};
 
const onAudioTrack: MediaParserOnAudioTrack = async ({track}) => {
  const audioDecoder = new AudioDecoder({
    output: console.log,
    error: console.error,
  });
 
  const {supported} = await AudioDecoder.isConfigSupported(track);
  if (!supported) {
    return null;
  }
 
  // ...
};

note

Perform these checks in addition to the previously mentioned ones.

Error handling

If an error occurs, you get the error in the error callback that you passed to the VideoDecoder or AudioDecoder constructor.
The decoder state will switch to "closed", however, you will still receive samples.

If the decoder is in "closed" state, you should stop passing them to VideoDecoder.

Error handling
tsx
import type {MediaParserOnVideoTrack} from '@remotion/media-parser';
 
const onVideoTrack: MediaParserOnVideoTrack = async ({track}) => {
  const videoDecoder = new VideoDecoder({
    output: console.log,
    error: console.error,
  });
 
  return async (sample) => {
    if (videoDecoder.state === 'closed') {
      return;
    }
  };
};

note

• The same logic goes for AudioDecoder.
• You should still perform the checks previously mentioned, but they are omitted from this example.

Handling rotation

WebCodecs do not seem to consider rotation.
For example, this video recorded with an iPhone has metadata that it should be displayed at 90 degrees rotation.

VideoDecoder is not able to rotate the video for you, so you might need to do it yourself, for example by drawing it to a canvas.
Fortunately parseMedia() returns the rotation of the track:

Handling stretched videos
tsx
import type {MediaParserOnVideoTrack} from '@remotion/media-parser';
 
const onVideoTrack: MediaParserOnVideoTrack = async ({track}) => {
  console.log(track.rotation); // -90
  return null;
};

See here for an example of how a video frame is turned into a bitmap and rotated.

Understanding the different dimensions of a video

As just mentioned, some videos might be stretched or rotated.
In an extreme case, it is possible that you stumble opon a video that has three different dimensions.

Handling stretched videos
tsx
import type {MediaParserOnVideoTrack} from '@remotion/media-parser';
 
const onVideoTrack: MediaParserOnVideoTrack = async ({track}) => {
  console.log(track);
  // {
  //   codedWidth: 1440,
  //   codedHeight: 1080,
  //   displayAspectWidth: 1920,
  //   displayAspectHeight: 1080,
  //   width: 1080,
  //   height: 1900,
  //   ...
  // }
 
  return null;
};

The meaning of it is:

• codedWidth and codedHeight are the dimensions of the video in the codec's internal format.
• displayAspectWidth and displayAspectHeight are scaled dimensions of how the video should be displayed, but with rotation not yet applied.
  note

  These are not necessarily the actual dimensions of how a video is presented to the user, because rotation is not yet applied.
  The fields are named like this because they correspond with what should be passed to new EncodedVideoChunk().

• width and height are the dimensions of the video how it would be displayed by a Player.

Google Chrome quirks

We find that as of now, AudioDecoder.isConfigSupported() are not 100% reliable. For example, Chrome marks this config as supported, but then throws an error nonetheless.

tsx
const config = {codec: 'opus', numberOfChannels: 6, sampleRate: 44100};
console.log(await AudioDecoder.isConfigSupported(config)); // {supported: true}
const decoder = new AudioDecoder({error: console.error, output: console.log});
decoder.configure(config); // Unsupported configuration. Check isConfigSupported() prior to calling configure().

Consider this in your implementation.

Why WebCodecs?

WebCodecs is the fastest way to decode videos in the browser.
WebAssembly solutions need to strip the CPU-specific optimizations and cannot benefit from hardware acceleration.

Using Remotion Media Parser and WebCodecs is at least 3 times faster than using WebAssembly solutions in any scenario we tested.

WebCodecs are built into the browser, meaning decoders and encoders do not need to be loaded.
Later in 2025, WebCodecs will be supported in all major browsers.

With @remotion/webcodecs

@remotion/webcodecs is a package that uses @remotion/media-parser to provide video processing in the browser.
It handles various browser quirks and hard implementation details for you.

• Convert, rotate and resize videos: convertMedia()
• Decode media: createVideoDecoder() and createAudioDecoder()

Reference implementation

A testbed with many different codecs and edge cases is available here.
Follow these instructions to run the testbed locally.

License reminder

Like Remotion itself, this package is licensed under the Remotion License.
TL;DR: Individuals and small teams can use this package, but teams of 4+ people need a company license.