Derek Herman on web.dev

Media encryption

2021-07-05T00:00:00Z

In this section we are going to cover two different strategies for encrypting your media, and some practical examples on how to use them with FFmpeg and Shaka Packager. The two strategies for encryption we'll discuss are Clear Key and using a service like Google Widevine. Both strategies are a form of digital rights management (DRM) to control what users can do with your media. However, one is inherently less secure than the other due to the way keys are passed for authentication and is why a DRM service might make more sense.

The primary DRM services for the web are Google Widevine, Microsoft PlayReady and Apple FairPlay, but we will not be covering all of them in this article. However, if you are targeting all the modern browsers you are likely going to be using all three DRM services.

Conversion and encryption is done with these applications:

Clear Key encryption #

First, you should have a good understanding of what Clear Key is and isn't before using it. When you do not want to use an existing DRM service and feel basic encryption of you media is a viable option, you would use Clear Key. But, keep in mind that this type of encryption does not provide the same level of security as using one of the DRM services. This is because the key value pair is not encrypted under another key, unlike encrypted keys which are generated by a decryption key that is stored on a licence server. Additionally, Clear Key sends the key value pair as plain text, so while you are encrypting your media the key to decrypt it is not a secret.

Create a key #

You can use the same method to create a key for both DASH and HLS. Do this using OpenSSL. The following will create an encryption key made of 16 hex values.

openssl rand -hex 16 > media.key

Create an IV #

Next we can generate an initialization vector (IV).

openssl rand -hex 16
6143b5373a51cb46209cfed0d747da66

Encrypt with Clear Key #

The following example uses Shaka Packager with raw keys, where keys and key_ids are provided to Shaka Packager directly. Read the documentation for more examples.

For the key flag, use the key created earlier, which is stored in the media.key file. However, when entering it on the command line, be sure you've removed its whitespace. For the key_id flag, repeat the media.id value or use the IV value generated above.

packager \
  input=glocken.mp4,stream=audio,output=glockena.m4a \
  input=glocken.mp4,stream=video,output=glockenv.mp4 \
  --enable_fixed_key_encryption \
  --keys label=audio:key=INSERT_AUDIO_KEY_HERE:key_id=INSERT_AUDIO_KEY_ID_HERE,label=video:key=INSERT_VIDEO_KEY_HERE:key_id=INSERT_VIDEO_KEY_ID_HERE

Create a key information file #

To encrypt for HLS you need a key information file in addition to a key file. A key information file is a text file with the format below. It should have the extension .keyinfo. For example: encrypt.keyinfo.

key URI
key file path
private key

The key URI is where the media.key (created above will be located on your server. The key file path is its location relative to the key information file. Finally, the private key is the contents of the media.key file itself, or the IV you created before. For example:

https://example.com/keys/media.key
/path/to/media.key
6143b5373a51cb46209cfed0d747da66

Encrypt for HLS #

packager \
  'input=input.mp4,stream=video,segment_template=output$Number$.ts,playlist_name=video_playlist.m3u8' \
  'input=input.mp4,stream=audio,segment_template=output_audio$Number$.ts,playlist_name=audio_playlist.m3u8,hls_group_id=audio,hls_name=ENGLISH' \
  --hls_master_playlist_output="master_playlist.m3u8" \
  --hls_base_url="http://localhost:5000/"

This command will accept a key with either 16 or 32 characters.

ffmpeg -i myvideo.mov -c:v libx264 -c:a aac -hls_key_info_file encrypt.keyinfo myvideo.m3u8

Widevine encryption #

Now you know what Clear Key encryption is and when to use it. But, when should you use a DRM service for additional security? This is where Widevine, or another service, would be used to securely encrypt and decrypt your media. Widevine supports MPEG-DASH and HLS and is a DRM from Google. Widevine is used by the Google Chrome and Firefox web browsers, Android MediaDRM, Android TV, and other consumer electronics devices that use Encrypted Media Extensions and Media Source Extensions, where Widevine decrypts content.

Encrypt with Widevine #

Most of the examples in this article used Clear Key encryption. However, for Widevine you will want to replace the following options.

--enable_fixed_key_encryption \
--enable_fixed_key_decryption \
--keys label=:key=INSERT_KEY_HERE:key_id=INSERT_KEY_ID_HERE

Everything in the demultiplexer (demux) command except the name of your files and the --content-id flag should be copied exactly from the example. The --content-id is 16 or 32 random hex digits. Use the keys provided here instead of your own. Read the Shaka Packager documentation on using the Widevine Key Server for more examples.

Demux (separate) the audio and video, encrypt the new files, and output a media presentation description (MPD) file.

packager \
  input=tmp_glocken.mp4,stream=video,output=glocken_video.mp4 \
  input=tmp_glocken.mp4,stream=audio,output=glocken_audio.m4a \
  --enable_widevine_encryption \
  --key_server_url "https://license.uat.widevine.com/cenc/getcontentkey/widevine_test" \
  --content_id "fd385d9f9a14bb09" \
  --signer "widevine_test" \
  --aes_signing_key "1ae8ccd0e7985cc0b6203a55855a1034afc252980e970ca90e5202689f947ab9" \
  --aes_signing_iv "d58ce954203b7c9a9a9d467f59839249"

Remux (combine) the audio and video streams. If you're using a video framework, you may not need to do this.
```
ffmpeg -i glocken_video.mp4 -i glocken_audio.m4a -c copy glocke.mp4
```

Media conversion sequence #

This section shows in order commands needed to get from a raw .mov file to encrypted assets packaged for DASH or HLS. For the sake of having a goal to illustrate, we're converting a source file to a bitrate of 8Mbs at a resolution of 1080p (1920 x 1080). Adjust these values as your needs dictate.

DASH/WebM #

Convert the file type and codec.

For this command you can use either liborbis or libopus for the audio codec.
```
ffmpeg -i glocken.mov -c:v libvpx-vp9 -c:a libvorbis -b:v 8M -vf setsar=1:1 -f webm tmp_glocken.webm
```
Create a Clear Key encryption key.
```
openssl rand -hex 16 > media.key
```

Demux (separate) the audio and video, encrypt the new files, and output a media presentation description (MPD) file.

packager \
  input=tmp_glocken.webm,stream=video,output=glocken_video.webm \
  input=tmp_glocken.webm,stream=audio,output=glocken_audio.webm \
  --enable_fixed_key_encryption \
  --enable_fixed_key_decryption \
  --keys label=:key=INSERT_KEY_HERE:key_id=INSERT_KEY_ID_HERE \
  --mpd_output glocken_webm_vod.mpd

Remux (combine) the audio and video streams. If you're using a video framework, you may not need to do this.
```
ffmpeg -i glocken_video.webm -i glocken_audio.webm -c copy glocken.webm
```

DASH/MP4 #

Convert the file type, video codec and bitrate.

ffmpeg -i glocken.mov -c:v libx264 -c:a aac -b:v 8M -strict -2 tmp_glocken.mp4

Create a Clear Key encryption key.
```
openssl rand -hex 16 > media.key
```

Demux (separate) the audio and video, encrypt the new files, and output a media presentation description (MPD) file.

packager \
  input=tmp_glocken.mp4,stream=video,output=glocken_video.mp4 \
  input=tmp_glocken.mp4,stream=audio,output=glocken_audio.m4a \
  --enable_fixed_key_encryption \
  --enable_fixed_key_decryption \
  --keys label=:key=INSERT_KEY_HERE:key_id=INSERT_KEY_ID_HERE \
  --mpd_output glocken_mp4_vod.mpd

Remux (combine) the audio and video streams. If you're using a video framework, you may not need to do this.
```
ffmpeg -i glocken_video.mp4 -i glocken_audio.m4a -c copy glocken.mp4
```

HLS/MP4 #

HLS only supports MP4, so first you'll need to convert to the MP4 container and supported codecs.

Convert the file type, video codec, and bitrate.

ffmpeg -i glocken.mov -c:v libx264 -c:a aac -b:v 8M -strict -2 glocken.mp4

Create a Clear Key encryption key.
```
openssl rand -hex 16 > media.key
```

Create a key information file

packager \
  'input=glocken.mp4,stream=video,segment_template=output$Number$.ts,playlist_name=video_playlist.m3u8' \
  'input=glocken.mp4,stream=audio,segment_template=output_audio$Number$.ts,playlist_name=audio_playlist.m3u8,hls_group_id=audio,hls_name=ENGLISH' \
  --hls_master_playlist_output="master_playlist.m3u8" \
  --hls_base_url="http://localhost:5000/" \
  --enable_fixed_key_encryption \
  --enable_fixed_key_decryption \
  --keys label=:key=INSERT_KEY_HERE:key_id=INSERT_KEY_ID_HERE

That was a lot to digest, but hopefully you are now able to encrypt you media with confidence. Next we'll show you how to add media to your site.

Media frameworks

2021-07-05T00:00:00Z

There are several ways to add media to a web page. Previously you learned how to use the standard <video> tag. In this article you will learn about a few of the media frameworks (or libraries) available that you can use to extend or replace the behavior of the default HTML5 video player.

Media frameworks come in both the proprietary and open-source variety, and at their core are a set of APIs that support audio and/or video playback for various container formats and transmission protocols. A good framework has a modular architecture and provides clear and detailed documentation. Ideally it should also be relatively easy to set up and use. You might be asking yourself, "If the HTML5 video player provides most features already then why would I use a framework or library?" That's a great question, let's dig in.

Benefits of using a framework #

In most situations that are beyond the needs of a basic web page, you are going to want to use a media framework to serve your content. Unless you are prepared to develop and maintain a rich feature set like offline playback, streaming, analytics, picture-in-picture, preview thumbnails, embedding, and monetization such as fill rate optimization, Ad scheduling, or header bidding to name a few then you should probably offload that complexity to an existing solution.

This is where media frameworks come in. They provide you with a set of features, and conditions in which you can use those features, and then you have to decide which framework is right for your project. In the next article we will discuss how we built a PWA with offline streaming that implements several complex features. Spoiler alert, it was a lot of work and is still far from being a production ready solution. Save yourself the headache and use a framework.

There are a lot of options out there to choose from, for now this article will focus on three, which are Shaka Player, JW Player, and Video.js.

Shaka Player #

According to the documentation, Google's Shaka Player is an open-source JavaScript library for adaptive media. It plays adaptive media formats (such as DASH and HLS) in a browser, without using plugins. Instead, Shaka Player uses the open web standards MediaSource Extensions and Encrypted Media Extensions.

Shaka Player also supports offline storage and playback of media using IndexedDB. Content can be stored on any browser. Storage of licenses depends on browser support.

There are directions for basic usage on the Shaka Player documentation site. However, in a nutshell to use Shaka Player you first need to create an HTML page with a video or audio element. Then in your application's JavaScript you install the polyfills and check for browser support. Once the browser has confirmed support for Shaka Player a script will create a Player object to wrap the media element, listen for errors, then load a manifest file. Shaka Player will take over from there.

With Shaka you will need to host and encode your media files yourself. Creating a media server is not overly complex, however encoding/transcoding media can be time-consuming and complicated. You will likely want to offload this part to a service such as Zencoder, Amazon Elastic Encoder, or Google Transcoder API to automate repetitive tasks and speed the process up.

The great thing about the Shaka Player is there is also a really fantastic tool and media packaging SDK for DASH and HLS packaging and encryption called Shaka Packager. It can prepare and package media content for online streaming, which you learned about earlier in Media conversion and Media encryption.

JW Player #

If you are looking for an option that provides hosting and encoding/transcoding services then you might look into JW Player, but keep in mind that JW Player is proprietary software. Meaning, you don't have much control over the platform or roadmap. There is a basic free version where videos are displayed with a watermark, and a commercial version.

JW Player supports streaming with MPEG-DASH (paid version only), Digital rights management (DRM) (with Vualto), interactive advertisement, customization of the interface, and embeds. There is a well documented API and SDK. However, if you are just looking for a quick and free way to host your media then embedding YouTube videos are typically going to be your best free option.

Video.js #

According to their website, Video.js is built from the ground up for an HTML5 world. It supports HTML5 video and modern streaming formats such as DASH and HLS, as well as YouTube, and Vimeo. It supports video playback on desktop and mobile devices and looks good everywhere with CSS-based skins.

There are a few ways to use Video.js, but the easiest is to use the free CDN version provided by Fastly. You can learn more on how to get the player set up on the getting started page. Video.js is a very powerful video player, and much like Shaka Player you will also need to host and encode your video. However, one difference is in the plugin system where you can do things like play YouTube videos in the Video.js player, which can also be customized.

Other frameworks #

There are lots of different frameworks and libraries available, this article only scratches the surface on what is out there. When choosing a framework you should consider what features you need for the project and how you plan to host and encode or transcode your media. Do you need pre-roll Ads or other monetization strategies? Will your media be available offline? How big or small is your budget? Or any other number of considerations. Do your research and ask people in your network for suggestions. There are dozens of other great options and before you make any choices take some time to pick one that will be right for your team and avoid creating unnecessary technical debt or complexity to maintain during the project lifecycle.

Next, you will learn about the PWA with offline streaming we built to demonstrate how to approach and tackle the main challenges that come with rolling your own solution using just the HTML5 video object without a framework to handle the heavy lifting.

Media streaming basics

2021-07-05T00:00:00Z

In this article, you are going to learn about the more advanced concept of media streaming and by the end should have a good understanding of the various streaming use cases, protocols, and extensions. Let's start with an explanation of what streaming actually is.

Media streaming is a way of delivering and playing back media content piece by piece. Instead of loading a single file, which can be slow if not optimized for the network, the player reads a manifest file describing how the target media is split into individual chunks of data. Media chunks are later dynamically stitched back together at runtime—probably at different bitrates, which you'll learn about later.

Keep in mind that to provide streaming on your website the server must support the Range HTTP request header. Learn more about the Accept-Ranges header in The <video> and <source> tags article.

Streaming use cases #

Producing media chunks and the necessary manifests describing the stream is not exactly straightforward, but streaming unlocks some interesting use cases that are not possible to achieve just by pointing a <video> element to a set of static source files. You'll learn more about how to add media to a web page in a later section. First, you should know about a few use cases for streaming multimedia if you want to go further than just loading multiple files into the <video> element.

Adaptive streaming is where media chunks are encoded in several bitrates, and the highest quality media chunk that fits the client's currently available bandwidth is returned to the media player.
Live broadcast is where media chunks are encoded and made available in real time.
Injecting media is where other media like advertisements are injected into a stream without the player having to change the media source.

Streaming protocols #

The two most commonly used streaming protocols on the web are Dynamic Adaptive Streaming over HTTP (DASH) and HTTP Live Streaming (HLS). Players that support these protocols will fetch the generated manifest file, figure out which media chunks to request, and then combine them into the final media experience.

Using `<video>` to play a stream #

Many browsers are not going to play your stream natively. While there is some native support for HLS playback, browsers generally don't support native DASH stream playback. This means often it's not enough to simply point the <source> in the <video> element to a manifest file.

<video controls>
  <source src="manifest.mpd" type="application/dash+xml">
</video>

What may seem as a deficit is actually a strength in disguise. Streams are powerful and applications that consume streams have different needs.

Manifest files usually describe many variants of single media. Think different bitrates, several audio tracks, and even the same media encoded in different formats.

Some applications may want to keep a larger amount of video in the buffer, others may want to prefetch the first few seconds of video from an upcoming episode, and some want to implement their own logic for adaptive streaming. This is where you would want to have some sort of built-in browser feature to generate media streams for playback, and it just so happens there is one.

Media Source Extensions #

Thankfully, the W3C defined something called Media Source Extensions (MSE) that will let JavaScript generate our media streams. In a nutshell, MSE allows developers to attach a MediaSource object to a <video> element and have it play back whatever media data is pumped into the buffers attached to the MediaSource instance.

Basic example #

const videoEl = document.querySelector('video');
const mediaSource = new MediaSource();

video.src = URL.createObjectURL(mediaSource);
mediaSource.addEventListener(
  'sourceopen',
  () => {
    const mimeString = 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"';
    const buffer = mediaSource.addSourceBuffer(mimeString);

    buffer.appendBuffer( /* Video data as `ArrayBuffer` object. */ )
  }
);

The simplified example above illustrates a few things:

As far as <video> is concerned, it is receiving media data from a URL.
The generated URL is just a pointer to a MediaSource instance.
The MediaSource instance creates one or more SourceBuffer instances.
We then just append binary media data into the buffer, e.g. using fetch.

While these basic concepts are simple, and it is certainly possible to write a DASH and HLS compatible video player from scratch, most people usually pick one of the mature open source solutions that already exist, such as Shaka Player, JW Player, or Video.js to name a few.

However, we have created a demo Media PWA called Kino that demonstrates how you would go about developing your own basic streaming media website that provides offline media playback using just the simple <video> element. There are plans in our roadmap to support frameworks and digital rights management, among other features. So check back for updates from time to time, or request a feature. Read more about it in the PWA with offline streaming article.

Media chunks format #

For a long time, DASH and HLS required media chunks to be encoded in different formats. In 2016, however, support for standard fragmented MP4 (fMP4) files was added to HLS, a format that DASH also supports.

Video chunks using the fMP4 container and the H.264 codec are supported by both protocols and playable by a vast majority of players. This allows content producers to encode their videos just once, which in turn saves time and disk space.

To achieve better quality and lower files sizes, you may want to choose to encode several sets of media chunks using more efficient formats like VP9, though before we get to far ahead you will first need to learn how to Prepare media files for the web, and that's up next.

PWA with offline streaming

2021-07-05T00:00:00Z

Progressive Web Apps bring a lot of features previously reserved for native applications to the web. One of the most prominent features associated with PWAs is an offline experience.

Even better would be an offline streaming media experience, which is an enhancement you could offer to your users in a few different ways. However, this creates a truly unique problem—media files can be very large. So you might be asking:

How do I download and store a large video file?
And how do I serve it to the user?

In this article we will discuss answers to these questions, while referencing the Kino demo PWA we built that provides you with practical examples of how you can implement an offline streaming media experience without using any functional or presentational frameworks. The following examples are mainly for educational purposes, because in most cases you should probably use one of the existing Media Frameworks to provide these features.

Unless you have a good business case for developing your own, building a PWA with offline streaming has its challenges. In this article you will learn about the APIs and techniques used to provide users with a high-quality offline media experience.

Downloading and storing a large media file #

Progressive Web Apps usually use the convenient Cache API to both download and store the assets required to provide the offline experience: documents, stylesheets, images, and others.

Here is a basic example of using the Cache API within a Service Worker:

const cacheStorageName = 'v1';

this.addEventListener('install', function(event) {
  event.waitUntil(
    caches.open(cacheStorageName).then(function(cache) {
      return cache.addAll([
        'index.html',
        'style.css',
        'scripts.js',

        // Don't do this.
        'very-large-video.mp4',
      ]);
    })
  );
});

While the example above does technically work, using the Cache API has several limitations that makes its use with large files impractical.

For example, the Cache API doesn't:

Allow you to easily pause and resume downloads
Let you track the progress of downloads
Offer a way to properly respond to HTTP range requests

All of these issues are pretty serious limitations for any video application. Let's review some other options that might be more appropriate.

Nowadays, the Fetch API is a cross-browser way to asynchronously access remote files. In our use case it allows you to access large video files as a stream and store them incrementally as chunks using an HTTP range request.

Now that you can read the chunks of data with the Fetch API you also need to store them. Chances are there is a bunch of metadata associated with your media file such as: name, description, runtime length, category, etc.

You're not storing just the one media file, you are storing a structured object, and the media file is just one of its properties.

In this case the IndexedDB API provides an excellent solution to store both the media data and metadata. It can hold huge amounts of binary data easily, and it also offers indexes that allow you to perform very fast data lookups.

Downloading media files using the Fetch API #

We built a couple of interesting features around the Fetch API in our demo PWA, which we named Kino—the source code is public so feel free to review it.

The ability to pause and resume incomplete downloads.
A custom buffer for storing chunks of data in the database.

Before showing how those features are implemented, we'll first do a quick recap of how you can use the Fetch API to download files.

/**
 * Downloads a single file.
 *
 * @param {string} url URL of the file to be downloaded.
 */
async function downloadFile(url) {
  const response = await fetch(url);
  const reader = response.body.getReader();
  do {
    const { done, dataChunk } = await reader.read();
    // Store the `dataChunk` to IndexedDB.
  } while (!done);
}

Notice that await reader.read() is in a loop? That's how you'll receive chunks of data from a readable stream as they arrive from the network. Consider how useful this is: you can start processing your data even before it all arrives from the network.

Resuming downloads #

When a download is paused or interrupted, the data chunks that have arrived will be safely stored in an IndexedDB database. You can then display a button to resume a download in your application. Because the Kino demo PWA server supports HTTP range requests resuming a download is somewhat straightforward:

async downloadFile() {
  // this.currentFileMeta contains data from IndexedDB.
  const { bytesDownloaded, url, downloadUrl } = this.currentFileMeta;
  const fetchOpts = {};

  // If we already have some data downloaded,
  // request everything from that position on.
  if (bytesDownloaded) {
    fetchOpts.headers = {
      Range: `bytes=${bytesDownloaded}-`,
    };
  }

  const response = await fetch(downloadUrl, fetchOpts);
  const reader = response.body.getReader();

  let dataChunk;
  do {
    dataChunk = await reader.read();
    if (!dataChunk.done) this.buffer.add(dataChunk.value);
  } while (!dataChunk.done && !this.paused);
}

Custom write buffer for IndexedDB #

On paper, the process of writing dataChunk values into an IndexedDB database is simple. Those values already are ArrayBuffer instances, which are storable in IndexedDB directly, so we can just create an object of an appropriate shape and store it.

const dataItem = {
  url: fileUrl,
  rangeStart: dataStartByte,
  rangeEnd: dataEndByte,
  data: dataChunk,
}

// Name of the store that will hold your data.
const storeName = 'fileChunksStorage'

// `db` is an instance of `IDBDatabase`.
const transaction = db.transaction([storeName], 'readwrite');
const store = transaction.objectStore(storeName);
const putRequest = store.put(data);

putRequest.onsuccess = () => { ... }

While this approach works, you will likely discover that your IndexedDB writes are significantly slower than your download. This isn't because IndexedDB writes are slow, it's because we are adding a lot of transactional overhead by creating a new transaction for every data chunk that we receive from a network.

The downloaded chunks can be rather small and can be emitted by the stream in rapid succession. You need to limit the rate of IndexedDB writes. In the Kino demo PWA we do this by implementing an intermediary write buffer.

As data chunks arrive from the network, we append them to our buffer first. If the incoming data doesn't fit, we flush the full buffer into the database and clear it before appending the rest of the data. As a result our IndexedDB writes are less frequent, which leads to significantly improved write performance.

Serving a media file from offline storage #

Once you have a media file downloaded, you probably want your service worker to serve it from IndexedDB instead of fetching the file from the network.

/**
 * The main service worker fetch handler.
 *
 * @param {FetchEvent} event Fetch event.
 */
const fetchHandler = async (event) => {
  const getResponse = async () => {
    // Omitted Cache API code used to serve static assets.

    const videoResponse = await getVideoResponse(event);
    if (videoResponse) return videoResponse;

    // Fallback to network.
    return fetch(event.request);
  };
  event.respondWith(getResponse());
};
self.addEventListener('fetch', fetchHandler);

So what do you need to do in getVideoResponse()?

The event.respondWith() method expects a Response object as a parameter.
The Response() constructor tells us that there are several types of objects we could use to instantiate a Response object: a Blob, BufferSource, ReadableStream, and more.
We need an object that doesn't hold all of its data in memory, so we'll probably want to choose the ReadableStream.

Also, because we're dealing with large files, and we wanted to allow browsers to only request the part of the file they currently need, we needed to implement some basic support for HTTP range requests.

/**
 * Respond to a request to fetch offline video file and construct a response
 * stream.
 *
 * Includes support for `Range` requests.
 *
 * @param {Request} request  Request object.
 * @param {Object}  fileMeta File meta object.
 *
 * @returns {Response} Response object.
 */
const getVideoResponse = (request, fileMeta) => {
  const rangeRequest = request.headers.get('range') || '';
  const byteRanges = rangeRequest.match(/bytes=(?<from>[0-9]+)?-(?<to>[0-9]+)?/);

  // Using the optional chaining here to access properties of
  // possibly nullish objects.
  const rangeFrom = Number(byteRanges?.groups?.from || 0);
  const rangeTo = Number(byteRanges?.groups?.to || fileMeta.bytesTotal - 1);

  // Omitting implementation for brevity.
  const streamSource = {
     pull(controller) {
       // Read file data here and call `controller.enqueue`
       // with every retrieved chunk, then `controller.close`
       // once all data is read.
     }
  }
  const stream = new ReadableStream(streamSource);

  // Make sure to set proper headers when supporting range requests.
  const responseOpts = {
    status: rangeRequest ? 206 : 200,
    statusText: rangeRequest ? 'Partial Content' : 'OK',
    headers: {
      'Accept-Ranges': 'bytes',
      'Content-Length': rangeTo - rangeFrom + 1,
    },
  };
  if (rangeRequest) {
    responseOpts.headers['Content-Range'] = `bytes ${rangeFrom}-${rangeTo}/${fileMeta.bytesTotal}`;
  }
  const response = new Response(stream, responseOpts);
  return response;

Feel free to check out the Kino demo PWA service worker source code to find out how we are reading file data from IndexedDB and constructing a stream in a real application.

Other considerations #

With the main obstacles out of your way, you can now start adding some nice-to-have features to your video application. Here are a few examples of features you would find in the Kino demo PWA:

Media Session API integration that allows your users to control media playback using dedicated hardware media keys or from media notification popups.
Caching of other assets associated with the media files like subtitles, and poster images using the good old Cache API.
Support for video streams (DASH, HLS) download within the app. Because stream manifests generally declare multiple sources of different bitrates, you need to transform the manifest file and only download one media version before storing it for offline viewing.

Up next, you will learn about Fast playback with audio and video preload.

Media accessibility

2020-08-20T00:00:00Z

In this article you will learn about the WebVTT (Web Video Text Tracks) format, which is used to describe timed text data such as closed captions or subtitles in order to make videos more accessible to your audience.

Accessibility isn't like icing on a cake. It's never anything you put on a backlog with the hope of introducing it later. Captions and screen reader descriptions are the only way many users can experience your videos, and in some jurisdictions, they're even required by law or regulation.

Add `<track>` tags #

To add captions or screen reader descriptions to a web video, add a <track> tag within a <video> tag. In addition to captions and screen reader descriptions, <track> tags may also be used for subtitles and chapter titles. The <track> tag can also help search engines understand what's in a video. However, those capabilities are outside the scope of this article.

Screenshot showing captions displayed using the track element in Chrome on Android

The <track> tag is similar to the <source> element in that both have a src attribute that points to referenced content. For a <track> tag, it points to a WebVTT file. The label attribute specifies how a particular track will be identified in the interface.

To provide tracks for multiple languages add a separate <track> tag for each WebVTT file you're providing and indicate the language using the srclang attribute.

An example <video> tag with two <track> tags is shown below. There's also a sample you can view on Glitch (source).

Add a <track> element as a child of the <video> element:

<video controls>
  <source src="https://storage.googleapis.com/webfundamentals-assets/videos/chrome.webm" type="video/webm" />
  <source src="https://storage.googleapis.com/webfundamentals-assets/videos/chrome.mp4" type="video/mp4" />
  <track src="chrome-subtitles-en.vtt" label="English captions" kind="captions" srclang="en" default>
  <track src="chrome-subtitles-zh.vtt" label="中文字幕" kind="captions" srclang="zh">
  <p>This browser does not support the video element.</p>
</video>

WebVTT file structure #

Below is a hypothetical WebVTT file for the demo linked to above. The file is a text file containing a series of cues. Each cue is a block of text to display on screen, and the time range during which it will be displayed.

WEBVTT

00:00.000 --> 00:04.999
Man sitting on a tree branch, using a laptop.

00:05.000 --> 00:08.000
The branch breaks, and he starts to fall.

...

Each item within the track file is called a cue. Each cue has a start time and end time separated by an arrow, with cue text in the line below. Cues can optionally also have IDs like railroad and manuscript in the example below. Cues are separated by an empty line.

WEBVTT

railroad
00:00:10.000 --> 00:00:12.500
Left uninspired by the crust of railroad earth

manuscript
00:00:13.200 --> 00:00:16.900
that touched the lead to the pages of your manuscript.

Cue times are in hours:minutes:seconds.milliseconds format. Parsing is strict. Meaning, numbers must be zero padded if necessary: hours, minutes, and seconds must have two digits (00 for a zero value) and milliseconds must have three digits (000 for a zero value). There is an excellent WebVTT validator at Live WebVTT Validator, which checks for errors in time formatting, and problems such as non-sequential times.

You can create a VTT file by hand, thought there are many services that will create them for you.

As you can see in our previous examples, the WebVTT format is pretty simple. Just add your text data along with timings.

However, what if you want your captions to render in a different position with left or right alignment? Perhaps to align the captions with the current speaker position, or to stay out of the way of in-camera text. WebVTT defines settings to do that, and more, directly inside the .vtt file. Take note of how the caption placement is defined by adding settings after the time interval definitions.

WEBVTT

00:00:05.000 --> 00:00:10.000 line:0 position:20% size:60% align:start
The first line of the subtitles.

Another handy feature is the ability to style cues using CSS. Perhaps you want to use a gray linear gradient as the background, with a foreground color of papayawhip for all captions and all bold text colored peachpuff.

video::cue {
  background-image: linear-gradient(to bottom, dimgray, lightgray);
  color: papayawhip;
}

video::cue(b) {
  color: peachpuff;
}

If you're interested in learning more about styling and tagging of individual cues, the WebVTT specification is a good source for advanced examples.

Kinds of text tracks #

Did you notice the kind attribute of the <track> element? It's used to indicate what relation the particular text track has to the video. The possible values of the kind attribute are:

captions: For closed captions from transcripts and possibly translations of any audio. Suitable for hearing impaired and in cases when the video is playing muted.
subtitles: For subtitles, that is, translations of speech and text in a language different from the main language of the video.
descriptions: For descriptions of visual parts of the video content. Suitable for visually impaired people.
chapters: Intended to be displayed when the user is navigating within the video.
metadata: Not visible, and may be used by scripts.

Now that you understand the basics of making a video available and accessible on your web page, you might wonder about more complex use cases. Next up, you'll lean about Media frameworks and how they can help you add videos to your web page while providing advanced features.

Add media to a web page

2020-06-19T00:00:00Z

In this section you'll learn how to embed a media file in a web page using the <video> and <source> tags, how to add captions to a video for the hearing-impaired, and some general concepts for using media frameworks. Additionally, you'll learn about using preload to speed up playback start, and we will discuss how we built an updated demo PWA with offline adaptive streaming capabilities called Kino.

This section assumes you have a video file that is ready for embedding in a web page. A .mov file downloaded from a camera will not work. If that's all you have, see Prepare media files for the web then come back.

This section covers these topics.

In The <video> and <source> tags you'll learn specifically how to embed a media file in a web page.
In Media accessibility you'll learn to add captions to a media file for hearing impaired.
In Media frameworks you'll learn basics about using media frameworks like Shaka Player, JW Player, and Video.js.
In Fast playback with audio and video preload you'll learn to accelerate your media playback by actively preloading resources.
In PWA with offline streaming you'll learn how we built an updated demo PWA that is capable of adaptive streaming and offline playback without using frameworks. Plus you can play with source code.

There is a lot of ground to cover in this section, get started by learning how to use the <video> and <source> tags.

Media file basics

2020-06-19T00:00:00Z

In this article you will learn about media file basics such as the concepts of a container, and a few of the many available codec formats that you can use in a stream. Plus lightly touch on topics such as adaptive streaming, bitrate, and resolution—but we'll dive deeper into all these in later sections.

Serving video files #

You might think that you can take a raw file from a video camera and just upload it to the web as-is. Indeed, video streaming sites such as YouTube or Vimeo let you do just that, and even provide live-streaming capabilities—typically by connecting to your camera's HDMI port and then processing it through a capture card. These services greatly simplify video processing and uploading, which includes generating the many files and manifests needed for adaptive streaming and various resolutions. Plus the many other complicated and nuanced requirements that make self-hosting a bit of a chore. Preparing and serving a video from your own site, and likely a separate media server, is a bit more complicated than just uploading a raw camera file if you care how your users experience your site.

Video files come in a variety of formats. The format that comes off your camera is typically a .mov file, or an .mp4 if you have a good modern mirrorless camera. However, while a .mov is good for recording and for editing and other early post-production processes, the file size means it's not good for streaming over the web. As well, the file size of a raw .mp4 in 4K is going to make playing that file on mobile very prohibitive. Because browsers support different file formats, you'll need to create multiple optimized files and potentially a manifest if you plan to support adaptive streaming. Before converting files you need to understand a few basics about them and about their characteristics.

Containers and codecs, and streams? #

The file that you see in your operating system shell is a container, identified by a file extension (.mp4, .webm, .ogg etc.). The container houses one or more streams. A media file can have any number of streams, of many more formats than we'll go into here.

The sample files used later in this section contain at most two streams: an audio stream and a video stream. Among the other types you might encounter are captions and data, both of which are beyond the scope of this article. There are instances where audio and video streams are dealt with separately. Most files you'll encounter will only contain a single audio stream and a single video stream.

Within the audio and video streams, the actual data is compressed using a codec. A codec, or coder/decoder, is a compression format for video or audio data. The distinction between a container and a codec is important because files with the same container can have their contents encoded with different codecs.

The image below illustrates this structure. On the left is the basic container structure with two streams. On the right are the specifics of that structure for a single WebM file.

Parts of a media file.

Files in WebM containers can be orders of magnitude smaller than other formats, making them a good choice for mobile sites to stream. Unfortunately, not all browsers support up-to-date containers and codecs. For example, WebM was created specifically for the web as a high-quality and open source option, but its support is not yet universal. Safari in particular does not, according to Can I use at the time of this writing, support WebM for embedded video. However, WebM does have partial support with the VP8 and VP9 codec used in WebRTC. So your best option is to provide a fallback video.

Codec formats #

Many file types support multiple codecs within the same container. A complete list of available video codecs and audio codecs would be a whole website to itself. The links just provided are for MDN's practical lists of what's usable on the web. Listed below are the currently preferred file types, and the codecs they might use. Follow the file type links to view the browsers that support them.

File type	Video Codec	Audio Codec
MP4	AV1, AVC (H.264)*, VP9	AAC
WebM	AV1, VP9*	Vorbis, Opus

* Indicates the preferred video codec.

Bitrate and resolution #

Bitrate is the maximum number of bits used to encode one second of a stream. The more bits used to encode a second of stream, the higher the potential detail and fidelity. We provide more information about this concept in Bitrate.

Resolution is the amount of information in a single frame of video, given as the number of logical pixels in each dimension. We provide more information about this concept in Resolution.

Up next, in Media application basics, we'll show you how to examine these characteristics using two command line tools: Shaka Packager and FFmpeg.

Media conversion

2018-09-20T00:00:00Z

In this article we are going to learn some common commands for converting and manipulating specific characteristics of media files. Although we've tried to show equivalent operations for all procedures, not all operations are possible in both applications.

In many cases, the commands we're showing may be combined in a single command line operation, and would be when actually used. For example, there's nothing preventing you from setting an output file's bitrate in the same operation as a file conversion. For this article, we often show these operations as separate commands for the sake of clarity.

Conversion is done with these applications:

Display characteristics #

Both Shaka Packager and FFmpeg can be used to inspect the content of a media file and then display the characteristics of a stream. However, both provide different output for the same media.

Characteristics using Shaka Packager #

packager input=glocken.mp4 --dump_stream_info

The output looks like:

File "glocken.mp4":
Found 2 stream(s).
Stream [0] type: Video
 codec_string: avc1.640028
 time_scale: 30000
 duration: 300300 (10.0 seconds)
 is_encrypted: false
 codec: H264
 width: 1920
 height: 1080
 pixel_aspect_ratio: 1:1
 trick_play_factor: 0
 nalu_length_size: 4

Stream [1] type: Audio
 codec_string: mp4a.40.2
 time_scale: 48000
 duration: 481280 (10.0 seconds)
 is_encrypted: false
 codec: AAC
 sample_bits: 16
 num_channels: 2
 sampling_frequency: 48000
 language: eng
 seek_preroll_ns: 20833

Characteristics using FFmpeg #

ffmpeg -i glocken.mp4

The output looks like:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'glocken.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.83.100
  Duration: 00:00:10.03, start: 0.000000, bitrate: 8063 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc), 1920x1080, 7939 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
At least one output file must be specified

Demux (separate) the audio and video streams #

Shaka Packager requires demuxing when converting files. This is also required for using media frameworks.

Shaka Packager demuxing #

MP4

packager input=myvideo.mp4,stream=video,output=myvideo_video.mp4
packager input=myvideo.mp4,stream=audio,output=myvideo_audio.m4a

Or:

packager \
  input=myvideo.mp4,stream=video,output=myvideo_video.mp4 \
  input=myvideo.mp4,stream=audio,output=myvideo_audio.m4a

WebM

packager \
  input=myvideo.webm,stream=video,output=myvideo_video.webm \
  input=myvideo.webm,stream=audio,output=myvideo_audio.webm

FFmpeg demuxing #

MP4

ffmpeg -i myvideo.mp4 -vcodec copy -an myvideo_video.mp4
ffmpeg -i myvideo.mp4 -acodec copy -vn myvideo_audio.m4a

WebM

ffmpeg -i myvideo.webm -vcodec copy -an myvideo_video.webm
ffmpeg -i myvideo.webm -acodec copy -vn myvideo_audio.webm

Remux (combine) the audio and video streams #

In some situation you will need to combine the audio and video back into a single container. Especially when not using a media framework. This is something FFmpeg can handle quite well and is something Shaka Packager does not currently support.

ffmpeg -i myvideo_video.webm -i myvideo_audio.webm -c copy myvideo.webm

Change characteristics #

Bitrate #

For FFmpeg, we can do this while converting to .mp4 or .webm.

ffmpeg -i myvideo.mov -b:v 350K myvideo.mp4
ffmpeg -i myvideo.mov -vf setsar=1:1 -b:v 350K myvideo.webm

Dimensions (resolution) #

ffmpeg -i myvideo.webm -s 1920x1080 myvideo_1920x1080.webm

File type #

Shaka Packager cannot process .mov files and hence cannot be used to convert files from that format.

.mov to .mp4

ffmpeg -i myvideo.mov myvideo.mp4

.mov to .webm

ffmpeg -i myvideo.mov myvideo.webm

Synchronize audio and video #

To ensure that audio and video synchronize during playback, insert keyframes.

ffmpeg -i myvideo.mp4 -keyint_min 150 -g 150 -f webm -vf setsar=1:1 out.webm

MP4/H.264

ffmpeg -i myvideo.mp4 -c:v libx264 -c:a copy myvideo.mp4

Audio for an MP4

ffmpeg -i myvideo.mp4 -c:v copy -c:a aac myvideo.mp4

WebM/VP9

ffmpeg -i myvideo.webm -v:c libvpx-vp9 -v:a copy myvideo.webm

Audio for a WebM

ffmpeg -i myvideo.webm -v:c copy -v:a libvorbis myvideo.webm
ffmpeg -i myvideo.webm -v:c copy -v:a libopus myvideo.webm

Video-on-demand and live-streaming #

There are two types of streaming protocols we are going to demonstrate in this article. The first is Dynamic Adaptive Streaming over HTTP (DASH), which is an adaptive bitrate streaming technique and web-standards-based method of presenting video-on-demand. The second is HTTP Live Streaming (HLS), which is Apple's standard for live-streaming and video-on-demand for the web.

DASH/MPD #

This example generates the Media Presentation Description (MPD) output file from the audio and video streams.

packager \
  input=myvideo.mp4,stream=audio,output=myvideo_audio.mp4 \
  input=myvideo.mp4,stream=video,output=myvideo_video.mp4 \
  --mpd_output myvideo_vod.mpd

HLS #

These examples generate an M3U8 output file from the audio and video streams, which is a UTF-8 encoded multimedia playlist.

ffmpeg -i myvideo.mp4 -c:a copy -b:v 8M -c:v copy -f hls \
  -hls_time 10 -hls_list_size 0 myvideo.m3u8

OR:

packager \
  'input=myvideo.mp4,stream=video,segment_template=output$Number$.ts,playlist_name=video_playlist.m3u8' \
  'input=myvideo.mp4,stream=audio,segment_template=output_audio$Number$.ts,playlist_name=audio_playlist.m3u8,hls_group_id=audio,hls_name=ENGLISH' \
  --hls_master_playlist_output="master_playlist.m3u8"

Now that we hopefully have a good grasp on how to convert files, we can build on what we've learned in this article and go learn about Media encryption next.

Bitrate

2017-06-30T00:00:00Z

In the previous Containers and codecs article, you learned how to change the container (extension) and codec of a media file. In this article, we'll show you how to change bitrate before explaining resolution.

Bitrate and resolution correlate to the amount of data in a media file. It probably goes without saying, but we're going to say it anyway. You can always lower bitrate and resolution, but increasing them is a problem. Without special software and algorithms, quality is going to take a hit.

So always start your conversion process with the highest quality source file you can get your hands on. Before doing anything, even before changing the codec or container, check the file's display characteristics and verify that your source file has a higher bitrate or resolution than your desired result.

Bitrate is the maximum number of bits used to encode one second of a media stream. The more bits used to encode a second of stream, the higher the fidelity.

Unsurprisingly, the different bitrates the web can handle are low. The table below shows you what bitrate you should target for common network conditions. For the sake of comparison, we've thrown in values for Blu-rays and DVDs.

Delivery method	Bitrate
Blu-ray	20Mbs
DVD	6 Mbs
Desktop web	2 Mbs
4G mobile	0.7 Mbs
3G mobile	0.35 Mbs
2G mobile	Depends on network type. EDGE: 0.4 Mbs GPRS: 0.04Mbs

Which value should I use for video on my web pages? The short answer is at least: desktop, 4G, and 3G. If you're serving video in one of the markets referred to as "the next billion users", say India, for example, you'll want to include 2G as well. For demonstration purposes, we're going to target 3G.

Using FFmpeg you set the bitrate with the (surprise!) bitrate (-b) flag.

If you don't have FFmpeg installed read Media application basics to get it set up with Docker.

MP4

/media # ffmpeg -i glocken.mov -b:v 350k -b:a 64k glocken_3g.mp4

WebM

/media # ffmpeg -i glocken.mov -b:v 350k -b:a 64k glocken_3g.webm

Notice that there are two bitrate flags, -b:a and -b:v. One is for the audio stream, and the other is for the video stream.

/media # ls -l
-rw-r--r-- 1 root root  12080306 Mar  7 12:16 glocken.mov
-rwx------ 1 root root    531117 Mar  7 13:42 glocken_3g.mp4
-rwx------ 1 root root    706119 Mar  7 13:46 glocken_3g.webm

Now that your files are prepared, it's time to adjust their resolutions.

Containers and codecs

2017-06-30T00:00:00Z

To support multiple browsers, you'll need to use FFmpeg to convert your .mov file to two different containers: an MP4 container and a WebM container. In actual practice, you would likely specify a codec at the same time. For now, we're letting FFmpeg use its defaults.

If these concepts are new to you, you should read Media file basics before going further. Additionally, if you don't have FFmpeg installed read Media application basics to get it set up with Docker.

We are using the suggested Docker install and the glocken.mov file from Prepare media files for the web added inside the media directory. We used FFmpeg version 4.3.2 for all the commands in this section.

Containers #

First, we need to create our two containers from the .mov file with the .mp4 and .webm file extensions with both an audio and video stream inside the file. Review Media file basics for more about containers and streams if you don't know the different between them.

MP4

/media # ffmpeg -i glocken.mov glocken.mp4

WebM

/media # ffmpeg -i glocken.mov glocken.webm

WebM takes longer to create than MP4. This isn't surprising when you look at the results. While MP4 compresses to about 83% of the original file's size, WebM is down to 78% of the original's size, but can be much smaller. Your results will vary. It's important to call out that FFmpeg 4.2.2 set the default video bitrate to 200k and in 4.3.2 it does not set a default bitrate. So the video is no longer a mere4% of the original. You can see this for yourself using the ls -a bash command in the folder where your media files are located.

For example:

/media # ls -l
-rw-r--r-- 1 root  root  12080306 Mar 7 12:16 glocken.mov
-rwx------ 1 root  root  10106446 Mar 7 12:33 glocken.mp4
-rwx------ 1 root  root   9503301 Mar 7 18:30 glocken.webm

To get a tiny file, you would do this instead:

/media # ffmpeg -i glocken.mov -b:v 200k glocken.webm
...
frame=  300 fps=3.6 q=0.0 Lsize=     483kB time=00:00:10.01 bitrate= 395.0kbits/s speed=0.121x
video:359kB audio:117kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.356068%
/media # ls -l
-rw-r--r-- 1 root  root  12080306 Mar 7 12:16 glocken.mov
-rwx------ 1 root  root  10106446 Mar 7 12:33 glocken.mp4
-rwx------ 1 root  root    494497 Mar 7 18:45 glocken.webm

Check your work #

To verify your results, use FFmpeg and Shaka Packager as already shown in Media Application basics:

/media # packager input=glocken.mp4 --dump_stream_info

/media # ffmpeg -i glocken.mp4

Codecs #

Next, the codec. As stated in Media file basics, a codec is not the same thing as a container (file type). Two files of the same container type could hold data compressed using different codecs. The WebM format for example allows audio to be encoded using either Vorbis or Opus. To change the codec we use FFmpeg. For example, this command outputs an .mkv file with a vorbis audio codec and an av1 video codec.

/media # ffmpeg -i glocken.mov -c:a vorbis -c:v av1 glocken.mkv

In this example, the -c:a flag and the -c:v are for specifying the audio and video codecs respectively.

The Media conversion page lists commands needed to convert codecs. The tables below summarize the libraries used in FFmpeg to perform the codec conversions for WebM and MP4 files. These are the formats recommended for DASH and HLS respectively.

Video #

Codec	Extension	Library
av1	WebM, mkv	libaom-av1
h264	MP4	libx264
vp9	WebM	libvpx-vp9

Audio #

Codec	Extension	Library
aac	MP4	aac
opus	WebM	libopus
vorbis	WebM	libvorbis

Next, we'll show you how to change the bitrate of your newly created files.

What is a media experience?

2017-06-30T00:00:00Z

Users like media, especially videos; they can be fun and informative. On mobile devices, videos can be an easier way to consume information than text. For a good user experience, videos should not need more than the available bandwidth. Users should be able to use them no matter what device they're viewing them with. Users should never need to wait for media to download. Nobody likes it when they press play and nothing happens.

You've no doubt consumed video on your own device, which means that nothing in that last paragraph surprises you. Now you need to learn how to put a video or other media file on your own website. The technical requirements of adding media should be in service to the user experience.

The technical requirements #

Versions of a media file are in common web-friendly formats containing both audio and video streams.
The resolution is appropriate for your users' devices.
The bitrate doesn't overload your users' network bandwidth.
The result is viewable on all major browsers using appropriate technologies.
The file is encrypted (Optional).

The media section of this website will help you achieve these technical requirements. Don't worry if these concepts are still a bit abstract. We'll explain them progressively throughout all the articles. In the first section you will learn about the basic concepts of media, then how to add media to the web in the second section, and in the final section the practical applications, with some advanced techniques, of using media on the web.

Displaying video on the web #

There are four approaches you can take when displaying video on a site.

Upload your videos to a media hosting provider such as YouTube or Vimeo. These options require you to embed their players within your site.
Basic self-hosted embedding using the HTML <video> and <audio> elements.
More full-featured embedding using a video library such as Shaka Player, JW Player, or Video.js.
Building your own media server and streaming application.

This site mainly covers the basics of embedding media. However, we do cover some more advanced topics to get you started on the path towards building your own media streaming application. The effort to do this is not trivial, so we've built a Media PWA with offline support to use as a reference, which should both show you ways this can be accomplished and just how much effort it requires. The application is by no means a production ready offering or competitor to services like YouTube or Vimeo, but it will provide you with a starting point to learn something challenging and new.

Frankly, building a competitor to hosted media services would require a team of expert engineers and thousands of human-hours of work. Unless your goal is to enter that market as a competitor, you're better off using one of the other methods. It's good to understand the technology and while you may not roll out your own application or video player, there is utility in exploring and experimenting on the cutting edge of browser support, or at the very least using one of the existing video libraries.

What we're going to learn #

The media collection has three parts. In this first section, we'll provide concepts and prerequisite information to adding media to your site. This includes explaining how media files are put together, basics about the applications you'll need to prepare your files for the web, and streaming concepts. The second section explains how to prepare your files and convert them to various formats and optionally add encryption. In the last section, we'll show you how to embed a media file in a web page, discuss autoplay best practices, using media frameworks, taking videos offline, and making your videos accessible.

There's a lot of ground to cover, so let's get started with Media file basics.

Prepare media files for the web

2017-06-30T00:00:00Z

Now that we've introduced you to the applications we use when manipulating media files, over the next few pages, we're going to take a raw video file from a camera and transform it into a resource that you can embed in a web page. We're going to show you how to format your video for mobile web playback, and how to create multiple files to cover a range of browsers. Specifically, we'll create a WebM file for use on Chrome and an MP4 file for use on other browsers.

Plus we will dive deeper into common commands used for Media conversion and Media encryption that will serve as a good reference for later.

By "appropriate technologies" we mean Dynamic Adaptive Streaming over HTTP (DASH) or HTTP Live Streaming (HLS), which are the two primary means of providing video in HTML on the major browsers. By the end of this section, you'll be able to create media files that are ready for use in DASH and HLS.

If you want to play along at home, you'll need a raw video file off a camera, preferably one that contains both audio and video. If you don't have one handy, then here's ten seconds of an .mov file named glocken.mov that was taken of the Rathaus-Glockenspiel in Munich's MarienPlatz.

Next, let's get started with Containers and codecs.

Resolution

2017-06-30T00:00:00Z

In the previous articles you learned how to change the containers, codecs, and bitrate of the glocken.mov media file. This article focuses on changing the resolution.

Resolution is the amount of information in a single frame of video, given as the number of logical pixels in each dimension. For example, a resolution of 1920 by 1080 works out to 1080 stacked horizontal lines, each of which is one logical pixel high and 1920 logical pixels wide. This resolution is frequently abbreviated 1080p because technically the width can vary. The dimensions 1080 by 1920 produce an aspect ratio of 16:9, which is the ratio of movie screens and modern television sets. By the way this is the resolution defined as full HD.

YouTube recommends the following resolutions for video uploads, all in the 16:9 aspect ratio. There's nothing specific to YouTube about this list. It's just a list of common 16:9 video resolutions.

Abbreviation	Dimensions
2160p	3840 x 2160
1440p	2560 x 1440
1080p	1920 x 1080
720p	1280 x 720
480p	854 x 480
360p	640 x 360
240p	426 x 240

Which one should you use? That depends on your application. For simple embedding you may decide to chose only a single resolution. If you're preparing files for DASH or HLS, you may chose one, several, or all. Fortunately, this is one of the simplest transformations you'll make with FFmpeg.

If you don't have FFmpeg installed read Media application basics to get it set up with Docker.

MP4

/media # ffmpeg -i glocken.mov -b:v 350k -b:a 64k -s 1280x720 glocken_3g_720p.mp4

WebM

/media # ffmpeg -i glocken.mov -b:v 350k -b:a 64k -s 1280x720 glocken_3g_720p.webm

The following files should now exist:

/media # ls -l
-rw-r--r-- 1 root root  12080306 Mar  7 12:16 glocken.mov
-rwx------ 1 root root    531117 Mar  7 13:42 glocken_3g.mp4
-rwx------ 1 root root    706119 Mar  7 13:46 glocken_3g.webm
-rwx------ 1 root root    539414 Mar  7 14:15 glocken_3g_720p.mp4
-rwx------ 1 root root    735930 Mar  7 14:19 glocken_3g_720p.webm

It's worth reiterating that you should start from the highest resolution and bitrate file you have available. If you're upgrading an older site, you'll want to find your original camera or other high resolution sources and convert from that rather than from older web files.

Now that your files are prepared, you can either add them to a web page as they are now or dive a bit deeper and continue to learn more command line options by reading the Media conversion page, and then close out the section with Media encryption.

Media application basics

2017-06-09T00:00:00Z

Working with media often requires changing the characteristics of media files, such as bitrate or resolution. Finding a straightforward way to get started can be pretty intimidating. On this page, you will learn about the tools used and how to install them quickly.

First, we describe the basic usage for two common command-line media utilities: Shaka Packager and FFmpeg and then we help you quickly get the tools installed. Why cover two applications? While both are powerful and useful by themselves, neither does everything needed to prepare media for the web. We also created the Media conversion and Media encryption pages which show many more common operations with these two applications.

These applications aren't the only options available for file manipulation tasks, but they are two of the most common and powerful. Other options include the GUI applications Miro, HandBrake, and VLC. There are also encoding/transcoding services such as Zencoder, Amazon Elastic Encoder, and Google Transcoder API (currently in beta).

Shaka Packager #

Shaka Packager is a free media packaging SDK. If you were using a media player on your site, Shaka Packager is what you would use to prepare the files. It supports conversion for the two most common video streaming protocols: Dynamic Adaptive Streaming over HTTP (DASH) or HTTP Live Streaming (HLS). Shaka Packager supports key security features: common encryption and Widevine digital rights management (DRM). It can also handle live-streaming, and video-on-demand.

Despite what it says on the package, this utility is for more than C++ developers. You can use it as both a library for building media software and as a command-line utility for preparing media files for web playback. It's the latter capacity that's useful for us here. In fact, for web media creators, Shaka Packager is the only way to do some tasks without spending money on expensive commercial applications.

Here's the basic pattern for a Shaka Packager command:

packager stream_descriptor [stream_descriptor-2 [stream_descriptor-n]] [flags]

This isn't quite what you get if you type packager -help. This example is easier to reason about, and this reflects the examples in the Shaka Packager documentation. Note that there are multiple stream_descriptor items in the pattern. Though we don't show it, you could manipulate the video and audio streams of a file separately in a single command.

Compare this basic pattern with a simple use that displays file characteristics. In the example, We've lined up equivalent parts.

packager stream_descriptor [stream_descriptor-n] [flags]

packager input=glocken.mp4                       --dump_stream_info

The command outputs this:

File "glocken.mp4":
Found 2 stream(s).
Stream [0] type: Video
 codec_string: avc1.640028
 time_scale: 30000
 duration: 300300 (10.0 seconds)
 is_encrypted: false
 codec: H264
 width: 1920
 height: 1080
 pixel_aspect_ratio: 1:1
 trick_play_factor: 0
 nalu_length_size: 4

Stream [1] type: Audio
 codec_string: mp4a.40.2
 time_scale: 48000
 duration: 481280 (10.0 seconds)
 is_encrypted: false
 codec: AAC
 sample_bits: 16
 num_channels: 2
 sampling_frequency: 48000
 language: eng
 seek_preroll_ns: 20833

Look for the characteristics discussed in Media file basics and notice a few things. The height and width are correct for full HD, and the audio and video codecs are among the preferred codecs for their container types, AAC for audio and H264 for video. Notice also that streams are identified with numbers. These are useful for operations that manipulate the audio and video separately.

Notice that the output above doesn't show the bitrate. Despite what's missing, this output is easier to read, so we use it whenever we can. When we need information that Shaka Packager can't get, such as the bitrate, we use FFmpeg.

FFmpeg #

FFmpeg is also a free application for recording, converting, and streaming media files. Its capabilities aren't better or worse than Shaka Packager's. They're just different.

The basic pattern for an FFmpeg command looks like this:

ffmpeg [GeneralOptions] [InputFileOptions] -i input [OutputFileOptions] output

Like Shaka Packager this application can handle multiple streams. Some of its options are used in multiple locations and manipulate the file output differently depending on where they are in the command. Be aware of this as you look at FFmpeg questions on Stack Overflow and similar sites.

We'll again compare the basic pattern to the example for displaying file characteristics.

    ffmpeg [GeneralOptions] [InputFileOptions] -i input        [OutputFileOptions] output

    ffmpeg                                     -i glocken.mp4

In addition to the information we requested, this also prints an error message as shown in the example below. That's because this is technically an incorrect usage of FFmpeg. We use it because it displays information we care about.

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'glocken.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.17.100
  Duration: 00:01:47.53, start: 0.000000, bitrate: 10715 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc), 1920x1080, 10579 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
At least one output file must be specified

Installing applications with Docker #

If you plan to follow along and try our commands you could either install the required tools manually, or take the easy path and use Docker. We suggest using Docker, because this is going to save you a lot of time. On top of that we've provided the instructions to get you set up quickly.

Start by creating a new directory somewhere on your computer named media-tools; you can use whatever name you like, just note that the following instructions assume you are using media-tools as the directory name.
Create a docker and media directory inside of media-tools. This will keep your media directory out of the build context. This is important because media is where files are stored that we plan to do operations on, and some of them could be quite large. Putting the Dockerfile directly in media-tools would slow down building the image if you ever rebuild it down the road—perhaps to change the versions installed.

Create /media-tools/docker/Dockerfile, and add the following build instructions:

FROM google/shaka-packager:release-v2.4.3 as packager
FROM jrottenberg/ffmpeg:4.3.2-alpine38
COPY --from=packager /usr/bin /usr/bin
ENTRYPOINT  ["sh"]

Build the image:
```
docker build -t media-tools ./docker
```

Run the image as an interactive shell. On Linux:

docker run -w /media -v ${PWD}/media:/media -it --rm media-tools
/media #

On Windows:

docker run -w /media -v %cd%/media:/media -it --rm media-tools
/media #

While running the image you can check versions for both FFmpeg and Shaka Packager to validate everything was successful by running ffmpeg -version and packager --version. The output should look like this:

/media # ffmpeg -version
ffmpeg version 4.3.2 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 6.4.0 (Alpine 6.4.0)
configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libass --enable-fontconfig --enable-libfreetype --enable-libvidstab --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libbluray --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-libsrt --enable-libaribb24 --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100

/media # packager --version
packager version v2.4.3-dd9870075f-release

Now that you've tried your hand at using Shaka Packager and FFmpeg, you can continue learning the basic concepts, next up Media streaming basics.

The

2014-02-15T00:00:00Z

You've properly prepared a video file for the web. You've given it correct dimensions and the correct resolution. You've even created separate WebM and MP4 files for different browsers.

For anyone to see your video, you still need to add it to a web page. Doing so properly requires adding two HTML elements: the <video> element and the <source> element. In addition to basics about these tags, this article explains attributes you should add to those tags to craft a good user experience.

Specify a single file #

Although it's not recommended, you can use the video element by itself. Always use the type attribute as shown below. The browser uses this to determine if it can play the provided video file. If it can't, the enclosed text displays.

<video src="chrome.webm" type="video/webm">
    <p>Your browser cannot play the provided video file.</p>
</video>

Specify multiple file formats #

Recall from Media file basics that not all browsers support the same video formats. The <source> element lets you specify multiple formats as a fallback in case the user's browser doesn't support one of them.

The example below produces the embedded video that is used as an example later in this article.

<video controls>
  <source src="https://storage.googleapis.com/web-dev-assets/video-and-source-tags/chrome.webm" type="video/webm">
  <source src="https://storage.googleapis.com/web-dev-assets/video-and-source-tags/chrome.mp4" type="video/mp4">
  <p>Your browser cannot play the provided video file.</p>
</video>

Try it on Glitch (source)

You should always add a type attribute to the <source> tags event though it is optional. This ensures that the browser only downloads the file that it is capable of playing.

This approach has several advantages over serving different HTML or server-side scripting, especially on mobile:

You can list formats in order of preference.
Client-side switching reduces latency; only one request is made to get content.
Letting the browser choose a format is simpler, quicker, and potentially more reliable than using a server-side support database with user-agent detection.
Specifying each file source's type improves network performance; the browser can select a video source without having to download part of the video to "sniff" the format.

These issues are especially important in mobile contexts, where bandwidth and latency are at a premium, and the user's patience is likely limited. Omitting the type attribute can affect performance when there are multiple sources with unsupported types.

There are a few ways you can dig into the details. Check out A Digital Media Primer for Geeks to find out more about how video and audio work on the web. You can also use remote debugging in DevTools to compare network activity with type attributes and without type attributes.

Specify start and end times #

Save bandwidth and make your site feel more responsive: use media fragments to add start and end times to the video element.

This browser does not support the video element.

To use a media fragment, add #t=[start_time][,end_time] to the media URL. For example, to play the video from seconds 5 to 10, specify:

<source src="video/chrome.webm#t=5,10" type="video/webm">

You can also specify the times in <hours>:<minutes>:<seconds>. For example, #t=00:01:05 starts the video at one minute, five seconds. To play only the first minute of video, specify #t=,00:01:00.

You can use this feature to deliver multiple views on the same video—like cue points in a DVD–without having to encode and serve multiple files.

For this feature to work, your server must support range requests and that capability must be enabled. Most servers enable range requests by default. Because some hosting services turn them off, you should confirm that range requests are available for using fragments on your site.

Fortunately, you can do this in your browser developer tools. In Chrome, for instance, it's in the Network panel. Look for the Accept-Ranges header and verify that it says bytes. In the image, I've drawn a red box around this header. If you do not see bytes as the value, you'll need to contact your hosting provider.

Chrome DevTools screenshot: Accept-Ranges: bytes.

Include a poster image #

Add a poster attribute to the video element so that viewers have an idea of the content as soon as the element loads, without needing to download the video or start playback.

<video poster="poster.jpg" ...>
  …
</video>

A poster can also be a fallback if the video src is broken or if none of the supplied video formats are supported. The only downside to a poster images is an additional file request, which consumes some bandwidth and requires rendering. For more information see Efficiently encode images.

Don't

Without a fallback poster, the video just looks broken.

A fallback poster makes it seem as if the first frame has been captured.

Ensure videos don't overflow containers #

When video elements are too big for the viewport, they may overflow their container, making it impossible for the user to see the content or use the controls.

Android Chrome screenshot, portrait: unstyled video element overflows viewport.

Android Chrome screenshot, landscape: unstyled video element overflows viewport.

You can control video dimensions using CSS. If CSS does not meet all of your needs, JavaScript libraries and plugins such as FitVids (outside the scope of this article) can help, even for videos from YouTube and other sources. Unfortunately, these resources can increase your network payload sizes with negative consequences for your revenues and your users' wallets.

For simple uses like the ones I'm describing here, use CSS media queries to specify the size of elements depending on the viewport dimensions; max-width: 100% is your friend.

For media content in iframes (such as YouTube videos), try a responsive approach (like the one proposed by John Surdakowski).

CSS #

.video-container {
    position: relative;
    padding-bottom: 56.25%;
    padding-top: 0;
    height: 0;
    overflow: hidden;
}

.video-container iframe,
.video-container object,
.video-container embed {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
}

HTML #

<div class="video-container">
  <iframe
    src="//www.youtube.com/embed/l-BA9Ee2XuM"
    frameborder="0"
    width="560"
    height="315"
  ></iframe>
</div>

Try it

Compare the responsive sample to the unresponsive version. As you can see, the unresponsive version isn't a great user experience.

Device orientation #

Device orientation isn't an issue for desktop monitors or laptops, but it's hugely important when considering web page design for mobile devices and tablets.

Safari on iPhone does a good job of switching between portrait and landscape orientation:

Screenshot of video playing in Safari on iPhone, portrait.

Screenshot of video playing in Safari on iPhone, landscape.

Device orientation on an iPad and Chrome on Android can be problematic. For example, without any customization a video playing on an iPad in landscape orientation looks like this:

Screenshot of video playing in Safari on iPad, landscape.

Setting the video width: 100% or max-width: 100% with CSS can resolve many device orientation layout problems.

Autoplay #

The autoplay attribute controls whether the browser downloads and plays a video immediately. The precise way it works depends on the platform and browser.

Chrome: Depends on multiple factors including but not limited to whether the viewing is on desktop and whether the mobile user has added your site or app to their homescreen. For details, see Autoplay best practices.
Firefox: Blocks all video and sound, but gives users the ability to relax these restrictions for either all sites or particular sites. For details, see Allow or block media autoplay in Firefox
Safari: Has historically required a user gesture, but has been relaxing that requirement in recent versions. For details, see New <video> Policies for iOS.

Even on platforms where autoplay is possible, you need to consider whether it's a good idea to enable it:

Data usage can be expensive.
Playing media before the user wants it can hog bandwidth and CPU, and thereby delay page rendering.
Users may be in a context where playing video or audio is intrusive.

Preload #

The preload attribute provides a hint to the browser as to how much information or content to preload.

Value	Description
`none`	The user might chose not to watch the video, so don't preload anything.
`metadata`	Metadata (duration, dimensions, text tracks) should be preloaded, but with minimal video.
`auto`	Downloading the entire video right away is considered desirable. An empty string produces the same result.

The preload attribute has different effects on different platforms. For example, Chrome buffers 25 seconds of video on desktop but none on iOS or Android. This means that on mobile, there may be playback startup delays that don't happen on desktop. See Fast playback with audio and video preload or Steve Souders' blog for more details.

Now that you know how to add media to your web page it's time to learn about Media accessibility where you will add captions to your video for hearing impaired, or when playing the audio is not a viable option.

Derek Herman on web.dev

Media encryption

Clear Key encryption #

Create a key #

Create an IV #

Encrypt with Clear Key #

Create a key information file #

Encrypt for HLS #

Widevine encryption #

Encrypt with Widevine #

Media conversion sequence #

DASH/WebM #

DASH/MP4 #

HLS/MP4 #

Media frameworks

Benefits of using a framework #

Shaka Player #

JW Player #

Video.js #

Other frameworks #

Media streaming basics

Streaming use cases #

Streaming protocols #

Using <video> to play a stream #

Media Source Extensions #

Basic example #

Media chunks format #

PWA with offline streaming

Downloading and storing a large media file #

Downloading media files using the Fetch API #

Resuming downloads #

Custom write buffer for IndexedDB #

Serving a media file from offline storage #

Other considerations #

Media accessibility

Add <track> tags #

WebVTT file structure #

Kinds of text tracks #

Add media to a web page

Media file basics

Serving video files #

Containers and codecs, and streams? #

Codec formats #

Bitrate and resolution #

Media conversion

Display characteristics #

Characteristics using Shaka Packager #

Characteristics using FFmpeg #

Demux (separate) the audio and video streams #

Shaka Packager demuxing #

FFmpeg demuxing #

Remux (combine) the audio and video streams #

Change characteristics #

Bitrate #

Dimensions (resolution) #

File type #

Synchronize audio and video #

Video-on-demand and live-streaming #

DASH/MPD #

HLS #

Bitrate

Containers and codecs

Containers #

Check your work #

Codecs #

Video #

Audio #

What is a media experience?

The technical requirements #

Displaying video on the web #

What we're going to learn #

Prepare media files for the web

Resolution

Media application basics

Shaka Packager #

FFmpeg #

Installing applications with Docker #

The and tags

Specify a single file #

Specify multiple file formats #

Using `<video>` to play a stream #

Add `<track>` tags #

The