Jaroslav Polakovič on web.dev

Media streaming basics

2021-07-05T00:00:00Z

In this article, you are going to learn about the more advanced concept of media streaming and by the end should have a good understanding of the various streaming use cases, protocols, and extensions. Let's start with an explanation of what streaming actually is.

Media streaming is a way of delivering and playing back media content piece by piece. Instead of loading a single file, which can be slow if not optimized for the network, the player reads a manifest file describing how the target media is split into individual chunks of data. Media chunks are later dynamically stitched back together at runtime—probably at different bitrates, which you'll learn about later.

Keep in mind that to provide streaming on your website the server must support the Range HTTP request header. Learn more about the Accept-Ranges header in The <video> and <source> tags article.

Streaming use cases #

Producing media chunks and the necessary manifests describing the stream is not exactly straightforward, but streaming unlocks some interesting use cases that are not possible to achieve just by pointing a <video> element to a set of static source files. You'll learn more about how to add media to a web page in a later section. First, you should know about a few use cases for streaming multimedia if you want to go further than just loading multiple files into the <video> element.

Adaptive streaming is where media chunks are encoded in several bitrates, and the highest quality media chunk that fits the client's currently available bandwidth is returned to the media player.
Live broadcast is where media chunks are encoded and made available in real time.
Injecting media is where other media like advertisements are injected into a stream without the player having to change the media source.

Streaming protocols #

The two most commonly used streaming protocols on the web are Dynamic Adaptive Streaming over HTTP (DASH) and HTTP Live Streaming (HLS). Players that support these protocols will fetch the generated manifest file, figure out which media chunks to request, and then combine them into the final media experience.

Using `<video>` to play a stream #

Many browsers are not going to play your stream natively. While there is some native support for HLS playback, browsers generally don't support native DASH stream playback. This means often it's not enough to simply point the <source> in the <video> element to a manifest file.

<video controls>
  <source src="manifest.mpd" type="application/dash+xml">
</video>

What may seem as a deficit is actually a strength in disguise. Streams are powerful and applications that consume streams have different needs.

Manifest files usually describe many variants of single media. Think different bitrates, several audio tracks, and even the same media encoded in different formats.

Some applications may want to keep a larger amount of video in the buffer, others may want to prefetch the first few seconds of video from an upcoming episode, and some want to implement their own logic for adaptive streaming. This is where you would want to have some sort of built-in browser feature to generate media streams for playback, and it just so happens there is one.

Media Source Extensions #

Thankfully, the W3C defined something called Media Source Extensions (MSE) that will let JavaScript generate our media streams. In a nutshell, MSE allows developers to attach a MediaSource object to a <video> element and have it play back whatever media data is pumped into the buffers attached to the MediaSource instance.

Basic example #

const videoEl = document.querySelector('video');
const mediaSource = new MediaSource();

video.src = URL.createObjectURL(mediaSource);
mediaSource.addEventListener(
  'sourceopen',
  () => {
    const mimeString = 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"';
    const buffer = mediaSource.addSourceBuffer(mimeString);

    buffer.appendBuffer( /* Video data as `ArrayBuffer` object. */ )
  }
);

The simplified example above illustrates a few things:

As far as <video> is concerned, it is receiving media data from a URL.
The generated URL is just a pointer to a MediaSource instance.
The MediaSource instance creates one or more SourceBuffer instances.
We then just append binary media data into the buffer, e.g. using fetch.

While these basic concepts are simple, and it is certainly possible to write a DASH and HLS compatible video player from scratch, most people usually pick one of the mature open source solutions that already exist, such as Shaka Player, JW Player, or Video.js to name a few.

However, we have created a demo Media PWA called Kino that demonstrates how you would go about developing your own basic streaming media website that provides offline media playback using just the simple <video> element. There are plans in our roadmap to support frameworks and digital rights management, among other features. So check back for updates from time to time, or request a feature. Read more about it in the PWA with offline streaming article.

Media chunks format #

For a long time, DASH and HLS required media chunks to be encoded in different formats. In 2016, however, support for standard fragmented MP4 (fMP4) files was added to HLS, a format that DASH also supports.

Video chunks using the fMP4 container and the H.264 codec are supported by both protocols and playable by a vast majority of players. This allows content producers to encode their videos just once, which in turn saves time and disk space.

To achieve better quality and lower files sizes, you may want to choose to encode several sets of media chunks using more efficient formats like VP9, though before we get to far ahead you will first need to learn how to Prepare media files for the web, and that's up next.

PWA with offline streaming

2021-07-05T00:00:00Z

Progressive Web Apps bring a lot of features previously reserved for native applications to the web. One of the most prominent features associated with PWAs is an offline experience.

Even better would be an offline streaming media experience, which is an enhancement you could offer to your users in a few different ways. However, this creates a truly unique problem—media files can be very large. So you might be asking:

How do I download and store a large video file?
And how do I serve it to the user?

In this article we will discuss answers to these questions, while referencing the Kino demo PWA we built that provides you with practical examples of how you can implement an offline streaming media experience without using any functional or presentational frameworks. The following examples are mainly for educational purposes, because in most cases you should probably use one of the existing Media Frameworks to provide these features.

Unless you have a good business case for developing your own, building a PWA with offline streaming has its challenges. In this article you will learn about the APIs and techniques used to provide users with a high-quality offline media experience.

Downloading and storing a large media file #

Progressive Web Apps usually use the convenient Cache API to both download and store the assets required to provide the offline experience: documents, stylesheets, images, and others.

Here is a basic example of using the Cache API within a Service Worker:

const cacheStorageName = 'v1';

this.addEventListener('install', function(event) {
  event.waitUntil(
    caches.open(cacheStorageName).then(function(cache) {
      return cache.addAll([
        'index.html',
        'style.css',
        'scripts.js',

        // Don't do this.
        'very-large-video.mp4',
      ]);
    })
  );
});

While the example above does technically work, using the Cache API has several limitations that makes its use with large files impractical.

For example, the Cache API doesn't:

Allow you to easily pause and resume downloads
Let you track the progress of downloads
Offer a way to properly respond to HTTP range requests

All of these issues are pretty serious limitations for any video application. Let's review some other options that might be more appropriate.

Nowadays, the Fetch API is a cross-browser way to asynchronously access remote files. In our use case it allows you to access large video files as a stream and store them incrementally as chunks using an HTTP range request.

Now that you can read the chunks of data with the Fetch API you also need to store them. Chances are there is a bunch of metadata associated with your media file such as: name, description, runtime length, category, etc.

You're not storing just the one media file, you are storing a structured object, and the media file is just one of its properties.

In this case the IndexedDB API provides an excellent solution to store both the media data and metadata. It can hold huge amounts of binary data easily, and it also offers indexes that allow you to perform very fast data lookups.

Downloading media files using the Fetch API #

We built a couple of interesting features around the Fetch API in our demo PWA, which we named Kino—the source code is public so feel free to review it.

The ability to pause and resume incomplete downloads.
A custom buffer for storing chunks of data in the database.

Before showing how those features are implemented, we'll first do a quick recap of how you can use the Fetch API to download files.

/**
 * Downloads a single file.
 *
 * @param {string} url URL of the file to be downloaded.
 */
async function downloadFile(url) {
  const response = await fetch(url);
  const reader = response.body.getReader();
  do {
    const { done, dataChunk } = await reader.read();
    // Store the `dataChunk` to IndexedDB.
  } while (!done);
}

Notice that await reader.read() is in a loop? That's how you'll receive chunks of data from a readable stream as they arrive from the network. Consider how useful this is: you can start processing your data even before it all arrives from the network.

Resuming downloads #

When a download is paused or interrupted, the data chunks that have arrived will be safely stored in an IndexedDB database. You can then display a button to resume a download in your application. Because the Kino demo PWA server supports HTTP range requests resuming a download is somewhat straightforward:

async downloadFile() {
  // this.currentFileMeta contains data from IndexedDB.
  const { bytesDownloaded, url, downloadUrl } = this.currentFileMeta;
  const fetchOpts = {};

  // If we already have some data downloaded,
  // request everything from that position on.
  if (bytesDownloaded) {
    fetchOpts.headers = {
      Range: `bytes=${bytesDownloaded}-`,
    };
  }

  const response = await fetch(downloadUrl, fetchOpts);
  const reader = response.body.getReader();

  let dataChunk;
  do {
    dataChunk = await reader.read();
    if (!dataChunk.done) this.buffer.add(dataChunk.value);
  } while (!dataChunk.done && !this.paused);
}

Custom write buffer for IndexedDB #

On paper, the process of writing dataChunk values into an IndexedDB database is simple. Those values already are ArrayBuffer instances, which are storable in IndexedDB directly, so we can just create an object of an appropriate shape and store it.

const dataItem = {
  url: fileUrl,
  rangeStart: dataStartByte,
  rangeEnd: dataEndByte,
  data: dataChunk,
}

// Name of the store that will hold your data.
const storeName = 'fileChunksStorage'

// `db` is an instance of `IDBDatabase`.
const transaction = db.transaction([storeName], 'readwrite');
const store = transaction.objectStore(storeName);
const putRequest = store.put(data);

putRequest.onsuccess = () => { ... }

While this approach works, you will likely discover that your IndexedDB writes are significantly slower than your download. This isn't because IndexedDB writes are slow, it's because we are adding a lot of transactional overhead by creating a new transaction for every data chunk that we receive from a network.

The downloaded chunks can be rather small and can be emitted by the stream in rapid succession. You need to limit the rate of IndexedDB writes. In the Kino demo PWA we do this by implementing an intermediary write buffer.

As data chunks arrive from the network, we append them to our buffer first. If the incoming data doesn't fit, we flush the full buffer into the database and clear it before appending the rest of the data. As a result our IndexedDB writes are less frequent, which leads to significantly improved write performance.

Serving a media file from offline storage #

Once you have a media file downloaded, you probably want your service worker to serve it from IndexedDB instead of fetching the file from the network.

/**
 * The main service worker fetch handler.
 *
 * @param {FetchEvent} event Fetch event.
 */
const fetchHandler = async (event) => {
  const getResponse = async () => {
    // Omitted Cache API code used to serve static assets.

    const videoResponse = await getVideoResponse(event);
    if (videoResponse) return videoResponse;

    // Fallback to network.
    return fetch(event.request);
  };
  event.respondWith(getResponse());
};
self.addEventListener('fetch', fetchHandler);

So what do you need to do in getVideoResponse()?

The event.respondWith() method expects a Response object as a parameter.
The Response() constructor tells us that there are several types of objects we could use to instantiate a Response object: a Blob, BufferSource, ReadableStream, and more.
We need an object that doesn't hold all of its data in memory, so we'll probably want to choose the ReadableStream.

Also, because we're dealing with large files, and we wanted to allow browsers to only request the part of the file they currently need, we needed to implement some basic support for HTTP range requests.

/**
 * Respond to a request to fetch offline video file and construct a response
 * stream.
 *
 * Includes support for `Range` requests.
 *
 * @param {Request} request  Request object.
 * @param {Object}  fileMeta File meta object.
 *
 * @returns {Response} Response object.
 */
const getVideoResponse = (request, fileMeta) => {
  const rangeRequest = request.headers.get('range') || '';
  const byteRanges = rangeRequest.match(/bytes=(?<from>[0-9]+)?-(?<to>[0-9]+)?/);

  // Using the optional chaining here to access properties of
  // possibly nullish objects.
  const rangeFrom = Number(byteRanges?.groups?.from || 0);
  const rangeTo = Number(byteRanges?.groups?.to || fileMeta.bytesTotal - 1);

  // Omitting implementation for brevity.
  const streamSource = {
     pull(controller) {
       // Read file data here and call `controller.enqueue`
       // with every retrieved chunk, then `controller.close`
       // once all data is read.
     }
  }
  const stream = new ReadableStream(streamSource);

  // Make sure to set proper headers when supporting range requests.
  const responseOpts = {
    status: rangeRequest ? 206 : 200,
    statusText: rangeRequest ? 'Partial Content' : 'OK',
    headers: {
      'Accept-Ranges': 'bytes',
      'Content-Length': rangeTo - rangeFrom + 1,
    },
  };
  if (rangeRequest) {
    responseOpts.headers['Content-Range'] = `bytes ${rangeFrom}-${rangeTo}/${fileMeta.bytesTotal}`;
  }
  const response = new Response(stream, responseOpts);
  return response;

Feel free to check out the Kino demo PWA service worker source code to find out how we are reading file data from IndexedDB and constructing a stream in a real application.

Other considerations #

With the main obstacles out of your way, you can now start adding some nice-to-have features to your video application. Here are a few examples of features you would find in the Kino demo PWA:

Media Session API integration that allows your users to control media playback using dedicated hardware media keys or from media notification popups.
Caching of other assets associated with the media files like subtitles, and poster images using the good old Cache API.
Support for video streams (DASH, HLS) download within the app. Because stream manifests generally declare multiple sources of different bitrates, you need to transform the manifest file and only download one media version before storing it for offline viewing.

Up next, you will learn about Fast playback with audio and video preload.