François Beaufort on web.dev

How to process audio from the user's microphone

2023-01-19T00:00:00Z

Accessing the user's camera and microphone is possible on the web platform with the Media Capture and Streams API. The getUserMedia() method prompts the user to access a camera and/or microphone to capture as a media stream. This stream can then be processed in a separate Web Audio thread with an AudioWorklet that provides very low latency audio processing.

The example below shows how you can process audio from the user's microphone in a performant way.

let stream;

startMicrophoneButton.addEventListener("click", async () => {
  // Prompt the user to use their microphone.
  stream = await navigator.mediaDevices.getUserMedia({
    audio: true,
  });
  const context = new AudioContext();
  const source = context.createMediaStreamSource(stream);

  // Load and execute the module script.
  await context.audioWorklet.addModule("processor.js");
  // Create an AudioWorkletNode. The name of the processor is the
  // one passed to registerProcessor() in the module script.
  const processor = new AudioWorkletNode(context, "processor");

  source.connect(processor).connect(context.destination);
  log("Your microphone audio is being used.");
});

stopMicrophoneButton.addEventListener("click", () => {
  // Stop the stream.
  stream.getTracks().forEach(track => track.stop());

  log("Your microphone audio is not used anymore.");
});

// processor.js
// This file is evaluated in the audio rendering thread
// upon context.audioWorklet.addModule() call.

class Processor extends AudioWorkletProcessor {
  process([input], [output]) {
    // Copy inputs to outputs.
    output[0].set(input[0]);
    return true;
  }
}

registerProcessor("processor", Processor);

Browser support #

MediaDevices.getUserMedia() #

Browser support

Chrome 53, Supported 53
Firefox 36, Supported 36
Edge 12, Supported 12
Safari 11, Supported 11

Source

Web Audio #

Browser support

Chrome 35, Supported 35
Firefox 25, Supported 25
Edge 12, Supported 12
Safari 14.1, Supported 14.1

Source

AudioWorklet #

Browser support

Chrome 66, Supported 66
Firefox 76, Supported 76
Edge 79, Supported 79
Safari 14.1, Supported 14.1

Source

Demo #

New patterns for media apps

2022-11-03T00:00:00Z

Let's welcome our new collection of media patterns to help you create modern media experiences on the web for great audio and video playback:

We hope these patterns will help you build amazing apps, and we're looking forward to your feedback! You can provide feedback by tweeting at @ChromiumDev or filing an Issue.

Photo by Kenny Eliason.

How to customize media notifications

2022-11-03T00:00:00Z

The Media Session API allows websites to let users know what's currently playing in their browser and control it without returning to the page that launched it. The user experience can be customized through metadata in custom media notifications, media events such as playing, pausing, seeking, and track changing. These customizations are available in several contexts, including desktop media hubs, media notifications on mobile, and even on wearable devices.

Browser support

Chrome 73, Supported 73
Firefox 82, Supported 82
Edge 79, Supported 79
Safari 15, Supported 15

Source

Media hub on desktop, media notification on mobile, and a wearable device.

When a website is playing audio or video, users automatically get media notifications either in the notification tray on mobile, or the media hub on desktop. The browser does its best to show appropriate information by using the document's title and the largest icon image it can find. The Media Session API allows you to customize the media notification with some richer media metadata such as the title, artist name, album name, and artwork as shown below.

Anatomy of a media notification on mobile.

The example below shows you how to create a custom media notification and respond to basic media actions such as play and pause.

const video = document.querySelector("video");

navigator.mediaSession.metadata = new MediaMetadata({
  title: "Never Gonna Give You Up",
  artist: "Rick Astley",
  album: "Whenever You Need Somebody",
  artwork: [
    { src: "https://via.placeholder.com/96",  sizes: "96x96" },
    { src: "https://via.placeholder.com/128", sizes: "128x128" },
    { src: "https://via.placeholder.com/256", sizes: "256x256" },
    { src: "https://via.placeholder.com/512", sizes: "512x512" },
  ],
});

navigator.mediaSession.setActionHandler("play", async () => {
  // Resume playback
  try {
    await video.play();
  } catch (err) {
    console.error(err.name, err.message);
  }
});

navigator.mediaSession.setActionHandler("pause", () => {
  // Pause active playback
  video.pause();
});

video.addEventListener("play", () => {
  navigator.mediaSession.playbackState = "playing";
});

video.addEventListener("pause", () => {
  navigator.mediaSession.playbackState = "paused";
});

Demo #

How to record audio from the user's microphone

2022-10-19T00:00:00Z

Accessing the user's camera and microphone is possible on the web platform with the Media Capture and Streams API. The getUserMedia() method prompts the user to access a camera and/or microphone to capture as a media stream. This stream can then be recorded with the MediaRecorder API or shared with others over the network. The recording can be saved to a local file via the showOpenFilePicker() method.

The example below shows how you can record audio from the user's microphone in the WebM format and save the recording to the user's file system.

let stream;
let recorder;

startMicrophoneButton.addEventListener("click", async () => {
  // Prompt the user to use their microphone.
  stream = await navigator.mediaDevices.getUserMedia({ audio: true });
  recorder = new MediaRecorder(stream);
});

stopMicrophoneButton.addEventListener("click", () => {
  // Stop the stream.
  stream.getTracks().forEach(track => track.stop());
});

startRecordButton.addEventListener("click", async () => {
  // For the sake of more legible code, this sample only uses the
  // `showSaveFilePicker()` method. In production, you need to
  // cater for browsers that don't support this method, as
  // outlined in https://web.dev/patterns/files/save-a-file/.

  // Prompt the user to choose where to save the recording file.
  const suggestedName = "microphone-recording.webm";
  const handle = await window.showSaveFilePicker({ suggestedName });
  const writable = await handle.createWritable();

  // Start recording.
  recorder.start();
  recorder.addEventListener("dataavailable", async (event) => {
    // Write chunks to the file.
    await writable.write(event.data);
    if (recorder.state === "inactive") {
      // Close the file when the recording stops.
      await writable.close();
    }
  });
});

stopRecordButton.addEventListener("click", () => {
  // Stop the recording.
  recorder.stop();
});

Browser support #

MediaDevices.getUserMedia() #

Browser support

Chrome 53, Supported 53
Firefox 36, Supported 36
Edge 12, Supported 12
Safari 11, Supported 11

Source

MediaRecorder API #

Browser support

Chrome 47, Supported 47
Firefox 25, Supported 25
Edge 79, Supported 79
Safari 14, Supported 14

Source

File System Access API's showSaveFilePicker() #

Browser support

Chrome 86, Supported 86
Firefox, Not supported
Edge 86, Supported 86
Safari, Not supported

Source

Demo #

How to add Picture-in-Picture to video controls

2022-10-14T00:00:00Z

The modern way #

The Picture-in-Picture (PiP) Web API allows websites to create a floating video window always on top of other windows so that users may continue consuming media while they interact with other content sites or applications on their device.

Browser support

Chrome, Not supported
Firefox, Not supported
Edge, Not supported
Safari, Not supported

The example below shows you how to create a button that users can use to toggle Picture-in-Picture on a video.

togglePipButton.addEventListener("click", async (event) => {
  togglePipButton.disabled = true;
  try {
    if (video !== document.pictureInPictureElement) {
      await video.requestPictureInPicture();
    } else {
      await document.exitPictureInPicture();
    }
  } finally {
    togglePipButton.disabled = false;
  }
});

video.addEventListener("enterpictureinpicture", (event) => {
  togglePipButton.classList.add("on");
});

video.addEventListener("leavepictureinpicture", (event) => {
  togglePipButton.classList.remove("on");
});

/* Feature detection */
if ("pictureInPictureEnabled" in document) {
  // Set button ability depending on whether Picture-in-Picture can be used.
  setPipButton();
  video.addEventListener("loadedmetadata", setPipButton);
  video.addEventListener("emptied", setPipButton);
} else {
  // Hide button if Picture-in-Picture is not supported.
  togglePipButton.hidden = true;
}

function setPipButton() {
  togglePipButton.disabled =
    video.readyState === 0 ||
    !document.pictureInPictureEnabled ||
    video.disablePictureInPicture;
}

The classic way #

Before the availability of the Picture-in-Picture Web API, there was no way to create a floating video window always on top of other windows. Playing the video on top of other elements on the web page is possible though with CSS.

The example below shows you how to display your video on top of other elements at the bottom right corner of your web page when the user clicks a button.

toggleFakePipButton.addEventListener("click", (event) => {
  video.classList.toggle("fake-pip");
});

/* Place the video in the bottom right corner on top of everything else via CSS. */
video.fake-pip {
  position: fixed;
  z-index: 1000;
  bottom: 10px;
  right: 10px;
}

Demo #

How to record the user's screen

2022-10-14T00:00:00Z

Sharing tabs, windows, and screens is possible on the web platform with the Screen Capture API. The getDisplayMedia() method allows the user to select a screen to capture as a media stream. This stream can then be recorded with the MediaRecorder API or shared with others over the network. The recording can be saved to a local file via the showOpenFilePicker() method.

The example below shows how you can record the user's screen in the WebM format, locally preview on the same page, and save the recording to the user's file system.

let stream;
let recorder;

shareScreenButton.addEventListener("click", async () => {
  // Prompt the user to share their screen.
  stream = await navigator.mediaDevices.getDisplayMedia();
  recorder = new MediaRecorder(stream);
  // Preview the screen locally.
  video.srcObject = stream;
});

stopShareScreenButton.addEventListener("click", () => {
  // Stop the stream.
  stream.getTracks().forEach(track => track.stop());
  video.srcObject = null;
});

startRecordButton.addEventListener("click", async () => {
  // For the sake of more legible code, this sample only uses the
  // `showSaveFilePicker()` method. In production, you need to
  // cater for browsers that don't support this method, as
  // outlined in https://web.dev/patterns/files/save-a-file/.

  // Prompt the user to choose where to save the recording file.
  const suggestedName = "screen-recording.webm";
  const handle = await window.showSaveFilePicker({ suggestedName });
  const writable = await handle.createWritable();

  // Start recording.
  recorder.start();
  recorder.addEventListener("dataavailable", async (event) => {
    // Write chunks to the file.
    await writable.write(event.data);
    if (recorder.state === "inactive") {
      // Close the file when the recording stops.
      await writable.close();
    }
  });
});

stopRecordButton.addEventListener("click", () => {
  // Stop the recording.
  recorder.stop();
});

Browser support #

MediaDevices.getDisplayMedia() #

Browser support

Chrome 72, Supported 72
Firefox 66, Supported 66
Edge 79, Supported 79
Safari 13, Supported 13

Source

MediaRecorder API #

Browser support

Chrome 47, Supported 47
Firefox 25, Supported 25
Edge 79, Supported 79
Safari 14, Supported 14

Source

File System Access API's showSaveFilePicker() #

Browser support

Chrome 86, Supported 86
Firefox, Not supported
Edge 86, Supported 86
Safari, Not supported

Source

Demo #

How to create app shortcuts

2022-10-10T00:00:00Z

App shortcuts help users quickly start common or recommended tasks within your web app. Easy access to those tasks from anywhere the app icon is displayed will enhance users' productivity as well as increase their engagement with the web app.

The modern way #

Define app shortcuts in the web app manifest #

The app shortcuts menu is invoked by right-clicking the app icon in the taskbar (Windows) or dock (macOS) on the user's desktop, or by touch & holding the app's launcher icon on Android.

App shortcuts menu opened on Android

App shortcuts menu opened on Windows

The app shortcuts menu is shown only for Progressive Web Apps that are installed. Check out Installation in our Learn PWA module to learn about installability requirements.

Each app shortcut expresses a user intent, each of which is associated with a URL within the scope of your web app. The URL is opened when a user activates the app shortcut.

App shortcuts are optionally declared in the shortcuts array member of the web app manifest. Below is an example of a potential web app manifest.

{
  "name": "Player FM",
  "start_url": "https://player.fm?utm_source=homescreen",
  "shortcuts": [
    {
      "name": "Open Play Later",
      "short_name": "Play Later",
      "description": "View the list of podcasts you saved for later",
      "url": "/play-later?utm_source=homescreen",
      "icons": [{ "src": "/icons/play-later.png", "sizes": "192x192" }]
    },
    {
      "name": "View Subscriptions",
      "short_name": "Subscriptions",
      "description": "View the list of podcasts you listen to",
      "url": "/subscriptions?utm_source=homescreen",
      "icons": [{ "src": "/icons/subscriptions.png", "sizes": "192x192" }]
    }
  ]
}

Browser support

Chrome 96, Supported 96
Firefox, Not supported
Edge 96, Supported 96
Safari, Not supported

Source

The classic way #

Let user drag links to the bookmark bar #

If the app is not installed yet, you can suggest the user to drag some links from your web page and release them in their browser bookmark bar. That way, they can quickly start common or recommended tasks within your web app.

Demo #

Synchronize audio and video playback on the web

2022-06-01T00:00:00Z

The outputLatency property of an AudioContext instance provides an estimate of audio hardware's output latency (for example, that of Bluetooth earbuds or of an external USB audio interface). This property is useful when you want to:

Synchronize the existing audio material and the newly recorded material. (in a music production scenario)
Synchronize the Web Audio output and other media (e.g. video or MIDI playback).

In this WebCodecs demo (source), the WebCodecs API is used to decode a MediaStream into raw video and audio data, and then played back into a HTML <canvas> element with audio data coming from an Audio Worklet. The outputLatency property allows the demo to determine when a given audio timestamp is reaching the user's ears and then properly paint video frames to match that.

Try it out for yourself, play the video with your favorite Bluetooth headset (🎧), wait for the bird (🐦) (see above), and toggle the checkbox (☑️) to observe audio playback changes. The total output latency value is updated in real time.

AudioContext outputLatency #

Browser support

Chrome 102, Supported 102
Firefox 70, Supported 70
Edge 102, Supported 102
Safari, Not supported

Source

Hero image by Wahid Khene on Unsplash.

Debug media playback errors on the web

2022-04-12T00:00:00Z

Debugging HTML media elements, such as <video> and <audio>, can be hard as decoder implementations vary in what they consider errors (hardware decoders generally being the most strict), especially when a playback may use more esoteric features of a particular codec. Luckily for us, there are a variety of tools we can leverage to help.

When debugging media playback errors, the first thing we usually check out is the MediaError error attribute on the HTML media element. This attribute is a high level hint of what caused the media playback error. The property MediaError.code returns a numeric value which represents the kind of error that occurred on a media element. The other property MediaError.message may provide a string with some diagnostic information from the browser.

const video = document.querySelector('video');
video.addEventListener('error', () => {
  console.log('Error code: ' + video.error.code);
  console.log('Error message: ' + video.error.message);
});

These properties are most useful for telemetry analysis, and may not always provide enough information to debug playback errors. For privacy reasons, the full error text must sometimes be omitted.

To access complete error information use the Chrome DevTools "Media Panel" to view media logs. You will find plenty of information such as events, warnings, and error messages that will give you some good hints of the media playback errors.

Media Panel in Chrome DevTools.

You can also use the FFmpeg free application to check media file integrity thanks to this command:

ffmpeg -err_detect explode -i <file> -f null -

Here are some errors you could get with these commands for a video file with invalid codec:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fc62df05380] Could not find codec parameters for stream 0 (Video: none (zzzz / 0x7A7A7A7A), none(smpte170m/smpte170m/bt709, progressive), 320x240, 4 kb/s): unknown codec
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options

[matroska,webm @ 0x7fd45b705380] Unknown EBML doctype '0000'

[matroska,webm @ 0x7f8d17904d40] Element at 0x8b ending at 0x10400000095 exceeds containing master element ending at 0x9b
Truncating packet of size 9069 to 94

The MP4Box.js / ISOBMFF Box Structure Viewer is a useful tool to debugging bitstream issues. It does require already understanding MP4 to use though.

Finally, some video stream analysis professional tools like VQAnalyzer, Elecard StreamEye, and Codecian CodecVisa, may be worth your money.

User preference media features client hints headers

2021-08-02T00:00:00Z

CSS media queries, and specifically user preference media features like prefers-color-scheme or prefers-reduced-motion, have a potentially significant impact on the amount of CSS that needs to be delivered by a page, and on the experience the user is going to have when the page loads.

Focusing on prefers-color-scheme—but highlighting that the reasoning applies to other user preference media features as well—it is a best practice to not load CSS for the particular non-matching color scheme in the critical rendering path, and instead to initially only load the currently relevant CSS. One way of doing so is via <link media>.

However, high-traffic sites like Google Search that wish to honor user preference media features like prefers-color-scheme and that inline CSS for performance reasons, need to know about the preferred color scheme (or other user preference media features respectively) ideally at request time, so that the initial HTML payload already has the right CSS inlined.

Additionally, and specifically for prefers-color-scheme, sites by all means want to avoid a flash of inaccurate color theme.

The Sec-CH-Prefers-Color-Scheme and the Sec-CH-Prefers-Reduced-Motion client hint headers are the first of a series of user preference media features client hints headers that aims to solve this issue.

Background on client hints #

HTTP Client Hints defines an Accept-CH response header that servers can use to advertise their use of request headers for proactive content negotiation, colloquially referred to as client hints. The User Preference Media Features Client Hints Headers proposal defines a set of client hints aimed at conveying user preference media features. These client hints are named after the corresponding user preference media feature that they report on. For example, the currently preferred color scheme as per prefers-color-scheme is reported via the aptly named Sec-CH-Prefers-Color-Scheme client hint.

Background on critical client hints #

The client hints proposed in User Preference Media Features Client Hints Headers will presumably most commonly be used as Critical Client Hints. Critical Client Hints are Client Hints which meaningfully change the resulting resource. Such a resource should be fetched consistently across page loads (including the initial page load) to avoid jarring user-visible switches.

Client hint syntax #

User preference media features consist of a name (like prefers-reduced-motion) and allowed values (like no-preference or reduce). Each client hint header field is represented as Structured Headers for HTTP object containing an item whose value is a string. For example, to convey that the user prefers a dark theme and reduced motion, the client hints look like in the example below.

GET / HTTP/2
Host: example.com
Sec-CH-Prefers-Color-Scheme: "dark"
Sec-CH-Prefers-Reduced-Motion: "reduce"

The CSS equivalent of the information conveyed in the above client hints is @media (prefers-color-scheme: dark) {} and @media (prefers-reduced-motion: reduce) {} respectively.

Complete list of the client hints #

The list of the client hints is modeled after the user preference media features in Media Queries Level 5.

Client Hint	Allowed Values	Corresponding User Preference Media Feature
`Sec-CH-Prefers-Reduced-Motion`	`no-preference`, `reduce`	`prefers-reduced-motion`
`Sec-CH-Prefers-Reduced-Transparency`	`no-preference`, `reduce`	`prefers-reduced-transparency`
`Sec-CH-Prefers-Contrast`	`no-preference`, `less`, `more`, `custom`	`prefers-contrast`
`Sec-CH-Forced-Colors`	`active`, `none`	`forced-colors`
`Sec-CH-Prefers-Color-Scheme`	`light`, `dark`	`prefers-color-scheme`
`Sec-CH-Prefers-Reduced-Data`	`no-preference`, `reduce`	`prefers-reduced-data`

Browser support #

The Sec-CH-Prefers-Color-Scheme client hint header is supported on Chromium 93. The Sec-CH-Prefers-Reduced-Motion client hint header is supported on Chromium 108. Other vendors' feedback, namely WebKit's and Mozilla's, is pending.

Demo of `Sec-CH-Prefers-Color-Scheme` #

Try the demo in Chromium 93 and notice how the inlined CSS changes according to the user's preferred color scheme.

Demo of `Sec-CH-Prefers-Reduced-Motion` #

Try the demo in Chromium 108 and notice how the inlined CSS changes according to the user's motion preferences.

How it works #

The client makes an initial request to the server.
```
GET / HTTP/2
Host: example.com
```
The server responds, telling the client via Accept-CH that it accepts the Sec-CH-Prefers-Color-Scheme and the Sec-CH-Prefers-Contrast Client Hints, out of which as per Critical-CH it considers Sec-CH-Prefers-Color-Scheme a Critical Client Hint that it also varies the response on as conveyed by Vary.
```
HTTP/2 200 OK
Content-Type: text/html
Accept-CH: Sec-CH-Prefers-Color-Scheme, Sec-CH-Prefers-Contrast
Vary: Sec-CH-Prefers-Color-Scheme
Critical-CH: Sec-CH-Prefers-Color-Scheme
```
The client then retries the request, telling the server via Sec-CH-Prefers-Color-Scheme that it has a user preference for dark-schemed content.
```
GET / HTTP/2
Host: example.com
Sec-CH-Prefers-Color-Scheme: "dark"
```
The server can then tailor the response to the client's preferences accordingly and, for example, inline the CSS responsible for the dark theme into the response body.

Node.js example #

The Node.js example below, written for the popular Express.js framework, shows how dealing with the Sec-CH-Prefers-Color-Scheme client hint header could look like in practice. This code is what actually powers the demo above.

app.get("/", (req, res) => {
  // Tell the client the server accepts the `Sec-CH-Prefers-Color-Scheme` client hint…
  res.set("Accept-CH", "Sec-CH-Prefers-Color-Scheme");
  // …and that the server's response will vary based on its value…
  res.set("Vary", "Sec-CH-Prefers-Color-Scheme");
  // …and that the server considers this client hint a _critical_ client hint.
  res.set("Critical-CH", "Sec-CH-Prefers-Color-Scheme");
  // Read the user's preferred color scheme from the headers…
  const prefersColorScheme = req.get("sec-ch-prefers-color-scheme");
  // …and send the adequate HTML response with the right CSS inlined.
  res.send(getHTML(prefersColorScheme));
});

Privacy and security considerations #

The Chromium team designed and implemented User Preference Media Features Client Hints Headers using the core principles defined in Controlling Access to Powerful Web Platform Features, including user control, transparency, and ergonomics.

The Security Considerations of HTTP Client Hints and the Security Considerations of Client Hint Reliability likewise apply to this proposal.

References #

Acknowledgements #

Many thanks for valuable feedback and advice from Yoav Weiss. Hero image by Tdadamemd on Wikimedia Commons.

Accessing hardware devices on the web

2021-02-12T00:00:00Z

The goal of this guide is to help you pick the best API to communicate with a hardware device (e.g. webcam, microphone, etc.) on the web. By "best" I mean it gives you everything you need with the shortest amount of work. In other words, you know the general use case you want to solve (e.g. accessing video) but you don't know what API to use or wonder if there's another way to achieve it.

One problem that I commonly see web developers fall into is jumping into low-level APIs without learning about the higher-level APIs that are easier to implement and provide a better UX. Therefore, this guide starts by recommending higher-level APIs first, but also mentions lower-level APIs in case you have determined that the higher-level API doesn't meet your needs.

🕹 Receive input events from this device #

Try listening for Keyboard and Pointer events. If this device is a game controller, use the Gamepad API to know which buttons are being pressed and which axes moved.

If none of these options work for you, a low-level API may be the solution. Check out Discover how to communicate with your device to start your journey.

📸 Access audio and video from this device #

Use MediaDevices.getUserMedia() to get live audio and video streams from this device and learn about capturing audio and video. You can also control the camera's pan, tilt, and zoom, and other camera settings such as brightness and contrast, and even take still images. Web Audio can be used to add effects to audio, create audio visualizations, or apply spatial effects (such as panning). Check out how to profile the performance of Web Audio apps in Chrome as well.

If none of these options work for you, a low-level API may be the solution. Check out Discover how to communicate with your device to start your journey.

🖨 Print to this device #

Use window.print() to open a browser dialog that lets the user pick this device as a destination to print the current document.

If this doesn't work for you, a low-level API may be the solution. Check out Discover how to communicate with your device to start your journey.

🔐 Authenticate with this device #

Use WebAuthn to create a strong, attested, and origin-scoped public-key credential with this hardware security device to authenticate users. It supports the use of Bluetooth, NFC, and USB-roaming U2F or FIDO2 authenticators —also known as security keys— as well as a platform authenticator, which lets users authenticate with their fingerprints or screen locks. Check out Build your first WebAuthn app.

If this device is another type of hardware security device (e.g. a cryptocurrency wallet), a low-level API may be the solution. Check out Discover how to communicate with your device to start your journey.

🗄 Access files on this device #

Use the File System Access API to read from and save changes directly to files and folders on the user's device. If not available, use the File API to ask the user to select local files from a browser dialog and then read the contents of those files.

If none of these options work for you, a low-level API may be the solution. Check out Discover how to communicate with your device to start your journey.

🧲 Access sensors on this device #

Use the Generic Sensor API to read raw sensor values from motion sensors (e.g. accelerometer or gyroscope) and environmental sensors (e.g. ambient light, magnetometer). If not available, use the DeviceMotion and DeviceOrientation events to get access to the built-in accelerometer, gyroscope, and compass in mobile devices.

If it doesn't work for you, a low-level API may be the solution. Check out Discover how to communicate with your device to start your journey.

🛰 Access GPS coordinates on this device #

Use the Geolocation API to get the latitude and longitude of the user's current position on this device.

If it doesn't work for you, a low-level API may be the solution. Check out Discover how to communicate with your device to start your journey.

🔋 Check the battery on this device #

Use the Battery API to get host information about the battery charge level and be notified when the battery level or charging status change.

If it doesn't work for you, a low-level API may be the solution. Check out Discover how to communicate with your device to start your journey.

📞 Communicate with this device over the network #

In the local network, use the Remote Playback API to broadcast audio and/or video on a remote playback device (e.g. a smart TV or a wireless speaker) or use the Presentation API to render a web page on a second screen (e.g. a secondary display connected with an HDMI cable or a smart TV connected wirelessly).

If this device exposes a web server, use the Fetch API and/or WebSockets to fetch some data from this device by hitting appropriate endpoints. While TCP and UDP sockets are not available on the web, check out WebTransport to handle interactive, bidirectional, and multiplexed network connections. Note that WebRTC can also be used to communicate data in real-time with other browsers using a peer-to-peer protocol.

🧱 Discover how to communicate with your device #

The decision of what low-level API you should use is determined by the nature of your physical connection to the device. If it is wireless, check out Web NFC for very short-range wireless connections and Web Bluetooth for nearby wireless devices.

With Web NFC, read and write to this device when it's in close proximity to the user's device (usually 5–10 cm, 2–4 inches). Tools like NFC TagInfo by NXP allow you to browse the content of this device for reverse-engineering purposes.
With Web Bluetooth, connect to this device over a Bluetooth Low Energy connection. It should be pretty easy to communicate with when it uses standardized Bluetooth GATT services (such as the battery service) as their behavior is well-documented. If not, at this point, you either have to find some hardware documentation for this device or reverse-engineer it. You can use external tools like nRF Connect for Mobile and built-in browser tools such as the internal page about://bluetooth-internals in Chromium-based browsers for that. Check out Reverse-Engineering a Bluetooth Lightbulb from Uri Shaked. Note that Bluetooth devices may also speak the HID or serial protocols.

If wired, check out these APIs in this specific order:

With WebHID, understanding HID reports and report descriptors through collections is key to your comprehension of this device. This can be challenging without vendor documentation for this device. Tools like Wireshark can help you reverse-engineer it. You can also use the HID Explorer web app for dumping HID devices info into a human-readable format.
With Web Serial, without vendor documentation for this device and what commands this device supports, it's hard but still possible with lucky guessing. Reverse-engineering this device can be done by capturing raw USB traffic with tools like Wireshark. You can also use the Serial Terminal web app to experiment with this device if it uses a human-readable protocol.
With WebUSB, without clear documentation for this device and what USB commands this device supports, it's hard but still possible with lucky guessing. Watch Exploring WebUSB and its exciting potential from Suz Hinton. You can also reverse-engineer this device by capturing raw USB traffic and inspecting USB descriptors with external tools like Wireshark and built-in browser tools such as the internal page about://usb-internals in Chromium-based browsers.

Acknowledgements #

Thanks to Reilly Grant, Thomas Steiner, and Kayce Basques for reviewing this article.

Photo by Darya Tryfanava on Unsplash.

Disable mouse acceleration to provide a better FPS gaming experience

2020-11-26T00:00:00Z

Accelerated movement is an ergonomic feature when using a mouse or trackpad to move the pointer on screen. It allows precise movement by moving slowly while also allowing the pointer to cross the entire screen with a quick short motion. Specifically, for the same physical distance that you move the mouse, the pointer on screen travels further if the distance was traveled faster.

Operating systems enable mouse acceleration by default. For some first-party perspective games, commonly first-person shooters (FPS), raw mouse input data is used to control camera rotation without an acceleration adjustment. The same physical motion, slow or fast, results in the same rotation. This results in a better gaming experience and higher accuracy according to professional gamers.

Pointer motion control in Windows 10 settings.

Starting in Chrome 88, web apps can switch back and forth between accelerated and non-accelerated mouse movement data thanks to the updated Pointer Lock API.

Web-based gaming platforms such as Google Stadia and Nvidia GeForce Now already use these new capabilities to please FPS gamers.

Browser support

Chrome 37, Supported 37
Firefox 50, Supported 50
Edge 13, Supported 13
Safari 10.1, Supported 10.1

Source

Using the API #

Request a pointer lock #

A pointer lock is the canonical term for when a desktop application hides the pointer icon and interprets mouse motion for something else, e.g. looking around in a 3D world.

The movementX and movementY attributes from the mousemove document events tell you how much the mouse pointer moved since the last move event. However, those are not updated when the pointer moves outside of the web page.

document.addEventListener("mousemove", (event) => {
  console.log(`movementX: ${event.movementX} movementY: ${event.movementY}`);
});

Capturing the mouse pointer (or requesting a pointer lock) allows you to not worry about the pointer moving outside anymore. This is especially useful for immersive web games. When the pointer is locked, all mouse events go to the target element of the pointer lock.

Call requestPointerLock() on the target element to request a pointer lock, and listen to pointerlockchange and pointerlockerror events to monitor pointer lock changes.

const myTargetElement = document.body;

// Call this function to request a pointer lock.
function requestPointerLock() {
  myTargetElement.requestPointerLock();
}

document.addEventListener("pointerlockchange", () => {
  if (document.pointerLockElement) {
    console.log(`pointer is locked on ${document.pointerLockElement}`);
  } else {
    console.log("pointer is unlocked");
  }
});

document.addEventListener("pointerlockerror", () => {
  console.log("pointer lock error");
});

Disable mouse acceleration #

Call requestPointerLock() with { unadjustedMovement: true } to disable OS-level adjustment for mouse acceleration, and access raw mouse input. This way, mouse movement data from mousemove events won't include mouse acceleration when the pointer is locked.

Use the new returned promise from requestPointerLock() to know if the request was successful.

function requestPointerLockWithUnadjustedMovement() {
  const promise = myTargetElement.requestPointerLock({
    unadjustedMovement: true,
  });

  if (!promise) {
    console.log("disabling mouse acceleration is not supported");
    return;
  }

  return promise
    .then(() => console.log("pointer is locked"))
    .catch((error) => {
      if (error.name === "NotSupportedError") {
        // Some platforms may not support unadjusted movement.
        // You can request again a regular pointer lock.
        return myTargetElement.requestPointerLock();
      }
    });
}

It is possible to toggle between accelerated and non-accelerated mouse movement data without releasing the pointer lock. Simply request the pointer lock again with the desired option. If that request fails, the original lock will remain intact and the returned promise will reject. No pointer lock events will fire for a failed change request.

Browser support #

The Pointer Lock API is well supported across browsers. However Chromium-based browsers (e.g. Chrome, Edge, etc.) are the only ones to support disabling OS-level adjustment for mouse acceleration as of October 2020. See MDN's Browser compatibility table for updates.

Operating system support #

Disabling OS-level adjustment for mouse acceleration is supported on ChromeOS, macOS Catalina 10.15.1, and Windows. Linux will follow.

Sample #

You can play with the Pointer Lock API by running the sample on Glitch. Be sure to check out the source code.

Helpful links #

Acknowledgements #

Thanks to James Hollyer, Thomas Steiner, Joe Medley, Kayce Basques, and Vincent Scheib for their reviews of this article.

How Chrome handles updates to the web app manifest

2020-10-14T00:00:00Z

When a PWA is installed, the browser uses information from the web app manifest for the app name, the icons the app should use, and the URL that should be opened when the app is launched. But what if you need to update app shortcuts or try a new theme color? When and how are those changes reflected in the browser?

In most cases, changes should be reflected within a day or two of the PWA being launched, after the manifest has been updated.

Updates on desktop Chrome #

When the PWA is launched, or opened in a browser tab, Chrome determines the last time the local manifest was checked for changes. If the manifest hasn't been checked since the browser last started, or it hasn't been checked in the last 24 hours, Chrome will make a network request for the manifest, then compare it against the local copy.

If select properties in the manifest have changed (see list below), Chrome queues the new manifest, and after all windows have been closed, installs it. Once installed, all fields from the new manifest (except name, short_name, and icons) are updated.

Which properties will trigger an update? #

display (see below)
scope
shortcuts
start_url
theme_color
file_handlers

What happens when the `display` field is updated? #

If you update your app's display mode from browser to standalone your existing users will not have their apps open in a window after updating. There are two display settings for a web app, the one from the manifest (that you control) and a window/browser tab setting controlled by the user. The user preference is always respected.

Testing manifest updates #

The chrome://web-app-internals page (available in Chrome 85 or later), includes detailed information about all of the PWAs installed on the device, and can help you understand when the manifest was last updated, how often it's updated, and more.

To manually force Chrome to check for an updated manifest, you can either launch Chrome with the command line flag --disable-manifest-update-throttle or restart Chrome (use about://restart), this resets the timer so that Chrome will check for an updated manifest when the PWA is next launched. Then launch the PWA. After closing the PWA, it should be updated with the new manifest properties.

References #

Updatable Web Manifest Fields

Updates on Chrome for Android #

When the PWA is launched, Chrome determines the last time the local manifest was checked for changes. If the manifest hasn't been checked in the last 24 hours, Chrome will schedule a network request for the manifest, then compare it against the local copy.

If select properties in the manifest have changed (see list below), Chrome queues the new manifest, and after all windows of the PWA have been closed, the device is plugged in, and connected to WiFi, Chrome requests an updated WebAPK from the server. Once updated, all fields from the new manifest are used.

Which properties will trigger an update? #

background_color
display
orientation
scope
shortcuts
start_url
theme_color
web_share_target

If Chrome is unable to get an updated manifest from the server, it may increase the time between checks to 30 days.

Testing manifest updates #

The about://webapks page includes detailed information about all of the PWAs installed on the device, and can tell you when the manifest was last updated, how often it's updated, and more.

To manually schedule an update to the manifest, overriding the timer and local manifest do the following:

Plug in the device and ensure it's connected to WiFi.
Use the Android task manager to shut down the PWA, then use the App panel in Android settings to force stop the PWA.
In Chrome, open about://webapks and click the "Update" button for the PWA. "Update Status" should change to "Pending".
Launch the PWA, and verify it's loaded properly.
Use the Android task manager to shut down the PWA, then use the App panel in Android settings to force stop the PWA.

The PWA usually updates within a few minutes, once the update has completed, "Update Status" should change to "Successful"

References #

UpdateReason enum for Chrome on Android

Control camera pan, tilt, and zoom

2020-10-05T00:00:00Z

Room-scale video conferencing solutions deploy cameras with pan, tilt, and zoom (PTZ) capabilities so that software can point the camera at meeting participants. Starting in Chrome 87, the pan, tilt, and zoom features on cameras are available to websites using media track constraints in MediaDevices.getUserMedia() and MediaStreamTrack.applyConstraints().

Using the API #

Feature detection #

Feature detection for hardware is different from what you're probably used to. The presence of "pan", "tilt", and "zoom" constraint names in navigator.mediaDevices.getSupportedConstraints() tells you that the browser supports the API to control camera PTZ, but not whether the camera hardware supports it. As of Chrome 87, controlling camera PTZ is supported on desktop, while Android still supports zoom only.

const supports = navigator.mediaDevices.getSupportedConstraints();
if (supports.pan && supports.tilt && supports.zoom) {
  // Browser supports camera PTZ.
}

Request camera PTZ access #

A website is allowed to control camera PTZ only if the user has explicitly granted the camera with PTZ permission through a prompt.

To request camera PTZ access, call navigator.mediaDevices.getUserMedia() with the PTZ constraints as shown below. This will prompt the user to grant both regular camera and camera with PTZ permissions.

Camera PTZ user prompt.

The returned promise will resolve with a MediaStream object used to show the camera video stream to the user. If the camera does not support PTZ, the user will get a regular camera prompt.

try {
  // User is prompted to grant both camera and PTZ access in a single call.
  // If camera doesn't support PTZ, it falls back to a regular camera prompt.
  const stream = await navigator.mediaDevices.getUserMedia({
    // Website asks to control camera PTZ as well without altering the
    // current pan, tilt, and zoom settings.
    video: { pan: true, tilt: true, zoom: true }
  });

  // Show camera video stream to user.
  document.querySelector("video").srcObject = stream;
} catch (error) {
  // User denies prompt or matching media is not available.
  console.log(error);
}

A previously-granted camera permission, specifically one without PTZ access, does not automatically gain PTZ access if it becomes available. This is true even when the camera itself supports PTZ. The permission must be requested again. Fortunately, you can use the Permissions API to query and monitor the status of PTZ permission.

try {
  const panTiltZoomPermissionStatus = await navigator.permissions.query({
    name: "camera",
    panTiltZoom: true
  });

  if (panTiltZoomPermissionStatus.state == "granted") {
    // User has granted access to the website to control camera PTZ.
  }

  panTiltZoomPermissionStatus.addEventListener("change", () => {
    // User has changed PTZ permission status.
  });
} catch (error) {
  console.log(error);
}

To know whether a Chromium-based browser supports PTZ for a camera, go to the internal about://media-internals page and check out the "Pan-Tilt-Zoom" column in the "Video Capture" tab; "pan tilt" and "zoom" respectively mean the camera supports the "PanTilt (Absolute)" and "Zoom (Absolute)" UVC controls. The "PanTilt (Relative)" and "Zoom (Relative)" UVC controls are not supported in Chromium-based browsers.

Internal page to debug PTZ camera support.

Control camera PTZ #

Manipulate camera PTZ capabilities and settings using the preview MediaStreamTrack from the stream object obtained earlier. MediaStreamTrack.getCapabilities() returns a dictionary with the supported capabilities and the ranges or allowed values. Correspondingly, MediaStreamTrack.getSettings() returns the current settings.

Pan, tilt, and zoom capabilities and settings are available only if supported by the camera and the user has granted PTZ permission to the camera.

Controlling Camera PTZ.

Call videoTrack.applyConstraints() with the appropriate PTZ advanced constraints to control camera pan, tilt, and zoom as shown in the example below. The returned promise will resolve if successful. Otherwise it will reject if either:

the camera with PTZ permission is not granted.
the camera hardware does not support the PTZ constraint.
the page is not visible to the user. Use the Page Visibility API to detect page visibility changes.

// Get video track capabilities and settings.
const [videoTrack] = stream.getVideoTracks();
const capabilities = videoTrack.getCapabilities();
const settings = videoTrack.getSettings();

// Let the user control the camera pan motion if the camera supports it
// and PTZ access is granted.
if ("pan" in settings) {
  const input = document.querySelector("input[type=range]");
  input.min = capabilities.pan.min;
  input.max = capabilities.pan.max;
  input.step = capabilities.pan.step;
  input.value = settings.pan;

  input.addEventListener("input", async () => {
    await videoTrack.applyConstraints({ advanced: [{ pan: input.value }] });
  });
}

if ("tilt" in settings) {
  // similar for tilt...
}
if ("zoom" in settings) {
  // similar for zoom...
}

It is also possible to configure camera pan, tilt, and zoom by calling navigator.mediaDevices.getUserMedia() with some camera PTZ ideal constraint values. This is handy when camera PTZ capabilities are known in advance. Note that mandatory constraints (min, max, exact) are not allowed here.

const stream = await navigator.mediaDevices.getUserMedia({
  // Website asks to reset known camera pan.
  video: { pan: 0, deviceId: { exact: "myCameraDeviceId" } }
});

Playground #

You can play with the API by running the demo on Glitch. Be sure to check out the source code.

Tip: If you don't have a camera that supports PTZ, you can run Chrome with the switch --use-fake-device-for-media-stream to simulate one on your machine. Enjoy!

Security Considerations #

The spec authors have designed and implemented this API using the core including user control, transparency, and ergonomics. The ability to use this API is primarily gated by the same permission model as the Media Capture and Streams API. In response to a user prompt, the website is allowed to control camera PTZ only when the page is visible to the user.

Browser compatibility #

MediaStream API #

Browser support

Chrome 55, Supported 55
Firefox 15, Supported 15
Edge 12, Supported 12
Safari 11, Supported 11

Source

Permissions API #

Browser support

Chrome 43, Supported 43
Firefox 46, Supported 46
Edge 79, Supported 79
Safari 16, Supported 16

Source

Page Visibility API #

Browser support

Chrome 33, Supported 33
Firefox 18, Supported 18
Edge 12, Supported 12
Safari 7, Supported 7

Source

`MediaDevices.getUserMedia()` #

Browser support

Chrome 53, Supported 53
Firefox 36, Supported 36
Edge 12, Supported 12
Safari 11, Supported 11

Source

`MediaDevices.getSupportedConstraints()` #

Browser support

Chrome 53, Supported 53
Firefox 44, Supported 44
Edge 12, Supported 12
Safari 11, Supported 11

Source

`MediaStreamTrack.applyConstraints()` #

Browser support

Chrome 59, Supported 59
Firefox 43, Supported 43
Edge 12, Supported 12
Safari 11, Supported 11

Source

`MediaStreamTrack.getCapabilities()` #

Browser support

Chrome 59, Supported 59
Firefox, Not supported
Edge 12, Supported 12
Safari 11, Supported 11

Source

`MediaStreamTrack.getSettings()` #

Browser support

Chrome 59, Supported 59
Firefox 50, Supported 50
Edge 12, Supported 12
Safari 11, Supported 11

Source

Helpful links #

Acknowledgements #

This article was reviewed by Joe Medley and Thomas Steiner. Thanks to Rijubrata Bhaumik and Eero Häkkinen at Intel for their work on the spec and the implementation. Hero image by Christina @ wocintechchat.com on Unsplash.

Get things done quickly with app shortcuts

2020-05-20T00:00:00Z

To improve users' productivity and facilitate reengagement with key tasks, the web platform now supports app shortcuts. They allow web developers to provide quick access to a handful of common actions that users need frequently.

This article will teach you how to define app shortcuts. Additionally, you'll learn some associated best practices.

About app shortcuts #

The app shortcuts menu is invoked by right-clicking the app icon in the taskbar (Windows) or dock (macOS) on the user's desktop, or touch & holding the app's launcher icon on Android.

App shortcuts menu opened on Android

App shortcuts menu opened on Windows

The app shortcuts menu is shown only for Progressive Web Apps that are installed on the user's desktop or mobile device. Check out Installation in our "Learn PWA" module to learn about installability requirements.

Each app shortcut expresses a user intent, each of which is associated with a URL within the scope of your web app. The URL is opened when a user activate the app shortcut. Examples of app shortcuts include the following:

Top-level navigation items (e.g., home, timeline, recent orders)
Search
Data entry tasks (e.g., compose an email or tweet, add a receipt)
Activities (e.g., start a chat with the most popular contacts)

Define app shortcuts in the web app manifest #

App shortcuts are optionally defined in the web app manifest, a JSON file that tells the browser about your web app and how it should behave when installed on the user's desktop or mobile device. More specifically, they are declared in the shortcuts array member. Below is an example of a potential web app manifest.

{
  "name": "Player FM",
  "start_url": "https://player.fm?utm_source=homescreen",
  …
  "shortcuts": [
    {
      "name": "Open Play Later",
      "short_name": "Play Later",
      "description": "View the list of podcasts you saved for later",
      "url": "/play-later?utm_source=homescreen",
      "icons": [{ "src": "/icons/play-later.png", "sizes": "192x192" }]
    },
    {
      "name": "View Subscriptions",
      "short_name": "Subscriptions",
      "description": "View the list of podcasts you listen to",
      "url": "/subscriptions?utm_source=homescreen",
      "icons": [{ "src": "/icons/subscriptions.png", "sizes": "192x192" }]
    }
  ]
}

Each member of the shortcuts array is a dictionary that contains at least a name and a url. Other members are optional.

name

The human-readable label for the app shortcut when displayed to the user.

short_name (optional)

The human-readable label used where space is limited. It is recommended that you provide it, even though it's optional.

description (optional)

The human-readable purpose for the app shortcut. It is not used at the time of writing but may be exposed to assistive technology in the future.

url

The URL opened when a user activates the app shortcut. This URL must exist within the scope of the web app manifest. If it is a relative URL, the base URL will be the URL of the web app manifest.

icons (optional)

An array of image resource objects. Each object must include the src and a sizes property. Unlike web app manifest icons, the type of image is optional. SVG files are not supported at the time of writing. Use PNG instead.

If you want pixel-perfect icons, provide them in increments of 48dp (i.e. 36x36, 48x48, 72x72, 96x96, 144x144, 192x192 pixel icons). Otherwise, it is recommended that you use a single 192x192 pixel icon.

As a quality measure, icons must be at least half of the device's ideal size on Android, which is 48dp. For example, to display on an xxhdpi screen, the icon must be at least 72 by 72 pixels. (This is derived from the formula for converting dp units for pixel units.)

Test your app shortcuts #

To verify that your app shortcuts are set up correctly, use the Manifest pane in the Application panel of DevTools.

App shortcuts shown in DevTools

This pane provides a human-readable version of many of your manifest's properties, including app shortcuts. It makes it easy to verify that all of the shortcut icons, if provided, are loading properly.

App shortcuts may not be available right away to all users because Progressive Web App updates are capped to once a day. Find out more about how Chrome handles updates to the web app manifest.

Best practices #

Order app shortcuts by priority #

Shortcuts are displayed in the order by which you define them in the manifest. You are encouraged to order app shortcuts by priority because the limit on the number of app shortcuts displayed varies byß platform. Chrome and Edge on Windows for instance limit the number of app shortcuts to 10 while Chrome for Android only display 3. Before Chrome 92 for Android 7, 4 were allowed. Chrome 92 added a shortcut to the site settings, taking one of the available shortcut slots for the app.

Use distinct app shortcut names #

You should not rely on icons to differentiate app shortcuts as they may not always be visible. For instance, macOS doesn't support icons in the dock shortcuts menu. Use distinct names for each app shortcut.

Measure app shortcuts usage #

You should annotate app shortcuts url entries like you would do with start_url for analytics purposes (e.g. url: "/my-shortcut?utm_source=homescreen").

Browser support #

App shortcuts are available on the platforms and versions listed below.

Browser support

Chrome 96, Supported 96
Firefox, Not supported
Edge 96, Supported 96
Safari, Not supported

Source

App shortcuts menu opened on ChromeOS

Trusted Web Activity support #

Bubblewrap, the recommended tool to build Android apps that use Trusted Web Activity, reads app shortcuts from the web app manifest and automatically generates the corresponding configuration for the Android app. Note that icons for app shortcuts are required and must be at least 96 by 96 pixels in Bubblewrap.

PWABuilder, a great tool to easily turn a Progressive Web App into a Trusted Web Activity, supports app shortcuts with some caveats.

For developers integrating Trusted Web Activity manually into their Android application, Android app shortcuts can be used to implement the same behaviors.

Sample #

Check out the app shortcuts sample and its source.

Helpful links #

MDN
Spec
App shortcuts sample | App shortcuts sample source
Blink Component: UI>Browser>WebAppInstalls

Customize media notifications and playback controls with the Media Session API

2020-03-06T00:00:00Z

To let users know what's currently playing in their browser and control it without returning to the page that launched it, the Media Session API has been introduced. It allows web developers to customize this experience through metadata in custom media notifications, media events such as playing, pausing, seeking, track changing, and video conferencing events such as mute/unmute microphone, turnon/turnoff camera, and hang up. These customizations are available in several contexts including desktop media hubs, media notifications on mobile, and even on wearable devices. I'll describe these customizations in this article.

Media hub on desktop, media notification on mobile, and a wearable device.

About the Media Session API #

The Media session API provides several benefits and capabilities:

Hardware media keys are supported.
Media notifications are customized on mobile, desktop, and paired wearable device.
The media hub is available on desktop.
Lock screen media controls are available on ChromeOS and mobile.
Picture-in-Picture window controls are available for audio playback, video conferencing, and presenting slides.
Assistant integration on mobile is available.

Browser support

Chrome 73, Supported 73
Firefox 82, Supported 82
Edge 79, Supported 79
Safari 15, Supported 15

Source

A few examples will illustrate some of these points.

Example 1: If users press the "next track" media key of their keyboard, web developers can handle this user action whether the browser is in the foreground or the background.

Example 2: If users listen to a podcast on the web while their device screen is locked, they can still hit the "seek backward" icon from the lock screen media controls so that web developers move playback time backward by few seconds.

Example 3: If users have tabs playing audio, they can easily stop playback from the media hub on desktop so that web developers have a chance to clear their state.

Example 4: If users are on a video call, they can press the "toggle microphone" control in the Picture-in-Picture window to stop the website from receiving microphone data.

This is all done through two different interfaces: The MediaSession interface and the MediaMetadata interface. The first lets users control whatever's playing. The second is how you tell MediaSession what needs to be controlled.

To illustrate, the image below shows how these interfaces relate to specific media controls, in this case a media notification on mobile.

Anatomy of a media notification on mobile.

Let users know what's playing #

When a website is playing audio or video, users automatically get media notifications either in the notification tray on mobile, or the media hub on desktop. The browser does its best to show appropriate information by using the document's title and the largest icon image it can find. With the Media Session API, it's possible to customize the media notification with some richer media metadata such as the title, artist name, album name, and artwork as shown below.

Chrome requests "full" audio focus to show media notifications only when the media duration is at least 5 seconds. This ensures that incidental sounds such as dings don't show notifications.

// After media (video or audio) starts playing
await document.querySelector("video").play();

if ("mediaSession" in navigator) {
  navigator.mediaSession.metadata = new MediaMetadata({
    title: 'Never Gonna Give You Up',
    artist: 'Rick Astley',
    album: 'Whenever You Need Somebody',
    artwork: [
      { src: 'https://via.placeholder.com/96',   sizes: '96x96',   type: 'image/png' },
      { src: 'https://via.placeholder.com/128', sizes: '128x128', type: 'image/png' },
      { src: 'https://via.placeholder.com/192', sizes: '192x192', type: 'image/png' },
      { src: 'https://via.placeholder.com/256', sizes: '256x256', type: 'image/png' },
      { src: 'https://via.placeholder.com/384', sizes: '384x384', type: 'image/png' },
      { src: 'https://via.placeholder.com/512', sizes: '512x512', type: 'image/png' },
    ]
  });

  // TODO: Update playback state.
}

When playback ends, there is no need to "release" the media session as the notification will automatically disappear. Keep in mind that navigator.mediaSession.metadata will be used when the next playback starts though. This is why it's important to update it when the media playback source changes to make sure relevant information is shown in the media notification.

There are a few things to note about the media metadata.

Notification artwork array supports blob URLs and data URLs.
If no artwork is defined and there is an icon image (specified using <link rel=icon>) at a desirable size, media notifications will use it.
Notification artwork target size in Chrome for Android is 512x512. For low-end devices, it is 256x256.
The title attribute of the media HTML element is used in the "Now playing" macOS widget.
If the media resource is embedded (for example in a iframe), Media Session API information must be set from the embedded context. See snippet below.

<iframe id="iframe">
  <video>...</video>
</iframe>
<script>
  iframe.contentWindow.navigator.mediaSession.metadata = new MediaMetadata({
    title: 'Never Gonna Give You Up',
    ...
  });
</script>

Let users control what's playing #

A media session action is an action (for example "play" or "pause") that a website can handle for users when they interact with the current media playback. Actions are analogous to and work much the same as events. Like events, actions are implemented by setting handlers on an appropriate object, an instance of MediaSession, in this case. Some actions are triggered when users press buttons from a headset, another remote device, a keyboard, or interact with a media notification.

Customized media notification in Windows 10.

Because some media session actions may not be supported, it is recommended to use a try…catch block when setting them.

const actionHandlers = [
  ['play',          () => { /* ... */ }],
  ['pause',         () => { /* ... */ }],
  ['previoustrack', () => { /* ... */ }],
  ['nexttrack',     () => { /* ... */ }],
  ['stop',          () => { /* ... */ }],
  ['seekbackward',  (details) => { /* ... */ }],
  ['seekforward',   (details) => { /* ... */ }],
  ['seekto',        (details) => { /* ... */ }],
  /* Video conferencing actions */
  ['togglemicrophone', () => { /* ... */ }],
  ['togglecamera',     () => { /* ... */ }],
  ['hangup',           () => { /* ... */ }],
  /* Presenting slides actions */
  ['previousslide', () => { /* ... */ }],
  ['nextslide',     () => { /* ... */ }],
];

for (const [action, handler] of actionHandlers) {
  try {
    navigator.mediaSession.setActionHandler(action, handler);
  } catch (error) {
    console.log(`The media session action "${action}" is not supported yet.`);
  }
}

Unsetting a media session action handler is as easy as setting it to null.

try {
  // Unset the "nexttrack" action handler at the end of a playlist.
  navigator.mediaSession.setActionHandler('nexttrack', null);
} catch (error) {
  console.log(`The media session action "nexttrack" is not supported yet.`);
}

Once set, media session action handlers will persist through media playbacks. This is similar to the event listener pattern except that handling an event means that the browser stops doing any default behavior and uses this as a signal that the website supports the media action. Hence, media action controls won't be shown unless the proper action handler is set.

Now Playing widget in macOS Big Sur.

Play / pause #

The "play" action indicates that the user wants to resume the media playback while "pause" indicates a desire to temporarily halt it.

The "play/pause" icon is always shown in a media notification and the related media events are handled automatically by the browser. To override their default behavior, handle "play" and "pause" media actions as shown below.

The browser may consider a website to not be playing media when seeking or loading for instance. In this case, override this behavior by setting navigator.mediaSession.playbackState to "playing" or "paused" to make sure the website UI stays in sync with media notification controls.

const video = document.querySelector('video');

navigator.mediaSession.setActionHandler('play', async () => {
  // Resume playback
  await video.play();
});

navigator.mediaSession.setActionHandler('pause', () => {
  // Pause active playback
  video.pause();
});

video.addEventListener('play', () => {
  navigator.mediaSession.playbackState = 'playing';
});

video.addEventListener('pause', () => {
  navigator.mediaSession.playbackState = 'paused';
});

Previous track #

The "previoustrack" action indicates that the user either wants to start the current media playback from the beginning if the media playback has a notion of beginning, or move to the previous item in the playlist if the media playback has a notion of a playlist.

navigator.mediaSession.setActionHandler('previoustrack', () => {
  // Play previous track.
});

Next track #

The "nexttrack" action indicates that the user wants to move media playback to the next item in the playlist if the media playback has a notion of a playlist.

navigator.mediaSession.setActionHandler('nexttrack', () => {
  // Play next track.
});

Stop #

The "stop" action indicates that the user wants to stop the media playback and clear the state if appropriate.

navigator.mediaSession.setActionHandler('stop', () => {
  // Stop playback and clear state if appropriate.
});

Seek backward / forward #

The "seekbackward" action indicates that the user wants to move the media playback time backward by a short period while "seekforward" indicates a desire to move the media playback time forward by a short period. In both cases, a short period means a few seconds.

The seekOffset value provided in the action handler is the time in seconds to move the media playback time by. If it is not provided (for example undefined), then you should use a sensible time (for example 10-30 seconds).

const video = document.querySelector('video');
const defaultSkipTime = 10; /* Time to skip in seconds by default */

navigator.mediaSession.setActionHandler('seekbackward', (details) => {
  const skipTime = details.seekOffset || defaultSkipTime;
  video.currentTime = Math.max(video.currentTime - skipTime, 0);
  // TODO: Update playback state.
});

navigator.mediaSession.setActionHandler('seekforward', (details) => {
  const skipTime = details.seekOffset || defaultSkipTime;
  video.currentTime = Math.min(video.currentTime + skipTime, video.duration);
  // TODO: Update playback state.
});

Seek to a specific time #

The "seekto" action indicates that the user wants to move the media playback time to a specific time.

The seekTime value provided in the action handler is the time in seconds to move the media playback time to.

The fastSeek boolean provided in the action handler is true if the action is being called multiple times as part of a sequence and this is not the last call in that sequence.

const video = document.querySelector('video');

navigator.mediaSession.setActionHandler('seekto', (details) => {
  if (details.fastSeek && 'fastSeek' in video) {
    // Only use fast seek if supported.
    video.fastSeek(details.seekTime);
    return;
  }
  video.currentTime = details.seekTime;
  // TODO: Update playback state.
});

Set playback position #

Accurately displaying the media playback position in a notification is as simple as setting the position state at an appropriate time as shown below. The position state is a combination of the media playback rate, duration, and current time.

Lock screen media controls in ChromeOS.

The duration must be provided and positive. The position must be positive and less than the duration. The playback rate must be greater than 0.

const video = document.querySelector('video');

function updatePositionState() {
  if ('setPositionState' in navigator.mediaSession) {
    navigator.mediaSession.setPositionState({
      duration: video.duration,
      playbackRate: video.playbackRate,
      position: video.currentTime,
    });
  }
}

// When video starts playing, update duration.
await video.play();
updatePositionState();

// When user wants to seek backward, update position.
navigator.mediaSession.setActionHandler('seekbackward', (details) => {
  /* ... */
  updatePositionState();
});

// When user wants to seek forward, update position.
navigator.mediaSession.setActionHandler('seekforward', (details) => {
  /* ... */
  updatePositionState();
});

// When user wants to seek to a specific time, update position.
navigator.mediaSession.setActionHandler('seekto', (details) => {
  /* ... */
  updatePositionState();
});

// When video playback rate changes, update position state.
video.addEventListener('ratechange', (event) => {
  updatePositionState();
});

Resetting the position state is as easy as setting it to null.

// Reset position state when media is reset.
navigator.mediaSession.setPositionState(null);

Video conferencing actions #

When the user puts their video call into a Picture-in-Picture window, the browser may display controls for the microphone and camera, and for hanging up. When the user clicks those, the website handles them through the video conferencing actions below. For an example, see the Video Conferencing sample.

Video conferencing controls in a Picture-in-Picture window.

Toggle microphone #

The "togglemicrophone" action indicates that the user wants to mute or unmute the microphone. The setMicrophoneActive(isActive) method tells the browser whether the website currently considers the microphone to be active.

let isMicrophoneActive = false;

navigator.mediaSession.setActionHandler('togglemicrophone', () => {
  if (isMicrophoneActive) {
    // Mute the microphone.
  } else {
    // Unmute the microphone.
  }
  isMicrophoneActive = !isMicrophoneActive;
  navigator.mediaSession.setMicrophoneActive(isMicrophoneActive);
});

Toggle camera #

The "togglecamera" action indicates that the user wants to turn the active camera on or off. The setCameraActive(isActive) method indicates whether the browser considers the website to be active.

let isCameraActive = false;

navigator.mediaSession.setActionHandler('togglecamera', () => {
  if (isCameraActive) {
    // Disable the camera.
  } else {
    // Enable the camera.
  }
  isCameraActive = !isCameraActive;
  navigator.mediaSession.setCameraActive(isCameraActive);
});

Hang up #

The "hangup" action indicates that the user wants to end a call.

navigator.mediaSession.setActionHandler('hangup', () => {
  // End the call.
});

Presenting slides actions #

When the user puts their slides presentation into a Picture-in-Picture window, the browser may display controls for navigating through slides. When the user clicks those, the website handles them through the Media Session API. For an example, see the Presenting Slides sample.

Previous slide #

The "previousslide" action indicates that the user wants to go back to the previous slide when presenting slides.

navigator.mediaSession.setActionHandler('previousslide', () => {
  // Show previous slide.
});

Browser support

Chrome 111, Supported 111
Firefox, Not supported
Edge 111, Supported 111
Safari, Not supported

Next slide #

The "nextslide" action indicates that the user wants to go to the next slide when presenting slides.

navigator.mediaSession.setActionHandler('nextslide', () => {
  // Show next slide.
});

Browser support

Chrome 111, Supported 111
Firefox, Not supported
Edge 111, Supported 111
Safari, Not supported

Samples #

Check out some Media Session samples featuring Blender Foundation and Jan Morgenstern's work.

A screencast illustrating the Media Session API.

Resources #

Media Session Spec: wicg.github.io/mediasession
Spec Issues: github.com/WICG/mediasession/issues
Chrome Bugs: crbug.com

How YouTube improved video performance with the Media Capabilities API

2019-06-12T00:00:00Z

In an experiment with the Media Capabilities API, YouTube saw a 7.1% increase in MTBR with only a 0.4% decrease in average resolution of videos served.

The Problem #

Typically, media sites have several variants of each video that they can present to users, encoded in different frame rates, resolutions, and codecs. Until recently, web developers had to rely solely on isTypeSupported() or canPlayType() to determine whether each variant could be played in an individual user's browser. While this told the developer whether media could be played at all, it didn't provide an indication of playback quality, such as whether there would be frame drops or device battery drain. Without this information, developers either had to create their own heuristics or just assume that if a device could play a codec/resolution combination, it could do so smoothly and with power efficiency. For users with less capable devices, this often led to a poor experience.

The Solution #

The Media Capabilities API allows websites to get more information about the client's video decode performance and make an informed decision about which codec and resolution to deliver to the user. Specifically, the API provides the developer with an estimate of the smoothness and power efficiency of a particular codec and resolution combination. This allows the developer to avoid scenarios where the client is likely to have a poor playback experience.

In Chrome, the Media Capabilities API uses metrics from previous playbacks to predict whether future playbacks in the same codec and at the same resolution will be smoothly decoded.

YouTube Case Study #

YouTube used the Media Capabilities API to prevent their adaptive bitrate algorithm from automatically selecting resolutions that a device could not play smoothly.

Users who were part of the experimental group collectively saw less frequent rebuffers (the mean time between rebuffers, or MTBR, increased by 7.1%) while the average resolution, measured by video height, served to the aggregate group only declined by 0.4%. The substantial increase in the MTBR with the small corresponding reduction in average resolution indicates that this change significantly improved quality for a small subset of users who previously had a poor experience.

Implementing Media Capabilities API on your site #

Check out the official sample to see how the Decoding Info API works.

Add a web app manifest

2018-11-05T00:00:00Z

The web app manifest is a JSON file that tells the browser about your Progressive Web App and how it should behave when installed on the user's desktop or mobile device. A typical manifest file includes the app name, the icons the app should use, and the URL that should be opened when the app is launched, among other things.

Browser support

Chrome 39, Supported 39
Firefox, Not supported
Edge 79, Supported 79
Safari, Not supported

Source

Create the manifest file #

The manifest file can have any name, but is commonly named manifest.json and served from the root (your website's top-level directory). The specification suggests the extension should be .webmanifest, but browsers also support .json extensions, which may be easier for developers to understand.

A typical manifest looks something like this:

{
  "short_name": "Weather",
  "name": "Weather: Do I need an umbrella?",
  "icons": [
    {
      "src": "/images/icons-vector.svg",
      "type": "image/svg+xml",
      "sizes": "512x512"
    },
    {
      "src": "/images/icons-192.png",
      "type": "image/png",
      "sizes": "192x192"
    },
    {
      "src": "/images/icons-512.png",
      "type": "image/png",
      "sizes": "512x512"
    }
  ],
  "id": "/?source=pwa",
  "start_url": "/?source=pwa",
  "background_color": "#3367D6",
  "display": "standalone",
  "scope": "/",
  "theme_color": "#3367D6",
  "shortcuts": [
    {
      "name": "How's weather today?",
      "short_name": "Today",
      "description": "View weather information for today",
      "url": "/today?source=pwa",
      "icons": [{ "src": "/images/today.png", "sizes": "192x192" }]
    },
    {
      "name": "How's weather tomorrow?",
      "short_name": "Tomorrow",
      "description": "View weather information for tomorrow",
      "url": "/tomorrow?source=pwa",
      "icons": [{ "src": "/images/tomorrow.png", "sizes": "192x192" }]
    }
  ],
  "description": "Weather forecast information",
  "screenshots": [
    {
      "src": "/images/screenshot1.png",
      "type": "image/png",
      "sizes": "540x720",
      "form_factor": "narrow"
    },
    {
      "src": "/images/screenshot2.jpg",
      "type": "image/jpg",
      "sizes": "720x540",
      "form_factor": "wide"
    }
  ]
}

Key manifest properties #

`short_name` and/or `name` #

You must provide at least the short_name or name property. If both are provided, short_name is used on the user's home screen, launcher, or other places where space may be limited. name is used when the app is installed.

`icons` #

When a user installs your PWA, you can define a set of icons for the browser to use on the home screen, app launcher, task switcher, splash screen, and so on.

The icons property is an array of image objects. Each object must include the src, a sizes property, and the type of image. To use maskable icons, sometimes referred to as adaptive icons on Android, you'll also need to add "purpose": "any maskable" to the icon property.

For Chromium, you must provide at least a 192x192 pixel icon, and a 512x512 pixel icon. If only those two icon sizes are provided, Chrome automatically scales the icons to fit the device. If you'd prefer to scale your own icons, and adjust them for pixel-perfection, provide icons in increments of 48dp.

`id` #

The id property allows you to explicitly define the identifier used for your application. Adding the id property to the manifest removes the dependency on the start_url or the location of the manifest, and makes it possible for them to be updated in the future. For more information, see Uniquely identifying PWAs with the web app manifest id property.

`start_url` #

The start_url is required and tells the browser where your application should start when it is launched, and prevents the app from starting on whatever page the user was on when they added your app to their home screen.

Your start_url should direct the user straight into your app, rather than a product landing page. Think about what the user will want to do once they open your app, and place them there.

`background_color` #

The background_color property is used on the splash screen when the application is first launched on mobile.

`display` #

You can customize what browser UI is shown when your app is launched. For example, you can hide the address bar and browser user interface elements. Games can even be made to launch full screen. The display property takes one of the following values:

Property	Use
`fullscreen`	Opens the web application without any browser UI and takes up the entirety of the available display area.
`standalone`	Opens the web app to look and feel like a standalone app. The app runs in its own window, separate from the browser, and hides standard browser UI elements such as the URL bar.
`minimal-ui`	This mode is similar to `standalone`, but provides the user a minimal set of UI elements for controlling navigation (such as back and reload).
`browser`	A standard browser experience.

`display_override` #

Web apps can choose how they are displayed by setting a display mode in their manifest as explained above. Browsers are not required to support all display modes, but they are required to support the spec-defined fallback chain ("fullscreen" → "standalone" → "minimal-ui" → "browser"). If they don't support a given mode, they fall back to the next display mode in the chain. This inflexible behavior can be problematic in rare cases. For example, a developer cannot request "minimal-ui" without being forced back into the "browser" display mode when "minimal-ui" is not supported. Another problem is that the current behavior makes it impossible to introduce new display modes in a backward compatible way, since explorations like tabbed application mode don't have a natural place in the fallback chain.

These problems are solved by the display_override property, which the browser considers before the display property. Its value is a sequence of strings that are considered in the listed order, and the first supported display mode is applied. If none are supported, the browser falls back to evaluating the display field.

Consider the example below. (The details of "window-control-overlay" are out-of-scope for this article.)

{
  "display_override": ["window-control-overlay", "minimal-ui"],
  "display": "standalone",
}

As stated, the browser will look at display_override first.

"window-control-overlay"
"minimal-ui"

If neither option is available, it falls back to display. If "standalone" is not available, it resumes spec-defined fallabck chain from that point.

"standalone"
"minimal-ui"
"browser"

`scope` #

The scope defines the set of URLs that the browser considers to be within your app, and is used to decide when the user has left the app. The scope controls the URL structure that encompasses all the entry and exit points in your web app. Your start_url must reside within the scope.

A few other notes on scope:

If you don't include a scope in your manifest, then the default implied scope is the directory that your web app manifest is served from.
The scope attribute can be a relative path (../), or any higher level path (/) which would allow for an increase in coverage of navigations in your web app.
The start_url must be in the scope.
The start_url is relative to the path defined in the scope attribute.
A start_url starting with / will always be the root of the origin.

`theme_color` #

The theme_color sets the color of the tool bar, and may be reflected in the app's preview in task switchers. The theme_color should match the meta theme color specified in your document head.

An example of a PWA window with custom theme_color.

As of Chromium 93 and Safari 15, you can adjust this color in a media query with the media attribute of the meta theme color element. The first one that matches will be picked. For example, you could have one color for light mode and another one for dark mode. At the time of writing, you can't define those in your manifest. See w3c/manifest#975 GitHub issue.

<meta name="theme-color" media="(prefers-color-scheme: light)" content="white">
<meta name="theme-color" media="(prefers-color-scheme: dark)"  content="black">

`shortcuts` #

The shortcuts property is an array of app shortcut objects whose goal is to provide quick access to key tasks within your app. Each member is a dictionary that contains at least a name and a url.

`description` #

The description property describes the purpose of your app.

`screenshots` #

The screenshots property is an array of image objects representing your app in common usage scenarios. Each object must include the src, a sizes property, and the type of image. The form_factor property is optional. You can set it either to "wide" for screenshots applicable to wide screens only or "narrow" for narrow screenshots. You should only use it when the layout varies by screen size.

In Chrome, the image must respond to certain criteria:

Width and height must be at least 320px and at most 3840px.
The maximum dimension can't be more than 2.3 times as long as the minimum dimension.
All screenshots matching the appropriate form factor must have the same aspect ratio.

Richer installation UI on desktop and mobile.

Add the web app manifest to your pages #

After creating the manifest, add a <link> tag to all the pages of your Progressive Web App. For example:

<link rel="manifest" href="/manifest.json">

Test your manifest #

To verify your manifest is setup correctly, use the Manifest pane in the Application panel of Chrome DevTools.

Test your manifest in DevTools.

This pane provides a human-readable version of many of your manifest's properties, and makes it easy to verify that all of the images are loading properly.

Splash screens on mobile #

When your app first launches on mobile, it can take a moment for the browser to spin up, and the initial content to begin rendering. Instead of showing a white screen that may look to the user like the app is stalled, the browser will show a splash screen until the first paint.

Chrome automatically creates the splash screen from the manifest properties, specifically:

name
background_color
icons

The background_color should be the same color as the load page, to provide a smooth transition from the splash screen to your app.

Chrome will choose the icon that closely matches the device resolution for the device. Providing 192px and 512px icons is sufficient for most cases, but you can provide additional icons for pixel perfection.

Fast playback with audio and video preload

2017-08-17T00:00:00Z

Faster playback start means more people watching your video or listening to your audio. That's a known fact. In this article I'll explore techniques you can use to accelerate your audio and video playback by actively preloading resources depending on your use case.

Credits: copyright Blender Foundation | www.blender.org .

I'll describe three methods of preloading media files, starting with their pros and cons.

	It's great...	But...
Video preload attribute	Simple to use for a unique file hosted on a web server.	Browsers may completely ignore the attribute.
		Resource fetching starts when the HTML document has been completely loaded and parsed.
		Media Source Extensions (MSE) ignore the `preload` attribute on media elements because the app is responsible for providing media to MSE.
Link preload	Forces the browser to make a request for a video resource without blocking the document's `onload` event.	HTTP Range requests are not compatible.
Link preload	Compatible with MSE and file segments.	Should be used only for small media files (<5 MB) when fetching full resources.
Manual buffering	Full control	Complex error handling is the website's responsibility.

Video preload attribute #

If the video source is a unique file hosted on a web server, you may want to use the video preload attribute to provide a hint to the browser as to how much information or content to preload. This means Media Source Extensions (MSE) is not compatible with preload.

Resource fetching will start only when the initial HTML document has been completely loaded and parsed (e.g. the DOMContentLoaded event has fired) while the very different load event will be fired when resource has actually been fetched.

Setting the preload attribute to metadata indicates that the user is not expected to need the video, but that fetching its metadata (dimensions, track list, duration, and so on) is desirable. Note that starting in Chrome 64, the default value for preload is metadata. (It was auto previously).

<video id="video" preload="metadata" src="file.mp4" controls></video>

<script>
  video.addEventListener('loadedmetadata', function() {
    if (video.buffered.length === 0) return;

    const bufferedSeconds = video.buffered.end(0) - video.buffered.start(0);
    console.log(`${bufferedSeconds} seconds of video are ready to play.`);
  });
</script>

Setting the preload attribute to auto indicates that the browser may cache enough data that complete playback is possible without requiring a stop for further buffering.

<video id="video" preload="auto" src="file.mp4" controls></video>

<script>
  video.addEventListener('loadedmetadata', function() {
    if (video.buffered.length === 0) return;

    const bufferedSeconds = video.buffered.end(0) - video.buffered.start(0);
    console.log(`${bufferedSeconds} seconds of video are ready to play.`);
  });
</script>

There are some caveats though. As this is just a hint, the browser may completely ignore the preload attribute. At the time of writing, here are some rules applied in Chrome:

When Data Saver is enabled, Chrome forces the preload value to none.
In Android 4.3, Chrome forces the preload value to none due to an Android Bug.
On a cellular connection (2G, 3G, and 4G), Chrome forces the preload value to metadata.

Tips #

If your website contains many video resources on the same domain, I would recommend you set the preload value to metadata or define the poster attribute and set preload to none. That way, you would avoid hitting the maximum number of HTTP connections to the same domain (6 according to the HTTP 1.1 spec) which can hang loading of resources. Note that this may also improve page speed if videos aren't part of your core user experience.

Link preload #

As covered in other articles, link preload is a declarative fetch that allows you to force the browser to make a request for a resource without blocking the load event and while the page is downloading. Resources loaded via <link rel="preload"> are stored locally in the browser, and are effectively inert until they're explicitly referenced in the DOM, JavaScript, or CSS.

Preload is different from prefetch in that it focuses on current navigation and fetches resources with priority based on their type (script, style, font, video, audio, etc.). It should be used to warm up the browser cache for current sessions.

Preload full video #

Here's how to preload a full video on your website so that when your JavaScript asks to fetch video content, it is read from cache as the resource may have already been cached by the browser. If the preload request hasn't finished yet, a regular network fetch will happen.

<link rel="preload" as="video" href="https://cdn.com/small-file.mp4">

<video id="video" controls></video>

<script>
  // Later on, after some condition has been met, set video source to the
  // preloaded video URL.
  video.src = 'https://cdn.com/small-file.mp4';
  video.play().then(() => {
    // If preloaded video URL was already cached, playback started immediately.
  });
</script>

Because the preloaded resource is going to be consumed by a video element in the example, the as preload link value is video. If it were an audio element, it would be as="audio".

Preload the first segment #

The example below shows how to preload the first segment of a video with <link rel="preload"> and use it with Media Source Extensions. If you're not familiar with the MSE JavaScript API, see MSE basics.

For the sake of simplicity, let's assume the entire video has been split into smaller files like file_1.webm, file_2.webm, file_3.webm, etc.

<link rel="preload" as="fetch" href="https://cdn.com/file_1.webm">

<video id="video" controls></video>

<script>
  const mediaSource = new MediaSource();
  video.src = URL.createObjectURL(mediaSource);
  mediaSource.addEventListener('sourceopen', sourceOpen, { once: true });

  function sourceOpen() {
    URL.revokeObjectURL(video.src);
    const sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="vp09.00.10.08"');

    // If video is preloaded already, fetch will return immediately a response
    // from the browser cache (memory cache). Otherwise, it will perform a
    // regular network fetch.
    fetch('https://cdn.com/file_1.webm')
    .then(response => response.arrayBuffer())
    .then(data => {
      // Append the data into the new sourceBuffer.
      sourceBuffer.appendBuffer(data);
      // TODO: Fetch file_2.webm when user starts playing video.
    })
    .catch(error => {
      // TODO: Show "Video is not available" message to user.
    });
  }
</script>

Support #

See MDN's Browser compatibility table to see which browsers support preload. You may want to detect its availability with the snippets below to adjust your performance metrics.

function preloadFullVideoSupported() {
  const link = document.createElement('link');
  link.as = 'video';
  return (link.as === 'video');
}

function preloadFirstSegmentSupported() {
  const link = document.createElement('link');
  link.as = 'fetch';
  return (link.as === 'fetch');
}

Manual buffering #

Before we dive into the Cache API and service workers, let's see how to manually buffer a video with MSE. The example below assumes that your web server supports HTTP Range requests but this would be pretty similar with file segments. Note that some middleware libraries such as Google's Shaka Player, JW Player, and Video.js are built to handle this for you.

<video id="video" controls></video>

<script>
  const mediaSource = new MediaSource();
  video.src = URL.createObjectURL(mediaSource);
  mediaSource.addEventListener('sourceopen', sourceOpen, { once: true });

  function sourceOpen() {
    URL.revokeObjectURL(video.src);
    const sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="vp09.00.10.08"');

    // Fetch beginning of the video by setting the Range HTTP request header.
    fetch('file.webm', { headers: { range: 'bytes=0-567139' } })
    .then(response => response.arrayBuffer())
    .then(data => {
      sourceBuffer.appendBuffer(data);
      sourceBuffer.addEventListener('updateend', updateEnd, { once: true });
    });
  }

  function updateEnd() {
    // Video is now ready to play!
    const bufferedSeconds = video.buffered.end(0) - video.buffered.start(0);
    console.log(`${bufferedSeconds} seconds of video are ready to play.`);

    // Fetch the next segment of video when user starts playing the video.
    video.addEventListener('playing', fetchNextSegment, { once: true });
  }

  function fetchNextSegment() {
    fetch('file.webm', { headers: { range: 'bytes=567140-1196488' } })
    .then(response => response.arrayBuffer())
    .then(data => {
      const sourceBuffer = mediaSource.sourceBuffers[0];
      sourceBuffer.appendBuffer(data);
      // TODO: Fetch further segment and append it.
    });
  }
</script>

Considerations #

As you're now in control of the entire media buffering experience, I suggest you consider the device's battery level, the "Data-Saver Mode" user preference and network information when thinking about preloading.

Battery awareness #

Take into account the battery level of users' devices before thinking about preloading a video. This will preserve battery life when the power level is low.

Disable preload or at least preload a lower resolution video when the device is running out of battery.

if ('getBattery' in navigator) {
  navigator.getBattery()
  .then(battery => {
    // If battery is charging or battery level is high enough
    if (battery.charging || battery.level > 0.15) {
      // TODO: Preload the first segment of a video.
    }
  });
}

Detect "Data-Saver" #

Use the Save-Data client hint request header to deliver fast and light applications to users who have opted-in to "data savings" mode in their browser. By identifying this request header, your application can customize and deliver an optimized user experience to cost- and performance-constrained users.

See Delivering Fast and Light Applications with Save-Data to learn more.

Smart loading based on network information #

You may want to check navigator.connection.type prior to preloading. When it's set to cellular, you could prevent preloading and advise users that their mobile network operator might be charging for the bandwidth, and only start automatic playback of previously cached content.

if ('connection' in navigator) {
  if (navigator.connection.type == 'cellular') {
    // TODO: Prompt user before preloading video
  } else {
    // TODO: Preload the first segment of a video.
  }
}

Check out the Network Information sample to learn how to react to network changes as well.

Pre-cache multiple first segments #

Now what if I want to speculatively pre-load some media content without knowing which piece of media the user will eventually pick? If the user is on a web page that contains 10 videos, we probably have enough memory to fetch one segment file from each but we should definitely not create 10 hidden <video> elements and 10 MediaSource objects and start feeding that data.

The two-part example below shows you how to pre-cache multiple first segments of video using the powerful and easy-to-use Cache API. Note that something similar can be achieved with IndexedDB as well. We're not using service workers yet as the Cache API is also accessible from the window object.

Fetch and cache #

const videoFileUrls = [
  'bat_video_file_1.webm',
  'cow_video_file_1.webm',
  'dog_video_file_1.webm',
  'fox_video_file_1.webm',
];

// Let's create a video pre-cache and store all first segments of videos inside.
window.caches.open('video-pre-cache')
.then(cache => Promise.all(videoFileUrls.map(videoFileUrl => fetchAndCache(videoFileUrl, cache))));

function fetchAndCache(videoFileUrl, cache) {
  // Check first if video is in the cache.
  return cache.match(videoFileUrl)
  .then(cacheResponse => {
    // Let's return cached response if video is already in the cache.
    if (cacheResponse) {
      return cacheResponse;
    }
    // Otherwise, fetch the video from the network.
    return fetch(videoFileUrl)
    .then(networkResponse => {
      // Add the response to the cache and return network response in parallel.
      cache.put(videoFileUrl, networkResponse.clone());
      return networkResponse;
    });
  });
}

Note that if I were to use HTTP Range requests, I would have to manually recreate a Response object as the Cache API doesn't support Range responses yet. Be mindful that calling networkResponse.arrayBuffer() fetches the whole content of the response at once into renderer memory, which is why you may want to use small ranges.

For reference, I've modified part of the example above to save HTTP Range requests to the video precache.

    ...
    return fetch(videoFileUrl, { headers: { range: 'bytes=0-567139' } })
    .then(networkResponse => networkResponse.arrayBuffer())
    .then(data => {
      const response = new Response(data);
      // Add the response to the cache and return network response in parallel.
      cache.put(videoFileUrl, response.clone());
      return response;
    });

Play video #

When a user clicks a play button, we'll fetch the first segment of video available in the Cache API so that playback starts immediately if available. Otherwise, we'll simply fetch it from the network. Keep in mind that browsers and users may decide to clear the Cache.

As seen before, we use MSE to feed that first segment of video to the video element.

function onPlayButtonClick(videoFileUrl) {
  video.load(); // Used to be able to play video later.

  window.caches.open('video-pre-cache')
  .then(cache => fetchAndCache(videoFileUrl, cache)) // Defined above.
  .then(response => response.arrayBuffer())
  .then(data => {
    const mediaSource = new MediaSource();
    video.src = URL.createObjectURL(mediaSource);
    mediaSource.addEventListener('sourceopen', sourceOpen, { once: true });

    function sourceOpen() {
      URL.revokeObjectURL(video.src);

      const sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="vp09.00.10.08"');
      sourceBuffer.appendBuffer(data);

      video.play().then(() => {
        // TODO: Fetch the rest of the video when user starts playing video.
      });
    }
  });
}

Create Range responses with a service worker #

Now what if you have fetched an entire video file and saved it in the Cache API? When the browser sends an HTTP Range request, you certainly don't want to bring the entire video into renderer memory as the Cache API doesn't support Range responses yet.

So let me show how to intercept these requests and return a customized Range response from a service worker.

addEventListener('fetch', event => {
  event.respondWith(loadFromCacheOrFetch(event.request));
});

function loadFromCacheOrFetch(request) {
  // Search through all available caches for this request.
  return caches.match(request)
  .then(response => {

    // Fetch from network if it's not already in the cache.
    if (!response) {
      return fetch(request);
      // Note that we may want to add the response to the cache and return
      // network response in parallel as well.
    }

    // Browser sends a HTTP Range request. Let's provide one reconstructed
    // manually from the cache.
    if (request.headers.has('range')) {
      return response.blob()
      .then(data => {

        // Get start position from Range request header.
        const pos = Number(/^bytes\=(\d+)\-/g.exec(request.headers.get('range'))[1]);
        const options = {
          status: 206,
          statusText: 'Partial Content',
          headers: response.headers
        }
        const slicedResponse = new Response(data.slice(pos), options);
        slicedResponse.setHeaders('Content-Range': 'bytes ' + pos + '-' +
            (data.size - 1) + '/' + data.size);
        slicedResponse.setHeaders('X-From-Cache': 'true');

        return slicedResponse;
      });
    }

    return response;
  }
}

It is important to note that I used response.blob() to recreate this sliced response as this simply gives me a handle to the file while response.arrayBuffer() brings the entire file into renderer memory.

My custom X-From-Cache HTTP header can be used to know whether this request came from the cache or from the network. It can be used by a player such as ShakaPlayer to ignore the response time as an indicator of network speed.

Have a look at the official Sample Media App and in particular its ranged-response.js file for a complete solution for how to handle Range requests.

Mobile Web Video Playback

2017-04-07T00:00:00Z

How do you create the best mobile media experience on the Web? Easy! It all depends on user engagement and the importance you give to the media on a web page. I think we all agree that if video is THE reason for a user's visit, the user's experience has to be immersive and re-engaging.

In this article I show you how to enhance in a progressive way your media experience and make it more immersive thanks to a plethora of Web APIs. That's why we're going to build a simple mobile player experience with custom controls, fullscreen, and background playback. You can try the sample now and find the code in our GitHub repository.

Custom controls #

Figure 1.HTML Layout

As you can see, the HTML layout we're going to use for our media player is pretty simple: a <div> root element contains a <video> media element and a <div> child element dedicated to video controls.

Video controls we will cover later, include: a play/pause button, a fullscreen button, seek backward and forward buttons, and some elements for current time, duration and time tracking.

<div id="videoContainer">
  <video id="video" src="file.mp4"></video>
  <div id="videoControls"></div>
</div>

Read video metadata #

First, let's wait for the video metadata to be loaded to set the video duration, the current time, and initialize the progress bar. Note that the secondsToTimeCode() function is a custom utility function I've written that converts a number of seconds to a string in "hh:mm:ss" format which is better suited in our case.

<div id="videoContainer">
  <video id="video" src="file.mp4"></video>
  <div id="videoControls">
    <strong>
      <div id="videoCurrentTime"></div>
      <div id="videoDuration"></div>
      <div id="videoProgressBar"></div>
    </strong>
  </div>
</div>

video.addEventListener('loadedmetadata', function () {
  videoDuration.textContent = secondsToTimeCode(video.duration);
  videoCurrentTime.textContent = secondsToTimeCode(video.currentTime);
  videoProgressBar.style.transform = `scaleX(${
    video.currentTime / video.duration
  })`;
});

Figure 2. Media Player showing video metadata

Play/pause video #

Now that video metadata are loaded, let's add our first button that lets user play and pause video with video.play() and video.pause() depending on its playback state.

<div id="videoContainer">
  <video id="video" src="file.mp4"></video>
  <div id="videoControls">
    <strong><button id="playPauseButton"></button></strong>
    <div id="videoCurrentTime"></div>
    <div id="videoDuration"></div>
    <div id="videoProgressBar"></div>
  </div>
</div>

playPauseButton.addEventListener('click', function (event) {
  event.stopPropagation();
  if (video.paused) {
    video.play();
  } else {
    video.pause();
  }
});

Rather than adjusting our video controls in the click event listener, we use the play and pause video events. Making our controls events based helps with flexibility (as we'll see later with the Media Session API) and will allow us to keep our controls in sync if the browser intervenes in the playback. When video starts playing, we change the button state to "pause" and hide the video controls. When the video pauses, we simply change button state to "play" and show the video controls.

video.addEventListener('play', function () {
  playPauseButton.classList.add('playing');
});

video.addEventListener('pause', function () {
  playPauseButton.classList.remove('playing');
});

When time indicated by video currentTime attribute changed via the timeupdate video event, we also update our custom controls if they're visible.

video.addEventListener('timeupdate', function () {
  if (videoControls.classList.contains('visible')) {
    videoCurrentTime.textContent = secondsToTimeCode(video.currentTime);
    videoProgressBar.style.transform = `scaleX(${
      video.currentTime / video.duration
    })`;
  }
});

When the video ends, we simply change button state to "play", set video currentTime back to 0 and show video controls for now. Note that we could also choose to load automatically another video if the user has enabled some kind of "AutoPlay" feature.

video.addEventListener('ended', function () {
  playPauseButton.classList.remove('playing');
  video.currentTime = 0;
});

Seek backward and forward #

Let's continue and add "seek backward" and "seek forward" buttons so that user can easily skip some content.

<div id="videoContainer">
  <video id="video" src="file.mp4"></video>
  <div id="videoControls">
    <button id="playPauseButton"></button>
    <strong
      ><button id="seekForwardButton"></button>
      <button id="seekBackwardButton"></button
    ></strong>
    <div id="videoCurrentTime"></div>
    <div id="videoDuration"></div>
    <div id="videoProgressBar"></div>
  </div>
</div>

var skipTime = 10; // Time to skip in seconds

seekForwardButton.addEventListener('click', function (event) {
  event.stopPropagation();
  video.currentTime = Math.min(video.currentTime + skipTime, video.duration);
});

seekBackwardButton.addEventListener('click', function (event) {
  event.stopPropagation();
  video.currentTime = Math.max(video.currentTime - skipTime, 0);
});

As before, rather than adjusting video styling in the click event listeners of these buttons, we'll use the fired seeking and seeked video events to adjust video brightness. My custom seeking CSS class is as simple as filter: brightness(0);.

video.addEventListener('seeking', function () {
  video.classList.add('seeking');
});

video.addEventListener('seeked', function () {
  video.classList.remove('seeking');
});

Here's below what we have created so far. In the next section, we'll implement the fullscreen button.

Fullscreen #

Here we are going to take advantage of several Web APIs to create a perfect and seamless fullscreen experience. To see it in action, check out the sample.

Obviously, you don't have to use all of them. Just pick the ones that make sense to you and combine them to create your custom flow.

Prevent automatic fullscreen #

On iOS, video elements automagically enter fullscreen mode when media playback begins. As we're trying to tailor and control as much as possible our media experience across mobile browsers, I recommend you set the playsinline attribute of the video element to force it to play inline on iPhone and not enter fullscreen mode when playback begins. Note that this has no side effects on other browsers.

<div id="videoContainer"></div>
  <video id="video" src="file.mp4"></video><strong>playsinline</strong></video>
  <div id="videoControls">...</div>
</div>

Toggle fullscreen on button click #

Now that we prevent automatic fullscreen, we need to handle ourselves the fullscreen mode for the video with the Fullscreen API. When user clicks the "fullscreen button", let's exit fullscreen mode with document.exitFullscreen() if fullscreen mode is currently in use by the document. Otherwise, request fullscreen on the video container with the method requestFullscreen() if available or fallback to webkitEnterFullscreen() on the video element only on iOS.

<div id="videoContainer">
  <video id="video" src="file.mp4"></video>
  <div id="videoControls">
    <button id="playPauseButton"></button>
    <button id="seekForwardButton"></button>
    <button id="seekBackwardButton"></button>
    <strong><button id="fullscreenButton"></button></strong>
    <div id="videoCurrentTime"></div>
    <div id="videoDuration"></div>
    <div id="videoProgressBar"></div>
  </div>
</div>

fullscreenButton.addEventListener('click', function (event) {
  event.stopPropagation();
  if (document.fullscreenElement) {
    document.exitFullscreen();
  } else {
    requestFullscreenVideo();
  }
});

function requestFullscreenVideo() {
  if (videoContainer.requestFullscreen) {
    videoContainer.requestFullscreen();
  } else {
    video.webkitEnterFullscreen();
  }
}

document.addEventListener('fullscreenchange', function () {
  fullscreenButton.classList.toggle('active', document.fullscreenElement);
});

Toggle fullscreen on screen orientation change #

As user rotates device in landscape mode, let's be smart about this and automatically request fullscreen to create an immersive experience. For this, we'll need the Screen Orientation API which is not yet supported everywhere and still prefixed in some browsers at that time. Thus, this will be our first progressive enhancement.

How does this work? As soon as we detect the screen orientation changes, let's request fullscreen if the browser window is in landscape mode (that is, its width is greater than its height). If not, let's exit fullscreen. That's all.

if ('orientation' in screen) {
  screen.orientation.addEventListener('change', function () {
    // Let's request fullscreen if user switches device in landscape mode.
    if (screen.orientation.type.startsWith('landscape')) {
      requestFullscreenVideo();
    } else if (document.fullscreenElement) {
      document.exitFullscreen();
    }
  });
}

Lock screen in landscape on button click #

As video may be better viewed in landscape mode, we may want to lock screen in landscape when user clicks the "fullscreen button". We're going to combine the previously used Screen Orientation API and some media queries to make sure this experience is the best.

Locking screen in landscape is as easy as calling screen.orientation.lock('landscape'). However, we should do this only when device is in portrait mode with matchMedia('(orientation: portrait)') and can be held in one hand with matchMedia('(max-device-width: 768px)') as this wouldn't be a great experience for users on tablet.

fullscreenButton.addEventListener('click', function (event) {
  event.stopPropagation();
  if (document.fullscreenElement) {
    document.exitFullscreen();
  } else {
    requestFullscreenVideo();
    <strong>lockScreenInLandscape();</strong>;
  }
});

function lockScreenInLandscape() {
  if (!('orientation' in screen)) {
    return;
  }
  // Let's force landscape mode only if device is in portrait mode and can be held in one hand.
  if (
    matchMedia('(orientation: portrait) and (max-device-width: 768px)').matches
  ) {
    screen.orientation.lock('landscape');
  }
}

Unlock screen on device orientation change #

You may have noticed the lock screen experience we've just created isn't perfect though as we don't receive screen orientation changes when screen is locked.

In order to fix this, let's use the Device Orientation API if available. This API provides information from the hardware measuring a device's position and motion in space: gyroscope and digital compass for its orientation, and accelerometer for its velocity. When we detect a device orientation change, let's unlock screen with screen.orientation.unlock() if user holds device in portrait mode and screen is locked in landscape mode.

function lockScreenInLandscape() {
  if (!('orientation' in screen)) {
    return;
  }
  // Let's force landscape mode only if device is in portrait mode and can be held in one hand.
  if (matchMedia('(orientation: portrait) and (max-device-width: 768px)').matches) {
    screen.orientation.lock('landscape')
    <strong>.then(function() {
      listenToDeviceOrientationChanges();
    })</strong>;
  }
}

function listenToDeviceOrientationChanges() {
  if (!('DeviceOrientationEvent' in window)) {
    return;
  }
  var previousDeviceOrientation, currentDeviceOrientation;
  window.addEventListener(
    'deviceorientation',
    function onDeviceOrientationChange(event) {
      // event.beta represents a front to back motion of the device and
      // event.gamma a left to right motion.
      if (Math.abs(event.gamma) > 10 || Math.abs(event.beta) < 10) {
        previousDeviceOrientation = currentDeviceOrientation;
        currentDeviceOrientation = 'landscape';
        return;
      }
      if (Math.abs(event.gamma) < 10 || Math.abs(event.beta) > 10) {
        previousDeviceOrientation = currentDeviceOrientation;
        // When device is rotated back to portrait, let's unlock screen orientation.
        if (previousDeviceOrientation == 'landscape') {
          screen.orientation.unlock();
          window.removeEventListener(
            'deviceorientation',
            onDeviceOrientationChange,
          );
        }
      }
    },
  );
}

As you can see, this is the seamless fullscreen experience we were looking for. To see this in action, check out the sample.

Background playback #

When you detect a web page or a video in the web page is not visible anymore, you may want to update your analytics to reflect this. This could also affect the current playback as in picking a different track, pause it, or even show custom buttons to the user for instance.

Pause video on page visibility change #

With the Page Visibility API, we can determine the current visibility of a page and be notified of visibility changes. Code below pauses video when page is hidden. This happens when screen lock is active or when you switch tabs for instance.

As most mobile browsers now offer controls outside of the browser that allow resuming a paused video, I recommend you set this behaviour only if user is allowed to play in the background.

document.addEventListener('visibilitychange', function () {
  // Pause video when page is hidden.
  if (document.hidden) {
    video.pause();
  }
});

Show/hide mute button on video visibility change #

If you use the new Intersection Observer API, you can be even more granular at no cost. This API lets you know when an observed element enters or exits the browser's viewport.

Let's show/hide a mute button based on the video visibility in the page. If video is playing but not currently visible, a mini mute button will be shown in the bottom right corner of the page to give user control over video sound. The volumechange video event is used to update the mute button styling.

<button id="muteButton"></button>

if ('IntersectionObserver' in window) {
  // Show/hide mute button based on video visibility in the page.
  function onIntersection(entries) {
    entries.forEach(function (entry) {
      muteButton.hidden = video.paused || entry.isIntersecting;
    });
  }
  var observer = new IntersectionObserver(onIntersection);
  observer.observe(video);
}

muteButton.addEventListener('click', function () {
  // Mute/unmute video on button click.
  video.muted = !video.muted;
});

video.addEventListener('volumechange', function () {
  muteButton.classList.toggle('active', video.muted);
});

Play only one video at a time #

If there are more than one video on a page, I would suggest you only play one and pause the other ones automatically so that user doesn't have to hear multiple audio tracks playing simultaneously.

// This array should be initialized once all videos have been added.
var videos = Array.from(document.querySelectorAll('video'));

videos.forEach(function (video) {
  video.addEventListener('play', pauseOtherVideosPlaying);
});

function pauseOtherVideosPlaying(event) {
  var videosToPause = videos.filter(function (video) {
    return !video.paused && video != event.target;
  });
  // Pause all other videos currently playing.
  videosToPause.forEach(function (video) {
    video.pause();
  });
}

Customize Media Notifications #

With the Media Session API, you can also customize media notifications by providing metadata for the currently playing video. It also allows you to handle media related events such as seeking or track changing which may come from notifications or media keys. To see this in action, check out the sample.

When your web app is playing audio or video, you can already see a media notification sitting in the notification tray. On Android, Chrome does its best to show appropriate information by using the document's title and the largest icon image it can find.

Let's see how to customize this media notification by setting some media session metadata such as the title, artist, album name, and artwork with the Media Session API.

playPauseButton.addEventListener('click', function(event) {
  event.stopPropagation();
  if (video.paused) {
    video.play()
    <strong>.then(function() {
      setMediaSession();
    });</strong>
  } else {
    video.pause();
  }
});

function setMediaSession() {
  if (!('mediaSession' in navigator)) {
    return;
  }
  navigator.mediaSession.metadata = new MediaMetadata({
    title: 'Never Gonna Give You Up',
    artist: 'Rick Astley',
    album: 'Whenever You Need Somebody',
    artwork: [
      {src: 'https://dummyimage.com/96x96', sizes: '96x96', type: 'image/png'},
      {
        src: 'https://dummyimage.com/128x128',
        sizes: '128x128',
        type: 'image/png',
      },
      {
        src: 'https://dummyimage.com/192x192',
        sizes: '192x192',
        type: 'image/png',
      },
      {
        src: 'https://dummyimage.com/256x256',
        sizes: '256x256',
        type: 'image/png',
      },
      {
        src: 'https://dummyimage.com/384x384',
        sizes: '384x384',
        type: 'image/png',
      },
      {
        src: 'https://dummyimage.com/512x512',
        sizes: '512x512',
        type: 'image/png',
      },
    ],
  });
}

Once playback is done, you don't have to "release" the media session as the notification will automatically disappear. Keep in mind that current navigator.mediaSession.metadata will be used when any playback starts. This is why you need to update it to make sure you're always showing relevant information in the media notification.

If your web app provides a playlist, you may want to allow the user to navigate through your playlist directly from the media notification with some "Previous Track" and "Next Track" icons.

if ('mediaSession' in navigator) {
  navigator.mediaSession.setActionHandler('previoustrack', function () {
    // User clicked "Previous Track" media notification icon.
    playPreviousVideo(); // load and play previous video
  });
  navigator.mediaSession.setActionHandler('nexttrack', function () {
    // User clicked "Next Track" media notification icon.
    playNextVideo(); // load and play next video
  });
}

Note that media action handlers will persist. This is very similar to the event listener pattern except that handling an event means that the browser stops doing any default behaviour and uses this as a signal that your web app supports the media action. Hence, media action controls won't be shown unless you set the proper action handler.

By the way, unsetting a media action handler is as easy as assigning it to null.

The Media Session API allows you to show "Seek Backward" and "Seek Forward" media notification icons if you want to control the amount of time skipped.

if ('mediaSession' in navigator) {
  let skipTime = 10; // Time to skip in seconds

  navigator.mediaSession.setActionHandler('seekbackward', function () {
    // User clicked "Seek Backward" media notification icon.
    video.currentTime = Math.max(video.currentTime - skipTime, 0);
  });
  navigator.mediaSession.setActionHandler('seekforward', function () {
    // User clicked "Seek Forward" media notification icon.
    video.currentTime = Math.min(video.currentTime + skipTime, video.duration);
  });
}

The "Play/Pause" icon is always shown in the media notification and the related events are handled automatically by the browser. If for some reason the default behaviour doesn't work out, you can still handle "Play" and "Pause" media events.

The cool thing about the Media Session API is that the notification tray is not the only place where media metadata and controls are visible. The media notification is synced automagically to any paired wearable device. And it also shows up on lock screens.

Feedback #

Media Source Extensions

2017-02-08T00:00:00Z

Media Source Extensions (MSE) is a JavaScript API that lets you build streams for playback from segments of audio or video. Although not covered in this article, understanding MSE is needed if you want to embed videos in your site that do things like:

Adaptive streaming, which is another way of saying adapting to device capabilities and network conditions
Adaptive splicing, such as ad insertion
Time shifting
Control of performance and download size

Figure 1: Basic MSE data flow

You can almost think of MSE as a chain. As illustrated in the figure, between the downloaded file and the media elements are several layers.

An <audio> or <video> element to play the media.
A MediaSource instance with a SourceBuffer to feed the media element.
A fetch() or XHR call to retrieve media data in a Response object.
A call to Response.arrayBuffer() to feed MediaSource.SourceBuffer.

In practice, the chain looks like this:

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  mediaSource.addEventListener('sourceopen', sourceOpen);
} else {
  console.log('The Media Source Extensions API is not supported.');
}

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  fetch(videoUrl)
    .then(function (response) {
      return response.arrayBuffer();
    })
    .then(function (arrayBuffer) {
      sourceBuffer.addEventListener('updateend', function (e) {
        if (!sourceBuffer.updating && mediaSource.readyState === 'open') {
          mediaSource.endOfStream();
        }
      });
      sourceBuffer.appendBuffer(arrayBuffer);
    });
}

If you can sort things out from the explanations so far, feel free to stop reading now. If you want a more detailed explanation, then please keep reading. I'm going to walk through this chain by building a basic MSE example. Each of the build steps will add code to the previous step.

A note about clarity #

Will this article tell you everything you need to know about playing media on a web page? No, it's only intended to help you understand more complicated code you might find elsewhere. For the sake of clarity, this document simplifies and excludes many things. We think we can get away with this because we also recommend using a library such as Google's Shaka Player. I will note throughout where I'm deliberately simplifying.

A few things not covered #

Here, in no particular order, are a few things I won't cover.

Playback controls. We get those for free by virtue of using the HTML5 <audio> and <video> elements.
Error handling.

For use in production environments #

Here are some things I'd recommend in a production usage of MSE related APIs:

Before making calls on these APIs, handle any error events or API exceptions, and check HTMLMediaElement.readyState and MediaSource.readyState. These values can change before associated events are delivered.
Make sure previous appendBuffer() and remove() calls are not still in progress by checking the SourceBuffer.updating boolean value before updating the SourceBuffer's mode, timestampOffset, appendWindowStart, appendWindowEnd, or calling appendBuffer() or remove() on the SourceBuffer.
For all SourceBuffer instances added to your MediaSource, ensure none of their updating values are true before calling MediaSource.endOfStream() or updating the MediaSource.duration.
If MediaSource.readyState value is ended, calls like appendBuffer() and remove(), or setting SourceBuffer.mode or SourceBuffer.timestampOffset will cause this value to transition to open. This means you should be prepared to handle multiple sourceopen events.
When handling HTMLMediaElement error events, the contents of MediaError.message can be useful to determine the root cause of the failure, especially for errors that are hard to reproduce in test environments.

Attach a MediaSource instance to a media element #

As with many things in web development these days, you start with feature detection. Next, get a media element, either an <audio> or <video> element. Finally create an instance of MediaSource. It gets turned into a URL and passed to the media element's source attribute.

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  // Is the MediaSource instance ready?
} else {
  console.log('The Media Source Extensions API is not supported.');
}

Figure 1: A source attribute as a blob

That a MediaSource object can be passed to a src attribute might seem a bit odd. They're usually strings, but they can also be blobs. If you inspect a page with embedded media and examine its media element, you'll see what I mean.

Is the MediaSource instance ready? #

URL.createObjectURL() is itself synchronous; however, it processes the attachment asynchronously. This causes a slight delay before you can do anything with the MediaSource instance. Fortunately, there are ways to test for this. The simplest way is with a MediaSource property called readyState. The readyState property describes the relation between a MediaSource instance and a media element. It can have one of the following values:

closed - The MediaSource instance is not attached to a media element.
open - The MediaSource instance is attached to a media element and is ready to receive data or is receiving data.
ended - The MediaSource instance is attached to a media element and all of its data has been passed to that element.

Querying these options directly can negatively affect performance. Fortunately, MediaSource also fires events when readyState changes, specifically sourceopen, sourceclosed, sourceended. For the example I'm building, I'm going to use the sourceopen event to tell me when to fetch and buffer the video.

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  <strong>mediaSource.addEventListener('sourceopen', sourceOpen);</strong>
} else {
  console.log("The Media Source Extensions API is not supported.")
}

<strong>function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  // Create a SourceBuffer and get the media file.
}</strong>

Notice that I've also called revokeObjectURL(). I know this seems premature, but I can do this any time after the media element's src attribute is connected to a MediaSource instance. Calling this method doesn't destroy any objects. It does allow the platform to handle garbage collection at an appropriate time, which is why I'm calling it immediately.

Create a SourceBuffer #

Now it's time to create the SourceBuffer, which is the object that actually does the work of shuttling data between media sources and media elements. A SourceBuffer has to be specific to the type of media file you're loading.

In practice you can do this by calling addSourceBuffer() with the appropriate value. Notice that in the example below the mime type string contains a mime type and two codecs. This is a mime string for a video file, but it uses separate codecs for the video and audio portions of the file.

Version 1 of the MSE spec allows user agents to differ on whether to require both a mime type and a codec. Some user agents don't require, but do allow just the mime type. Some user agents, Chrome for example, require a codec for mime types that don't self-describe their codecs. Rather than trying to sort all this out, it's better to just include both.

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  mediaSource.addEventListener('sourceopen', sourceOpen);
} else {
  console.log('The Media Source Extensions API is not supported.');
}

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  <strong>
    var mime = 'video/webm; codecs="opus, vp09.00.10.08"'; // e.target refers to
    the mediaSource instance. // Store it in a variable so it can be used in a
    closure. var mediaSource = e.target; var sourceBuffer =
    mediaSource.addSourceBuffer(mime); // Fetch and process the video.
  </strong>;
}

Get the media file #

If you do an internet search for MSE examples, you'll find plenty that retrieve media files using XHR. To be more cutting edge, I'm going to use the Fetch API and the Promise it returns. If you're trying to do this in Safari, it won't work without a fetch() polyfill.

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  <strong>
    fetch(videoUrl) .then(function(response){' '}
    {
      // Process the response object.
    }
    );
  </strong>;
}

A production quality player would have the same file in multiple versions to support different browsers. It could use separate files for audio and video to allow audio to be selected based on language settings.

Real world code would also have multiple copies of media files at different resolutions so that it could adapt to different device capabilities and network conditions. Such an application is able to load and play videos in chunks either using range requests or segments. This allows for adaption to network conditions while media are playing. You may have heard the terms DASH or HLS, which are two methods of accomplishing this. A full discussion of this topic is beyond the scope of this introduction.

Process the response object #

The code looks almost done, but the media doesn't play. We need to get media data from the Response object to the SourceBuffer.

The typical way to pass data from the response object to the MediaSource instance is to get an ArrayBuffer from the response object and pass it to the SourceBuffer. Start by calling response.arrayBuffer(), which returns a promise to the buffer. In my code, I've passed this promise to a second then() clause where I append it to the SourceBuffer.

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  fetch(videoUrl)
    .then(function(response) {
      <strong>return response.arrayBuffer();</strong>
    })
    <strong>.then(function(arrayBuffer) {
      sourceBuffer.appendBuffer(arrayBuffer);
    });</strong>
}

Call endOfStream() #

After all ArrayBuffers are appended, and no further media data is expected, call MediaSource.endOfStream(). This will change MediaSource.readyState to ended and fire the sourceended event.

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  fetch(videoUrl)
    .then(function(response) {
      return response.arrayBuffer();
    })
    .then(function(arrayBuffer) {
      <strong>sourceBuffer.addEventListener('updateend', function(e) {
        if (!sourceBuffer.updating && mediaSource.readyState === 'open') {
          mediaSource.endOfStream();
        }
      });</strong>
      sourceBuffer.appendBuffer(arrayBuffer);
    });
}

The final version #

Here's the complete code example. I hope you have learned something about Media Source Extensions.

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  mediaSource.addEventListener('sourceopen', sourceOpen);
} else {
  console.log('The Media Source Extensions API is not supported.');
}

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  fetch(videoUrl)
    .then(function (response) {
      return response.arrayBuffer();
    })
    .then(function (arrayBuffer) {
      sourceBuffer.addEventListener('updateend', function (e) {
        if (!sourceBuffer.updating && mediaSource.readyState === 'open') {
          mediaSource.endOfStream();
        }
      });
      sourceBuffer.appendBuffer(arrayBuffer);
    });
}

François Beaufort on web.dev

How to process audio from the user's microphone

Browser support #

MediaDevices.getUserMedia() #

Web Audio #

AudioWorklet #

Further reading #

Demo #

New patterns for media apps

How to customize media notifications

Further reading #

Demo #

How to record audio from the user's microphone

Browser support #

MediaDevices.getUserMedia() #

MediaRecorder API #

File System Access API's showSaveFilePicker() #

Further reading #

Demo #

How to add Picture-in-Picture to video controls

The modern way #

The classic way #

Further reading #

Demo #

How to record the user's screen

Browser support #

MediaDevices.getDisplayMedia() #

MediaRecorder API #

File System Access API's showSaveFilePicker() #

Further reading #

Demo #

How to create app shortcuts

The modern way #

Define app shortcuts in the web app manifest #

The classic way #

Let user drag links to the bookmark bar #

Further reading #

Demo #

Synchronize audio and video playback on the web

AudioContext outputLatency #

Debug media playback errors on the web

User preference media features client hints headers

Background on client hints #

Background on critical client hints #

Client hint syntax #

Complete list of the client hints #

Browser support #

Demo of Sec-CH-Prefers-Color-Scheme #

Demo of Sec-CH-Prefers-Reduced-Motion #

How it works #

Node.js example #

Privacy and security considerations #

References #

Acknowledgements #

Accessing hardware devices on the web

🕹 Receive input events from this device #

📸 Access audio and video from this device #

🖨 Print to this device #

🔐 Authenticate with this device #

🗄 Access files on this device #

🧲 Access sensors on this device #

🛰 Access GPS coordinates on this device #

🔋 Check the battery on this device #

📞 Communicate with this device over the network #

🧱 Discover how to communicate with your device #

Acknowledgements #

Disable mouse acceleration to provide a better FPS gaming experience

Using the API #

Request a pointer lock #

Disable mouse acceleration #

Browser support #

Operating system support #

Sample #

Helpful links #

Acknowledgements #

How Chrome handles updates to the web app manifest

Updates on desktop Chrome #

Which properties will trigger an update? #

What happens when the display field is updated? #

Testing manifest updates #

Demo of `Sec-CH-Prefers-Color-Scheme` #

Demo of `Sec-CH-Prefers-Reduced-Motion` #

What happens when the `display` field is updated? #

`MediaDevices.getUserMedia()` #

`MediaDevices.getSupportedConstraints()` #

`MediaStreamTrack.applyConstraints()` #

`MediaStreamTrack.getCapabilities()` #

`MediaStreamTrack.getSettings()` #

`short_name` and/or `name` #

`icons` #

`id` #

`start_url` #

`background_color` #

`display` #

`display_override` #