Joe Medley on web.dev

How to copy text

2022-10-10T00:00:00Z

Cutting and pasting text is one of the most commonly used features of apps in general and desktop applications in particular. How do I copy text on the web? There's an old way and a new way. And it depends on which browser is used.

The modern way #

Using the Async Clipboard API #

The Clipboard.writeText() method takes a string and returns a Promise that resolves when text is successfully written to the clipboard. Clipboard.writeText() can only be used from the window object that has focus.

Browser support

Chrome 66, Supported 66
Firefox 63, Supported 63
Edge 79, Supported 79
Safari 13.1, Supported 13.1

Source

The classic way #

Using `document.execCommand()` #

Calling document.execCommand('copy') returns a boolean value that indicates whether the copy was successful. Call this command inside of a user gesture such as a click handler. This approach has limitations when compared to the Async Clipboard API. The execCommand() method only works with DOM elements. Because it's synchronous, copying large amounts of data, especially data that must be transformed or sanitized in some way, can block the page.

Browser support

Chrome 1, Supported 1
Firefox 1, Supported 1
Edge 12, Supported 12
Safari 1.3, Supported 1.3

Source

Progressive enhancement #

const copyButton = document.querySelector('#copyButton');
const out = document.querySelector('#out');

if ('clipboard' in navigator) {
  copyButton.addEventListener('click', () => {
    navigator.clipboard.writeText(out.value)
    .then(() => {
      console.log('Text copied');
    })
    .catch((err) => console.error(err.name, err.message));
  });
} else {
  copyButton.addEventListener('click', () => {
    const textArea = document.createElement('textarea');
    textArea.value = out.value;
    textArea.style.opacity = 0;
    document.body.appendChild(textArea);
    textArea.focus();
    textArea.select();
    try {
      const success = document.execCommand('copy');
      console.log(`Text copy was ${success ? 'successful' : 'unsuccessful'}.`);
    } catch (err) {
      console.error(err.name, err.message);
    }
    document.body.removeChild(textArea);
  });
}

Demo #

Media encryption

2021-07-05T00:00:00Z

In this section we are going to cover two different strategies for encrypting your media, and some practical examples on how to use them with FFmpeg and Shaka Packager. The two strategies for encryption we'll discuss are Clear Key and using a service like Google Widevine. Both strategies are a form of digital rights management (DRM) to control what users can do with your media. However, one is inherently less secure than the other due to the way keys are passed for authentication and is why a DRM service might make more sense.

The primary DRM services for the web are Google Widevine, Microsoft PlayReady and Apple FairPlay, but we will not be covering all of them in this article. However, if you are targeting all the modern browsers you are likely going to be using all three DRM services.

Conversion and encryption is done with these applications:

Clear Key encryption #

First, you should have a good understanding of what Clear Key is and isn't before using it. When you do not want to use an existing DRM service and feel basic encryption of you media is a viable option, you would use Clear Key. But, keep in mind that this type of encryption does not provide the same level of security as using one of the DRM services. This is because the key value pair is not encrypted under another key, unlike encrypted keys which are generated by a decryption key that is stored on a licence server. Additionally, Clear Key sends the key value pair as plain text, so while you are encrypting your media the key to decrypt it is not a secret.

Create a key #

You can use the same method to create a key for both DASH and HLS. Do this using OpenSSL. The following will create an encryption key made of 16 hex values.

openssl rand -hex 16 > media.key

Create an IV #

Next we can generate an initialization vector (IV).

openssl rand -hex 16
6143b5373a51cb46209cfed0d747da66

Encrypt with Clear Key #

The following example uses Shaka Packager with raw keys, where keys and key_ids are provided to Shaka Packager directly. Read the documentation for more examples.

For the key flag, use the key created earlier, which is stored in the media.key file. However, when entering it on the command line, be sure you've removed its whitespace. For the key_id flag, repeat the media.id value or use the IV value generated above.

packager \
  input=glocken.mp4,stream=audio,output=glockena.m4a \
  input=glocken.mp4,stream=video,output=glockenv.mp4 \
  --enable_fixed_key_encryption \
  --keys label=audio:key=INSERT_AUDIO_KEY_HERE:key_id=INSERT_AUDIO_KEY_ID_HERE,label=video:key=INSERT_VIDEO_KEY_HERE:key_id=INSERT_VIDEO_KEY_ID_HERE

Create a key information file #

To encrypt for HLS you need a key information file in addition to a key file. A key information file is a text file with the format below. It should have the extension .keyinfo. For example: encrypt.keyinfo.

key URI
key file path
private key

The key URI is where the media.key (created above will be located on your server. The key file path is its location relative to the key information file. Finally, the private key is the contents of the media.key file itself, or the IV you created before. For example:

https://example.com/keys/media.key
/path/to/media.key
6143b5373a51cb46209cfed0d747da66

Encrypt for HLS #

packager \
  'input=input.mp4,stream=video,segment_template=output$Number$.ts,playlist_name=video_playlist.m3u8' \
  'input=input.mp4,stream=audio,segment_template=output_audio$Number$.ts,playlist_name=audio_playlist.m3u8,hls_group_id=audio,hls_name=ENGLISH' \
  --hls_master_playlist_output="master_playlist.m3u8" \
  --hls_base_url="http://localhost:5000/"

This command will accept a key with either 16 or 32 characters.

ffmpeg -i myvideo.mov -c:v libx264 -c:a aac -hls_key_info_file encrypt.keyinfo myvideo.m3u8

Widevine encryption #

Now you know what Clear Key encryption is and when to use it. But, when should you use a DRM service for additional security? This is where Widevine, or another service, would be used to securely encrypt and decrypt your media. Widevine supports MPEG-DASH and HLS and is a DRM from Google. Widevine is used by the Google Chrome and Firefox web browsers, Android MediaDRM, Android TV, and other consumer electronics devices that use Encrypted Media Extensions and Media Source Extensions, where Widevine decrypts content.

Encrypt with Widevine #

Most of the examples in this article used Clear Key encryption. However, for Widevine you will want to replace the following options.

--enable_fixed_key_encryption \
--enable_fixed_key_decryption \
--keys label=:key=INSERT_KEY_HERE:key_id=INSERT_KEY_ID_HERE

Everything in the demultiplexer (demux) command except the name of your files and the --content-id flag should be copied exactly from the example. The --content-id is 16 or 32 random hex digits. Use the keys provided here instead of your own. Read the Shaka Packager documentation on using the Widevine Key Server for more examples.

Demux (separate) the audio and video, encrypt the new files, and output a media presentation description (MPD) file.

packager \
  input=tmp_glocken.mp4,stream=video,output=glocken_video.mp4 \
  input=tmp_glocken.mp4,stream=audio,output=glocken_audio.m4a \
  --enable_widevine_encryption \
  --key_server_url "https://license.uat.widevine.com/cenc/getcontentkey/widevine_test" \
  --content_id "fd385d9f9a14bb09" \
  --signer "widevine_test" \
  --aes_signing_key "1ae8ccd0e7985cc0b6203a55855a1034afc252980e970ca90e5202689f947ab9" \
  --aes_signing_iv "d58ce954203b7c9a9a9d467f59839249"

Remux (combine) the audio and video streams. If you're using a video framework, you may not need to do this.
```
ffmpeg -i glocken_video.mp4 -i glocken_audio.m4a -c copy glocke.mp4
```

Media conversion sequence #

This section shows in order commands needed to get from a raw .mov file to encrypted assets packaged for DASH or HLS. For the sake of having a goal to illustrate, we're converting a source file to a bitrate of 8Mbs at a resolution of 1080p (1920 x 1080). Adjust these values as your needs dictate.

DASH/WebM #

Convert the file type and codec.

For this command you can use either liborbis or libopus for the audio codec.
```
ffmpeg -i glocken.mov -c:v libvpx-vp9 -c:a libvorbis -b:v 8M -vf setsar=1:1 -f webm tmp_glocken.webm
```
Create a Clear Key encryption key.
```
openssl rand -hex 16 > media.key
```

Demux (separate) the audio and video, encrypt the new files, and output a media presentation description (MPD) file.

packager \
  input=tmp_glocken.webm,stream=video,output=glocken_video.webm \
  input=tmp_glocken.webm,stream=audio,output=glocken_audio.webm \
  --enable_fixed_key_encryption \
  --enable_fixed_key_decryption \
  --keys label=:key=INSERT_KEY_HERE:key_id=INSERT_KEY_ID_HERE \
  --mpd_output glocken_webm_vod.mpd

Remux (combine) the audio and video streams. If you're using a video framework, you may not need to do this.
```
ffmpeg -i glocken_video.webm -i glocken_audio.webm -c copy glocken.webm
```

DASH/MP4 #

Convert the file type, video codec and bitrate.

ffmpeg -i glocken.mov -c:v libx264 -c:a aac -b:v 8M -strict -2 tmp_glocken.mp4

Create a Clear Key encryption key.
```
openssl rand -hex 16 > media.key
```

Demux (separate) the audio and video, encrypt the new files, and output a media presentation description (MPD) file.

packager \
  input=tmp_glocken.mp4,stream=video,output=glocken_video.mp4 \
  input=tmp_glocken.mp4,stream=audio,output=glocken_audio.m4a \
  --enable_fixed_key_encryption \
  --enable_fixed_key_decryption \
  --keys label=:key=INSERT_KEY_HERE:key_id=INSERT_KEY_ID_HERE \
  --mpd_output glocken_mp4_vod.mpd

Remux (combine) the audio and video streams. If you're using a video framework, you may not need to do this.
```
ffmpeg -i glocken_video.mp4 -i glocken_audio.m4a -c copy glocken.mp4
```

HLS/MP4 #

HLS only supports MP4, so first you'll need to convert to the MP4 container and supported codecs.

Convert the file type, video codec, and bitrate.

ffmpeg -i glocken.mov -c:v libx264 -c:a aac -b:v 8M -strict -2 glocken.mp4

Create a Clear Key encryption key.
```
openssl rand -hex 16 > media.key
```

Create a key information file

packager \
  'input=glocken.mp4,stream=video,segment_template=output$Number$.ts,playlist_name=video_playlist.m3u8' \
  'input=glocken.mp4,stream=audio,segment_template=output_audio$Number$.ts,playlist_name=audio_playlist.m3u8,hls_group_id=audio,hls_name=ENGLISH' \
  --hls_master_playlist_output="master_playlist.m3u8" \
  --hls_base_url="http://localhost:5000/" \
  --enable_fixed_key_encryption \
  --enable_fixed_key_decryption \
  --keys label=:key=INSERT_KEY_HERE:key_id=INSERT_KEY_ID_HERE

That was a lot to digest, but hopefully you are now able to encrypt you media with confidence. Next we'll show you how to add media to your site.

Introducing 1.1

2020-09-03T00:00:00Z

3D models are more relevant than ever. Retailers bring in-store shopping experiences to customers' homes. Museums are making 3D models of their artifacts available to everyone on the web. Unfortunately, it can be difficult to add a 3D model to a website in a way that provides a great user experience without a deep knowledge of 3D technologies or resorting to hosting 3D content on a third-party site. The <model-viewer> web component, introduced in early 2019, seeks to make putting 3D models on the web as easy as writing a few lines of HTML. Since then, the team has been working to address feedback and requests from the community. The culmination of that work was <model-viewer> version 1.0, released earlier this year. We're now announcing the release of <model-viewer> 1.1. You can read the release notes in GitHub.

What's new since last year? #

Version 1.1 includes built-in support for augmented reality (AR) on the web, improvements to speed and fidelity, and other frequently-requested features.

Augmented reality #

Viewing a 3D model on a blank canvas is great, but being able to view it in your space is even better. For an entirely-within-the-browser 3D and AR Chrome Android supports augmented reality using WebXR .

A demonstration of the <model-viewer> AR capability.

When it's ready, you'll be able to use it by add an ar attribute to the <model-viewer> tag. Other attributes allow you to customize the WebXR AR experience, as shown in the WebXR sample on modelviewer.dev. The code sample below shows what this might look like.

<model-viewer src="Chair.glb"
  ar ar-scale="auto"
  camera-controls
  alt="A 3D model of an office chair.">
</model-viewer>

It looks something like the embedded video shown under this heading.

Camera controls #

<model-viewer> now gives full control over the view's virtual camera (the perspective of the viewer). This includes the camera target, orbit (position relative to the model), and field of view. You can also enable auto-rotation and set limits on user interaction (e.g. maximum and minimum fields of view).

Annotations #

You can also annotate your models using HTML and CSS. This capability is often used to "attach" labels to parts of the model in a way that moves with the model as it's manipulated. The annotations are customizable, including their appearance and the extent to which they're hidden by the model. Annotations also work in AR.

<style>
  button{
    display: block;
    width: 6px;
    height: 6px;
    border-radius: 3px;
    border: 3px solid blue;
    background-color: blue;
    box-sizing: border-box;
  }

  #annotation{
    background-color: #dddddd;
    position: absolute;
    transform: translate(10px, 10px);
    border-radius: 10px;
    padding: 10px;
  }
</style>
<model-viewer src="https://modelviewer.dev/assets/ShopifyModels/ToyTrain.glb" alt="A 3D model of a Toy Train" camera-controls>
  <button slot="hotspot-hand" data-position="-0.023 0.0594 0.0714" data-normal="-0.3792 0.0004 0.9253">
    <div id="annotation">Whistle</div>
  </button>
</model-viewer>

A space suit with an annotation.

See the annotations documentation page for more information.

Editor #

Version 1.1 introduces and hosts a <model-viewer> "editing" tool, which enables you to quickly preview your model, try out different <model-viewer> configurations (e.g. exposure and shadow softness), generate a poster image, and interactively get coordinates for annotations.

Rendering and performance improvements #

Rendering fidelity is greatly improved, especially for high dynamic range (HDR) environments. <model-viewer> now also uses a direct render path when only one <model-viewer> element is in the viewport, which increases performance (especially on Firefox). Lastly, dynamically scaling the render resolution improved frame rate dramatically. The example below shows off some of these recent improvements.

<model-viewer camera-controls
              skybox-image="spruit_sunrise_1k_HDR.hdr"
              alt="A 3D model of a well-worn  helmet"
              src="DamagedHelmet.glb"></model-viewer>

A 3D model of a well-worn helmet.

Stability #

With <model-viewer> reaching its first major version, API stability is a priority, so breaking changes will be avoided until version 2.0 is released.

What's next? #

<model-viewer> version 1.0 includes the most-requested capabilities, but the team is not done yet. More features will be added, as will improvements in performance, stability, documentation, and tooling. If you have suggestions, file an issue in GitHub; also, PRs are always welcome. You can stay connected by following <model-viewer> on Twitter and checking out the community chat on Spectrum.

Media accessibility

2020-08-20T00:00:00Z

In this article you will learn about the WebVTT (Web Video Text Tracks) format, which is used to describe timed text data such as closed captions or subtitles in order to make videos more accessible to your audience.

Accessibility isn't like icing on a cake. It's never anything you put on a backlog with the hope of introducing it later. Captions and screen reader descriptions are the only way many users can experience your videos, and in some jurisdictions, they're even required by law or regulation.

Add `<track>` tags #

To add captions or screen reader descriptions to a web video, add a <track> tag within a <video> tag. In addition to captions and screen reader descriptions, <track> tags may also be used for subtitles and chapter titles. The <track> tag can also help search engines understand what's in a video. However, those capabilities are outside the scope of this article.

Screenshot showing captions displayed using the track element in Chrome on Android

The <track> tag is similar to the <source> element in that both have a src attribute that points to referenced content. For a <track> tag, it points to a WebVTT file. The label attribute specifies how a particular track will be identified in the interface.

To provide tracks for multiple languages add a separate <track> tag for each WebVTT file you're providing and indicate the language using the srclang attribute.

An example <video> tag with two <track> tags is shown below. There's also a sample you can view on Glitch (source).

Add a <track> element as a child of the <video> element:

<video controls>
  <source src="https://storage.googleapis.com/webfundamentals-assets/videos/chrome.webm" type="video/webm" />
  <source src="https://storage.googleapis.com/webfundamentals-assets/videos/chrome.mp4" type="video/mp4" />
  <track src="chrome-subtitles-en.vtt" label="English captions" kind="captions" srclang="en" default>
  <track src="chrome-subtitles-zh.vtt" label="中文字幕" kind="captions" srclang="zh">
  <p>This browser does not support the video element.</p>
</video>

WebVTT file structure #

Below is a hypothetical WebVTT file for the demo linked to above. The file is a text file containing a series of cues. Each cue is a block of text to display on screen, and the time range during which it will be displayed.

WEBVTT

00:00.000 --> 00:04.999
Man sitting on a tree branch, using a laptop.

00:05.000 --> 00:08.000
The branch breaks, and he starts to fall.

...

Each item within the track file is called a cue. Each cue has a start time and end time separated by an arrow, with cue text in the line below. Cues can optionally also have IDs like railroad and manuscript in the example below. Cues are separated by an empty line.

WEBVTT

railroad
00:00:10.000 --> 00:00:12.500
Left uninspired by the crust of railroad earth

manuscript
00:00:13.200 --> 00:00:16.900
that touched the lead to the pages of your manuscript.

Cue times are in hours:minutes:seconds.milliseconds format. Parsing is strict. Meaning, numbers must be zero padded if necessary: hours, minutes, and seconds must have two digits (00 for a zero value) and milliseconds must have three digits (000 for a zero value). There is an excellent WebVTT validator at Live WebVTT Validator, which checks for errors in time formatting, and problems such as non-sequential times.

You can create a VTT file by hand, thought there are many services that will create them for you.

As you can see in our previous examples, the WebVTT format is pretty simple. Just add your text data along with timings.

However, what if you want your captions to render in a different position with left or right alignment? Perhaps to align the captions with the current speaker position, or to stay out of the way of in-camera text. WebVTT defines settings to do that, and more, directly inside the .vtt file. Take note of how the caption placement is defined by adding settings after the time interval definitions.

WEBVTT

00:00:05.000 --> 00:00:10.000 line:0 position:20% size:60% align:start
The first line of the subtitles.

Another handy feature is the ability to style cues using CSS. Perhaps you want to use a gray linear gradient as the background, with a foreground color of papayawhip for all captions and all bold text colored peachpuff.

video::cue {
  background-image: linear-gradient(to bottom, dimgray, lightgray);
  color: papayawhip;
}

video::cue(b) {
  color: peachpuff;
}

If you're interested in learning more about styling and tagging of individual cues, the WebVTT specification is a good source for advanced examples.

Kinds of text tracks #

Did you notice the kind attribute of the <track> element? It's used to indicate what relation the particular text track has to the video. The possible values of the kind attribute are:

captions: For closed captions from transcripts and possibly translations of any audio. Suitable for hearing impaired and in cases when the video is playing muted.
subtitles: For subtitles, that is, translations of speech and text in a language different from the main language of the video.
descriptions: For descriptions of visual parts of the video content. Suitable for visually impaired people.
chapters: Intended to be displayed when the user is navigating within the video.
metadata: Not visible, and may be used by scripts.

Now that you understand the basics of making a video available and accessible on your web page, you might wonder about more complex use cases. Next up, you'll lean about Media frameworks and how they can help you add videos to your web page while providing advanced features.

Add media to a web page

2020-06-19T00:00:00Z

In this section you'll learn how to embed a media file in a web page using the <video> and <source> tags, how to add captions to a video for the hearing-impaired, and some general concepts for using media frameworks. Additionally, you'll learn about using preload to speed up playback start, and we will discuss how we built an updated demo PWA with offline adaptive streaming capabilities called Kino.

This section assumes you have a video file that is ready for embedding in a web page. A .mov file downloaded from a camera will not work. If that's all you have, see Prepare media files for the web then come back.

This section covers these topics.

In The <video> and <source> tags you'll learn specifically how to embed a media file in a web page.
In Media accessibility you'll learn to add captions to a media file for hearing impaired.
In Media frameworks you'll learn basics about using media frameworks like Shaka Player, JW Player, and Video.js.
In Fast playback with audio and video preload you'll learn to accelerate your media playback by actively preloading resources.
In PWA with offline streaming you'll learn how we built an updated demo PWA that is capable of adaptive streaming and offline playback without using frameworks. Plus you can play with source code.

There is a lot of ground to cover in this section, get started by learning how to use the <video> and <source> tags.

Media file basics

2020-06-19T00:00:00Z

In this article you will learn about media file basics such as the concepts of a container, and a few of the many available codec formats that you can use in a stream. Plus lightly touch on topics such as adaptive streaming, bitrate, and resolution—but we'll dive deeper into all these in later sections.

Serving video files #

You might think that you can take a raw file from a video camera and just upload it to the web as-is. Indeed, video streaming sites such as YouTube or Vimeo let you do just that, and even provide live-streaming capabilities—typically by connecting to your camera's HDMI port and then processing it through a capture card. These services greatly simplify video processing and uploading, which includes generating the many files and manifests needed for adaptive streaming and various resolutions. Plus the many other complicated and nuanced requirements that make self-hosting a bit of a chore. Preparing and serving a video from your own site, and likely a separate media server, is a bit more complicated than just uploading a raw camera file if you care how your users experience your site.

Video files come in a variety of formats. The format that comes off your camera is typically a .mov file, or an .mp4 if you have a good modern mirrorless camera. However, while a .mov is good for recording and for editing and other early post-production processes, the file size means it's not good for streaming over the web. As well, the file size of a raw .mp4 in 4K is going to make playing that file on mobile very prohibitive. Because browsers support different file formats, you'll need to create multiple optimized files and potentially a manifest if you plan to support adaptive streaming. Before converting files you need to understand a few basics about them and about their characteristics.

Containers and codecs, and streams? #

The file that you see in your operating system shell is a container, identified by a file extension (.mp4, .webm, .ogg etc.). The container houses one or more streams. A media file can have any number of streams, of many more formats than we'll go into here.

The sample files used later in this section contain at most two streams: an audio stream and a video stream. Among the other types you might encounter are captions and data, both of which are beyond the scope of this article. There are instances where audio and video streams are dealt with separately. Most files you'll encounter will only contain a single audio stream and a single video stream.

Within the audio and video streams, the actual data is compressed using a codec. A codec, or coder/decoder, is a compression format for video or audio data. The distinction between a container and a codec is important because files with the same container can have their contents encoded with different codecs.

The image below illustrates this structure. On the left is the basic container structure with two streams. On the right are the specifics of that structure for a single WebM file.

Parts of a media file.

Files in WebM containers can be orders of magnitude smaller than other formats, making them a good choice for mobile sites to stream. Unfortunately, not all browsers support up-to-date containers and codecs. For example, WebM was created specifically for the web as a high-quality and open source option, but its support is not yet universal. Safari in particular does not, according to Can I use at the time of this writing, support WebM for embedded video. However, WebM does have partial support with the VP8 and VP9 codec used in WebRTC. So your best option is to provide a fallback video.

Codec formats #

Many file types support multiple codecs within the same container. A complete list of available video codecs and audio codecs would be a whole website to itself. The links just provided are for MDN's practical lists of what's usable on the web. Listed below are the currently preferred file types, and the codecs they might use. Follow the file type links to view the browsers that support them.

File type	Video Codec	Audio Codec
MP4	AV1, AVC (H.264)*, VP9	AAC
WebM	AV1, VP9*	Vorbis, Opus

* Indicates the preferred video codec.

Bitrate and resolution #

Bitrate is the maximum number of bits used to encode one second of a stream. The more bits used to encode a second of stream, the higher the potential detail and fidelity. We provide more information about this concept in Bitrate.

Resolution is the amount of information in a single frame of video, given as the number of logical pixels in each dimension. We provide more information about this concept in Resolution.

Up next, in Media application basics, we'll show you how to examine these characteristics using two command line tools: Shaka Packager and FFmpeg.

Positioning virtual objects in real-world views

2020-02-13T00:00:00Z

The WebXR Device API shipped last fall in Chrome 79. As stated then, Chrome's implementation of the API is a work in progress. Chrome is happy to announce that some of the work is finished. In Chrome 81, two new features have arrived:

This article covers the WebXR Hit Test API, a means of placing virtual objects in a real-world camera view.

In this article I assume you already know how to create an augmented reality session and that you know how to run a frame loop. If you're not familiar with these concepts, you should read the earlier articles in this series.

The immersive AR session sample #

The code in this article is based on, but not identical to, that found in the Immersive Web Working Group's Hit Test sample (demo, source). This example lets you place virtual sunflowers on surfaces in the real world.

When you first open the app, you'll see a blue circle with a dot in the middle. The dot is the intersection between an imaginary line from your device to the point in the environment. It moves as you move the device. As it finds intersection points, it appears to snap to surfaces such as floors, table tops, and walls. It does this because hit testing provides the position and orientation of the intersection point, but nothing about the surfaces themselves.

This circle is called a reticle, which is a temporary image that aids in placing an object in augmented reality. If you tap the screen, a sunflower is placed on the surface at the reticle location and orientation of the reticle point, regardless of where you tapped the screen. The reticle continues to move with your device.

The reticle is a temporary image that aids in placing an object in augmented reality.

Create the reticle #

You must create the reticle image yourself since it is not provided by the browser or the API. The method of loading and drawing it is framework specific. If you're not drawing it directly using WebGL or WebGL2 consult your framework documentation. For this reason, I won't go into detail about how the reticle is drawn in the sample. Below I show one line of it for one reason only: so that in later code samples, you'll know what I'm referring to when I use the reticle variable.

let reticle = new Gltf2Node({url: 'media/gltf/reticle/reticle.gltf'});

Request a session #

When requesting a session, you must request 'hit-test' in the requiredFeatures array as shown below.

navigator.xr.requestSession('immersive-ar', {
  requiredFeatures: ['local', 'hit-test']
})
.then((session) => {
  // Do something with the session
});

Entering a session #

In previous articles I've presented code for entering an XR session. I've shown a version of this below with some additions. First I've added the select event listener. When the user taps the screen, a flower will be placed in the camera view based on the pose of the reticle. I'll describe that event listener later.

function onSessionStarted(xrSession) {
  xrSession.addEventListener('end', onSessionEnded);
  xrSession.addEventListener('select', onSelect);

  let canvas = document.createElement('canvas');
  gl = canvas.getContext('webgl', { xrCompatible: true });

  xrSession.updateRenderState({
    baseLayer: new XRWebGLLayer(session, gl)
  });

  xrSession.requestReferenceSpace('viewer').then((refSpace) => {
    xrViewerSpace = refSpace;
    xrSession.requestHitTestSource({ space: xrViewerSpace })
    .then((hitTestSource) => {
      xrHitTestSource = hitTestSource;
    });
  });

  xrSession.requestReferenceSpace('local').then((refSpace) => {
    xrRefSpace = refSpace;
    xrSession.requestAnimationFrame(onXRFrame);
  });
}

Multiple reference spaces #

Notice that the highlighted code calls XRSession.requestReferenceSpace() twice. I initially found this confusing. I asked why does the hit test code not request an animation frame (starting the frame loop) and why does the frame loop seem to not involve hit tests. The source of the confusion was a misunderstanding of reference spaces. Reference spaces express relationships between an origin and the world.

To understand what this code is doing, pretend that you're viewing this sample using a standalone rig, and you have both a headset and a controller. To measure distances from the controller, you would use a controller-centered frame of reference. But to draw something to the screen you would use user-centered coordinates.

In this sample, the viewer and the controller are the same device. But I have a problem. What I draw must be stable with regard to the environment, but the 'controller' I'm drawing with is moving.

For image drawing, I use the local reference space, which gives me stability in terms of the environment. After getting this I start the frame loop by calling requestAnimationFrame().

For hit testing, I use the viewer reference space, which is based on the device's pose at the time of the hit test. The label 'viewer' is somewhat confusing in this context because I'm talking about a controller. It makes sense if you think of the controller as an electronic viewer. After getting this, I call xrSession.requestHitTestSource(), which creates the source of hit test data that I'll use when drawing.

Running a frame loop #

The requestAnimationFrame() callback also gets new code to handle hit testing.

As you move your device, the reticle needs to move with it as it tries to find surfaces. To create the illusion of movement, redraw the reticle in every frame. But don't show the reticle if the hit test fails. So, for the reticle I created earlier, I set it's visible property to false.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);

  reticle.visible = false;

  // Reminder: the hitTestSource was acquired during onSessionStart()
  if (xrHitTestSource && xrViewerPose) {
    let hitTestResults = xrFrame.getHitTestResults(xrHitTestSource);
    if (hitTestResults.length > 0) {
      let pose = hitTestResults[0].getPose(xrRefSpace);
      reticle.visible = true;
      reticle.matrix = pose.transform.matrix;
    }
  }

  // Draw to the screen
}

To draw anything in AR, I need to know where the viewer is and where they're looking. So I test that hitTestSource and the xrViewerPose are still valid.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);

  reticle.visible = false;

  // Reminder: the hitTestSource was acquired during onSessionStart()
  if (xrHitTestSource && xrViewerPose) {
    let hitTestResults = xrFrame.getHitTestResults(xrHitTestSource);
    if (hitTestResults.length > 0) {
      let pose = hitTestResults[0].getPose(xrRefSpace);
      reticle.visible = true;
      reticle.matrix = pose.transform.matrix;
    }
  }

  // Draw to the screen
}

Now I call getHitTestResults(). It takes the hitTestSource as an argument and returns an array of HitTestResult instances. The hit test may find multiple surfaces. The first one in the array is the one closest to the camera. Most of the time you will use it, but an array is returned for advanced use cases. For example, imagine your camera is pointed at a box on a table on a floor. It's possible that the hit test will return all three surfaces in the array. In most cases, it will be the box that I care about. If the length of the returned array is 0, in other words, if no hit test is returned, continue onward. Try again in the next frame.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);

  reticle.visible = false;

  // Reminder: the hitTestSource was acquired during onSessionStart()
  if (xrHitTestSource && xrViewerPose) {
    let hitTestResults = xrFrame.getHitTestResults(xrHitTestSource);
    if (hitTestResults.length > 0) {
      let pose = hitTestResults[0].getPose(xrRefSpace);
      reticle.visible = true;
      reticle.matrix = pose.transform.matrix;
    }
  }

  // Draw to the screen
}

Finally, I need to process the hit test results. The basic process is this. Get a pose from the hit test result, transform (move) the reticle image to the hit test position, then set its visible property to true. The pose represents the pose of a point on a surface.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);

  reticle.visible = false;

  // Reminder: the hitTestSource was acquired during onSessionStart()
  if (xrHitTestSource && xrViewerPose) {
    let hitTestResults = xrFrame.getHitTestResults(xrHitTestSource);
    if (hitTestResults.length > 0) {
      let pose = hitTestResults[0].getPose(xrRefSpace);
      reticle.matrix = pose.transform.matrix;
      reticle.visible = true;

    }
  }

  // Draw to the screen
}

Placing an object #

An object is placed in AR when the user taps the screen. I already added a select event handler to the session. (See above.)

The important thing in this step is knowing where to place it. Since the moving reticle gives you a constant source of hit tests, the simplest way to place an object is to draw it at the location of the reticle at the last hit test.

function onSelect(event) {
  if (reticle.visible) {
    // The reticle should already be positioned at the latest hit point,
    // so we can just use its matrix to save an unnecessary call to
    // event.frame.getHitTestResults.
    addARObjectAt(reticle.matrix);
  }
}

Conclusion #

The best way to get a handle on this is to step through the sample code or try out the codelab. I hope I've given you enough background to make sense of both.

We're not done building immersive web APIs, not by a long shot. We'll publish new articles here as we make progress.

Photo by Daniel Frank on Unsplash

Virtual reality comes to the web, part II

2020-02-13T00:00:00Z

Recently, I published Virtual reality comes to the web, an article that introduced basic concepts behind the WebXR Device API. I also provided instructions for requesting, entering, and ending an XR session.

This article describes the frame loop, which is a user-agent controlled infinite loop in which content is repeatedly drawn to the screen. Content is drawn in discrete blocks called frames. The succession of frames creates the illusion of movement.

What this article is not #

WebGL and WebGL2 are the only means of rendering content during a frame loop in a WebXR App. Fortunately many frameworks provide a layer of abstraction on top of WebGL and WebGL2. Such frameworks include three.js, babylonjs, and PlayCanvas, while A-Frame and React 360 was designed for interacting with WebXR.

This article is neither a WebGL nor a framework tutorial. It explains basics of a frame loop using the Immersive Web Working Group's Immersive VR Session sample (demo, source). If you want to dive into WebGL or one of the frameworks, the internet provides a growing list of articles.

The players and the game #

When trying to understand the frame loop, I kept getting lost in the details. There's a lot of objects in play, and some of them are only named by reference properties on other objects. To help you keep it straight, I'll describe the objects, which I'm calling 'players'. Then I'll describe how they interact, which I'm calling 'the game'.

The players #

XRViewerPose #

A pose is the position and orientation of something in 3D space. Both viewers and input devices have a pose, but it's the viewer's pose we're concerned with here. Both viewer and input device poses have a transform attribute describing its position as a vector and its orientation as a quaternion relative to the origin. The origin is specified based on the requested reference space type when calling XRSession.requestReferenceSpace().

Reference spaces take a bit to explain. I cover them in depth in Augmented reality. The sample I'm using as the basis for this article uses a 'local' reference space which means the origin is at the viewer's position at the time of session creation without a well-defined floor, and its precise position may vary by platform.

XRView #

A view corresponds to a camera viewing the virtual scene. A view also has a transform attribute describing it's position as a vector and its orientation. These are provided both as a vector/quaternion pair and as an equivalent matrix, you can use either representation depending on which best fits your code. Each view corresponds to a display or a portion of a display used by a device to present imagery to the viewer. XRView objects are returned in an array from the XRViewerPose object. The number of views in the array varies. On mobile devices an AR scene has one view, which may or may not cover the device screen. Headsets typically have two views, one for each eye.

XRWebGLLayer #

Layers provide a source of bitmap images and descriptions of how those images are to be rendered in the device. This description doesn't quite capture what this player does. I've come to think of it as a middleman between a device and a WebGLRenderingContext. MDN takes much the same view, stating that it 'provides a linkage' between the two. As such, it provides access to the other players.

In general, WebGL objects store state information for rendering 2D and 3D graphics.

WebGLFramebuffer #

A framebuffer provides image data to the WebGLRenderingContext. After retrieving it from the XRWebGLLayer, you simply pass it to the current WebGLRenderingContext. Other than calling bindFramebuffer() (more about that later) you will never access this object directly. You will merely pass it from the XRWebGLLayer to the WebGLRenderingContext.

XRViewport #

A viewport provides the coordinates and dimensions of a rectangular region in the WebGLFramebuffer.

WebGLRenderingContext #

A rendering context is a programmatic access point for a canvas (the space we're drawing on). To do this, it needs both a WebGLFramebuffer and an XRViewport.

Notice the relationship between XRWebGLLayer and WebGLRenderingContext. One corresponds to the viewer's device and the other corresponds to the web page. WebGLFramebuffer and XRViewport are passed from the former to the latter.

The relationship between XRWebGLLayer and WebGLRenderingContext

The game #

Now that we know who the players are, let's look at the game they play. It's a game that starts over with every frame. Recall that frames are part of a frame loop that happens at a rate that depends on the underlying hardware. For VR applications the frames per second can be anywhere from 60 to 144. AR for Android runs at 30 frames per second. Your code should not assume any particular frame rate.

The basic process for the frame loop looks like this:

Call XRSession.requestAnimationFrame(). In response, the user agent invokes the XRFrameRequestCallback, which is defined by you.
Inside your callback function:
1. Call XRSession.requestAnimationFrame() again.
2. Get the viewer's pose.
3. Pass ('bind') the WebGLFramebuffer from the XRWebGLLayer to the WebGLRenderingContext.
4. Iterate over each XRView object, retrieving its XRViewport from the XRWebGLLayer and passing it to the WebGLRenderingContext.
5. Draw something to the framebuffer.

Because steps 1 and 2a were covered in the previous article, I'll start at step 2b.

Get the viewer's pose #

It probably goes without saying. To draw anything in AR or VR, I need to know where the viewer is and where they're looking. The viewer's position and orientation are provided by an XRViewerPose object. I get the viewer's pose by calling XRFrame.getViewerPose() on the current animation frame. I pass it the reference space I acquired when I set up the session. The values returned by this object are always relative to the reference space I requested when I entered the current session. As you may recall, I have to pass the current reference space when requesting the pose.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);
  if (xrViewerPose) {
    // Render based on the pose.
  }
}

There's one viewer pose that represents the user's overall position, meaning either the viewer's head or the phone camera in the case of a smartphone. The pose tells your application where the viewer is. Actual image rendering uses XRView objects, which I'll get to in a bit.

Before moving on, I test whether the viewer pose was returned in case the system loses tracking or blocks the pose for privacy reasons. Tracking is the XR device's ability to know where it and/or it's input devices are relative to the environment. Tracking can be lost in several ways, and varies depending on the method used for tracking. For example, if cameras on the headset or phone are used for tracking the device may lose its ability to determine where it is in situations with low or no light, or if the cameras are covered.

An example of blocking the pose for privacy reasons is if the headset is showing a security dialog such as a permission prompt, the browser may stop providing poses to the application while this is happening. But I"ve already called XRSession.requestAnimationFrame() so that if the system can recover, the frame loop will continue. If not, the user agent will end the session and call the end event handler.

A short detour #

The next step requires objects created during session set-up. Recall that I created a canvas and instructed it to create an XR-compatible Web GL rendering context, which I got by calling canvas.getContext(). All drawing is done using the WebGL API, the WebGL2 API, or a WebGL-based framework such as Three.js. This context was passed to the session object via updateRenderState(), along with a new instance of XRWebGLLayer.

let canvas = document.createElement('canvas');
// The rendering context must be based on WebGL or WebGL2
let webGLRenContext = canvas.getContext('webgl', { xrCompatible: true });
xrSession.updateRenderState({
    baseLayer: new XRWebGLLayer(xrSession, webGLRenContext)
  });

Pass ('bind') the WebGLFramebuffer #

The XRWebGLLayer provides a framebuffer for the WebGLRenderingContext provided specifically for use with WebXR and replacing the rendering contexts default framebuffer. This is called 'binding' in the language of WebGL.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);
  if (xrViewerPose) {
    let glLayer = xrSession.renderState.baseLayer;
    webGLRenContext.bindFramebuffer(webGLRenContext.FRAMEBUFFER, glLayer.framebuffer);
    // Iterate over the views
  }
}

Iterate over each XRView object #

After getting the pose and binding the framebuffer, it's time to get the viewports. The XRViewerPose contains an array of XRView interfaces each of which represents a display or a portion of a display. They contain information that's needed to render content that's correctly positioned for the device and the viewer such as the field of view, eye offset, and other optical properties. Since I'm drawing for two eyes, I have two views, which I loop through and draw a separate image for each.

When implementing for phone-based augmented reality, I would have only one view but I'd still use a loop. Though it may seem pointless to iterate through one view, doing so allows you to have a single rendering path for a spectrum of immersive experiences. This is an important difference between WebXR and other immersive systems.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);
  if (xrViewerPose) {
    let glLayer = xrSession.renderState.baseLayer;
    webGLRenContext.bindFramebuffer(webGLRenContext.FRAMEBUFFER, glLayer.framebuffer);
    for (let xrView of xrViewerPose.views) {
      // Pass viewports to the context
    }
  }
}

Pass the XRViewport object to the WebGLRenderingContext #

An XRView object refers to what's observable on a screen. But to draw to that view I need coordinates and dimensions that are specific to my device. As with the framebuffer, I request them from the XRWebGLLayer and pass them to the WebGLRenderingContext.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);
  if (xrViewerPose) {
    let glLayer = xrSession.renderState.baseLayer;
    webGLRenContext.bindFramebuffer(webGLRenContext.FRAMEBUFFER, glLayer.framebuffer);
    for (let xrView of xrViewerPose.views) {
      let viewport = glLayer.getViewport(xrView);
      webGLRenContext.viewport(viewport.x, viewport.y, viewport.width, viewport.height);
      // Draw something to the framebuffer
    }
  }
}

The webGLRenContext #

In writing this article I had a debate with a few collegues over the naming of the webGLRenContext object. The sample scripts and most WebXR code simpley calls this variable gl. When I was working to understand the samples, I kept forgetting what gl referred to. I've called it webGLRenContext to remind you while your learning that this is an instance of WebGLRenderingContext.

The reason is that using gl allows method names to look like their counterparts in the OpenGL ES 2.0 API, used for creating VR in compiled languages. This fact is obvious if you've written VR apps using OpenGL, but confusing if you're completely new to this technology.

Draw something to the framebuffer #

If you're feeling really ambitious, you can use WebGL directly, but I don't recommend that. It's much simpler to use one of the frameworks listed at the top.

Conclusion #

This is not the end of WebXR updates or articles. You can find a reference for all of WebXR's interfaces and members at MDN. For upcoming enhancements to the interfaces themselves, follow individual features on Chrome Status.

Photo by JESHOOTS.COM on Unsplash

Augmented reality: You may already know it

2020-02-13T00:00:00Z

This article covers augmented reality. If you've already used the WebXR Device API, you'll be happy to know there's very little new to learn. Entering a WebXR session is largely the same. Running a frame loop is largely the same. The differences lie in configurations that allow content to be shown appropriately for augmented reality. If you're not familiar with the basic concepts of WebXR, you should read my earlier posts on the WebXR Device API, or at least be familiar with the topics covered therein. You should know how to request and enter a session and you should know how to run a frame loop.

For information on hit testing, see the companion article Positioning virtual objects in real-world views. The code in this article is based on the Immersive AR Session sample (demo source) from the Immersive Web Working Group's WebXR Device API samples.

Before diving into the code you should use the Immersive AR Session sample at least once. You'll need a modern Android phone with Chrome 81 or later.

What's it useful for? #

Augmented reality will be a valuable addition to a lot of existing or new web pages by allowing them to implement AR use cases without leaving the browser. For example, it can help people learn on education sites, and allow potential buyers to visualize objects in their home while shopping.

Consider the second use case. Imagine simulating placing a life-size representation of a virtual object in a real scene. Once placed, the image stays on the selected surface, appears the size it would be if the actual item were on that surface, and allows the user to move around it as well as closer to it or farther from it. This gives viewers a deeper understanding of the object than is possible with a two dimensional image.

I'm getting a little ahead of myself. To actually do what I've described, you need AR functionality and some means of detecting surfaces. This article covers the former. The accompanying article on the WebXR Hit Test API (linked to above) covers the latter.

Requesting a session #

Requesting a session is very much like what you've seen before. First find out if the session type you want is available on the current device by calling xr.isSessionSupported(). Instead of requesting 'immersive-vr' as before, request 'immersive-ar'.

if (navigator.xr) {
  const supported = await navigator.xr.isSessionSupported('immersive-ar');
  if (supported) {
    xrButton.addEventListener('click', onButtonClicked);
    xrButton.textContent = 'Enter AR';
    xrButton.enabled = supported; // supported is Boolean
  }
}

As before, this enables an 'Enter AR' button. When the user clicks it, call xr.requestSession(), also passing 'immersive-ar'.

let xrSession = null;
function onButtonClicked() {
  if (!xrSession) {
    navigator.xr.requestSession('immersive-ar')
    .then((session) => {
      xrSession = session;
      xrSession.isImmersive = true;
      xrButton.textContent = 'Exit AR';
      onSessionStarted(xrSession);
    });
  } else {
    xrSession.end();
  }
}

A convenience property #

You've probably noticed that I highlighted two lines in the last code sample. The XRSession object would seem to have a property called isImmersive. This is a convenience property I've created myself, and not part of the spec. I'll use it later to make decisions about what to show the viewer. Why isn't this property part of the API? Because your app may need to track this property differently so the spec authors decided to keep the API clean.

Entering a session #

Recall what onSessionStarted() looked like in my earlier article:

function onSessionStarted(xrSession) {
  xrSession.addEventListener('end', onSessionEnded);

  let canvas = document.createElement('canvas');
  gl = canvas.getContext('webgl', { xrCompatible: true });

  xrSession.updateRenderState({
    baseLayer: new XRWebGLLayer(session, gl)
  });

  xrSession.requestReferenceSpace('local-floor')
  .then((refSpace) => {
    xrRefSpace = refSpace;
    xrSession.requestAnimationFrame(onXRFrame);
  });
}

I need to add a few things to account for rendering augmented reality. Turn off the background First, I'm going to determine whether I need the background. This is the first place I'm going to use my convenience property.

function onSessionStarted(xrSession) {
  xrSession.addEventListener('end', onSessionEnded);

  if (session.isImmersive) {
    removeBackground();
  }

  let canvas = document.createElement('canvas');
  gl = canvas.getContext('webgl', { xrCompatible: true });

  xrSession.updateRenderState({
    baseLayer: new XRWebGLLayer(session, gl)
  });

  refSpaceType = xrSession.isImmersive ? 'local' : 'viewer';
  xrSession.requestReferenceSpace(refSpaceType).then((refSpace) => {
    xrSession.requestAnimationFrame(onXRFrame);
  });

}

Reference spaces #

My earlier articles skimmed over reference spaces. The sample I'm describing uses two of them, so it's time to correct that omission.

A reference space describes the relationship between the virtual world and the user's physical environment. It does this by:

Specifying the origin for the coordinate system used for expressing positions in the virtual world.
Specifying whether the user is expected to move within that coordinate system.
Whether that coordinate system has pre-established boundaries. (The examples shown here do not use coordinate systems with pre-established boundaries.)

For all reference spaces, the X coordinate expresses left and right, the Y expresses up and down, and Z expresses forward and backward. Positive values are right, up, and backward, respectively.

The coordinates returned by XRFrame.getViewerPose() depend on the requested reference space type. More about that when we get to the frame loop. Right now we need to select a reference type that's appropriate for augmented reality. Again, this uses my convenience property.

let refSpaceType
function onSessionStarted(xrSession) {
  xrSession.addEventListener('end', onSessionEnded);

  if (session.isImmersive) {
    removeBackground();
  }

  let canvas = document.createElement('canvas');
  gl = canvas.getContext('webgl', { xrCompatible: true });

  xrSession.updateRenderState({
    baseLayer: new XRWebGLLayer(session, gl)
  });

  refSpaceType = xrSession.isImmersive ? 'local' : 'viewer';
  xrSession.requestReferenceSpace(refSpaceType).then((refSpace) => {
    xrSession.requestAnimationFrame(onXRFrame);
  });
}

If you've visited the Immersive AR Session Sample you'll notice that initially the scene is static and not at all augmented reality. You can drag and swipe with your finger to move around the scene. If you click "START AR", the background drops out and you can move around the scene by moving the device. The modes use different reference space types. The highlighted text above shows how this is selected. It uses the following reference types:

local - The origin is at the viewer's position at the time of session creation. This means the experience doesn't necessarily have a well-defined floor and the exact position of the origin may vary by platform. Though there are no pre-established boundaries to the space, it's expected that content can be viewed with no movement other than rotation. As you can see from our own AR example, some movement within the space may be possible.

viewer - Used most frequently for content presented inline in the page, this space follows the viewing device. When passed to getViewerPose it provides no tracking, and thus always reports a pose at the origin unless the application modifies it with XRReferenceSpace.getOffsetReferenceSpace(). The sample uses this to enable touch-based panning of the camera.

Running a frame loop #

Conceptually, nothing changes from what I did in the VR session described in my earlier articles. Pass the reference space type to XRFrame.getViewerPose(). The returned XRViewerPose will be for the current reference space type. Using viewer as the default allows a page to show content previews before user consent is requested for AR or VR. This illustrates an important point: the inline content uses the same frame loop as the immersive content, cutting down the amount of code that needs to be maintained.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  let xrViewerPose = xrFrame.getViewerPose(refSpaceType);
  if (xrViewerPose) {
    // Render based on the pose.
  }
}

Conclusion #

This series of articles only covers the basics of implementing immersive content on the web. Many more capabilities and use cases are presented by the Immersive Web Working Group's WebXR Device API samples. We've also just published a hit test article which explains an API for detecting surfaces and placing virtual items in a real-world camera view. Check them out and watch The web.dev blog for more articles in the year to come.

Photo by David Grandmougin on Unsplash

Integrate PWAs into built-in sharing UIs with Workbox

2019-12-19T00:00:00Z

The Web Share Target API lets you display your Progressive Web App in a user's system-level share sheet after it's been installed. While it works great if you have a server available to receive the request, it's much harder to get working if you don't.

In this article we'll use Workbox, a set of JavaScript libraries for adding offline support to web apps, to create a share target URL that lives entirely inside your service worker. This lets static sites and single-page apps serve as share targets without a dedicated server endpoint.

System-level share target picker with an installed PWA called Share Target Test as an option.

On the same page #

If you're unfamiliar with how Web Share Target Works, Receiving shared data with the Web Share Target API gives you an in-depth introduction. Here's a quick review.

There are two parts to implementing web share target functionality. First, update your web app manifest to indicate that you want your app to be a share target when installed. The following example directs shares to the /share url via a POST request. It is encoded as a multipart form, with title being called name, text being called description, and JPEG images being called photos.

…
"share_target": {
  "action": "/share",
  "method": "POST",
  "enctype": "multipart/form-data",
  "params": {
    "title": "name",
    "text": "description",
    "files": [
      {
        "name": "photos",
        "accept": ["image/jpeg", ".jpg"]
      }
    ]
  }
}
…

While normally handled by a server endpoint, a neat trick you can do for a share target is to register a route directly in your service worker to handle the request. This will let your app be a share target without a backend.

You do this in Workbox by registering a route that's handled by your service worker. Start by importing registerRoute from 'workbox-routing'. Notice that it's registered for the /share route, the same one listed in the example web app manifest. In response it calls shareTargetHandler().

import { registerRoute } from 'workbox-routing';

registerRoute(
  '/share',
  shareTargetHandler,
  'POST'
);

The shareTargetHandler() function is asynchronous and takes the event, awaits the form data, then retrieves the media files from that.

async function shareTargetHandler ({event}) {
  const formData = await event.request.formData();
  const mediaFiles = formData.getAll('media');

  for (const mediaFile of mediaFiles) {
    // Do something with mediaFile
    // Maybe cache it or post it back to a server
  });

  // Do something with the rest of formData as you need
  // Maybe save it to IndexedDB
};

You can then do whatever you'd like with these files. You can cache them. You can send them somewhere with a fetch request. You can even use the other manifest options, maybe serving a page with some query parameters for the other shared items or storing the data and pointers to the media in the Cache Storage API or IndexedDB.

You can try it out on the sample app Fugu Journal and see its service worker implementation in its source code.

One common thing you might do is hold shared resources until better network connections are available. Workbox also supports periodic background sync.

Conclusion #

The Share Target API is a simple way to deeply integrate your Progressive Web App into user's devices, putting them on-par with platform-specific applications for the critical task of sharing content between apps. But doing so usually requires a server available to receive the request. By leveraging Workbox to create a share target route directly in your service worker, your app is free of this constraint, allowing Share Target to work for apps while offline and without backends.

Photo by Elaine Casap on Unsplash

Improving page dismissal in synchronous XMLHttpRequest()

2019-12-18T00:00:00Z

It's common for a page or app to have unsubmitted analytics or other data at the time a user closes it. To prevent data loss, some sites use a synchronous call to XMLHttpRequest() to keep the page or app open until its data is passed to the server. Not only are there better ways to save data, but this technique creates a bad user experience by delaying closing of the page for up to several seconds.

This practice needs to change, and browsers are responding. The XMLHttpRequest() specification is already slated for deprecation and removal. Chrome 80 takes the first step by disallowing synchronous calls inside several event handlers, specifically beforeunload, unload, pagehide, and visibilitychange when they are fired in the dismissal. WebKit also recently landed a commit implementing the same behavior change.

In this article I'll briefly describe options for those who need time to update their sites and outline the alternatives to XMLHttpRequest().

Temporary opt-outs #

Chrome does not simply want to pull the plug on XMLHttpRequest(), which is why a few temporary opt-out options are available. For sites on the internet, an origin trial is available. With this, you add an origin-specific token to your page headers that enables synchronous XMLHttpRequest() calls. This option ends shortly before Chrome 89 ships, sometime in March 2021. Enterprise Chrome customers can also use the AllowSyncXHRInPageDismissal policy flag, which ends at the same time.

Alternatives #

Regardless of how you send data back to the server, it's best to avoid waiting until page unload to send all the data at once. Aside from creating a bad user experience, unload is unreliable on modern browsers and risks data loss if something goes wrong. Specifically, unload events often don't fire on mobile browsers because there are many ways to close a tab or browser on mobile operating systems without the unload event firing. With XMLHttpRequest(), using small payloads was a choice. Now it's a requirement. Both of its alternatives have an upload limit of 64 KB per context, as required by the specification.

Fetch keepalive #

The Fetch API provides a robust means of dealing with server interactions and a consistent interface for use across different platform APIs. Among its options is keepalive, which ensures that a request continues whether or not the page that made it stays open:

window.addEventListener('unload', {
  fetch('/siteAnalytics', {
    method: 'POST',
    body: getStatistics(),
    keepalive: true
  });
}

The fetch() method has the advantage of greater control over what's sent to the server. What I don't show in the example is that fetch() also returns a promise that resolves with a Response object. Since I'm trying to get out of the way of the page's unloading, I chose not to do anything with it.

SendBeacon() #

SendBeacon() actually uses the Fetch API under the hood, which is why it has the same 64 KB payload limitation and why it also ensures that a request continues after a page unload. Its primary advantage is its simplicity. It lets you submit your data with a single line of code:

window.addEventListener('unload', {
  navigator.sendBeacon('/siteAnalytics', getStatistics());
}

Conclusion #

With the increased availability of fetch() across browsers, XMLHttpRequest() will hopefully be removed from the web platform at some point. Browser vendors agree it should be removed, but it will take time. Deprecating one of its worst use cases is a first step that improves the user experience for everyone.

Photo by Matthew Hamilton on Unsplash

Integrate with the OS sharing UI with the Web Share API

2019-11-08T00:00:00Z

With the Web Share API, web apps are able to use the same system-provided share capabilities as platform-specific apps. The Web Share API makes it possible for web apps to share links, text, and files to other apps installed on the device in the same way as platform-specific apps.

System-level share target picker with an installed PWA as an option.

Capabilities and limitations #

Web share has the following capabilities and limitations:

It can only be used on a site that is accessed via HTTPS.
If the share happens in a third-party iframe, the allow attribute must be used.
It must be invoked in response to a user action such as a click. Invoking it through the onload handler is impossible.
It can share URLs, text, or files.

Browser support

Chrome 89, Supported 89
Firefox 71, Behind a flag
Edge 93, Supported 93
Safari 12.1, Supported 12.1

Source

To share links and text, use the share() method, which is a promise-based method with a required properties object. To keep the browser from throwing a TypeError, the object must contain at least one of the following properties: title, text, url or files. You can, for example, share text without a URL or vice versa. Allowing all three members expands the flexibility of use cases. Imagine if after running the code below, the user chose an email application as the target. The title parameter might become the email subject, the text, the message body, and the files, the attachments.

if (navigator.share) {
  navigator.share({
    title: 'web.dev',
    text: 'Check out web.dev.',
    url: 'https://web.dev/',
  })
    .then(() => console.log('Successful share'))
    .catch((error) => console.log('Error sharing', error));
}

If your site has multiple URLs for the same content, share the page's canonical URL instead of the current URL. Instead of sharing document.location.href, you would check for a canonical URL <meta> tag in the page's <head> and share that. This will provide a better experience to the user. Not only does it avoid redirects, but it also ensures that a shared URL serves the correct user experience for a particular client. For example, if a friend shares a mobile URL and you look at it on a desktop computer, you should see a desktop version:

let url = document.location.href;
const canonicalElement = document.querySelector('link[rel=canonical]');
if (canonicalElement !== null) {
    url = canonicalElement.href;
}
navigator.share({url});

To share files, first test for and call navigator.canShare(). Then include an array of files in the call to navigator.share():

if (navigator.canShare && navigator.canShare({ files: filesArray })) {
  navigator.share({
    files: filesArray,
    title: 'Vacation Pictures',
    text: 'Photos from September 27 to October 14.',
  })
  .then(() => console.log('Share was successful.'))
  .catch((error) => console.log('Sharing failed', error));
} else {
  console.log(`Your system doesn't support sharing files.`);
}

Notice that the sample handles feature detection by testing for navigator.canShare() rather than for navigator.share(). The data object passed to canShare() only supports the files property. Certain types of audio, image, pdf, video, and text files can be shared. See Permitted File Extensions in Chromium for a complete list. More file types may be added in the future.

To trigger the share action from within a third-party iframe, embed the iframe with the allow attribute with a value of web-share:

<!-- On https://example.com/index.html -->
<iframe allow="web-share" src="https://third-party.example.com/iframe.html"></iframe>

You can see this in action in a demo on Glitch and view the source code. Failing to provide the attribute will result in a NotAllowedError with the message Failed to execute 'share' on 'Navigator': Permission denied.

Santa Tracker case study #

Santa Tracker share button.

Santa Tracker, an open-source project, is a holiday tradition at Google. Every December, you can celebrate the season with games and educational experiences.

In 2016, the Santa Tracker team used the Web Share API on Android. This API was a perfect fit for mobile. In previous years, the team removed share buttons on mobile because space is at a premium, and they couldn't justify having several share targets.

But with the Web Share API, they were able to present one button, saving precious pixels. They also found that users shared with Web Share around 20% more than users without the API enabled. Head to Santa Tracker to see Web Share in action.

Browser support #

Browser support for the Web Share API is nuanced, and it's recommended that you use feature detection (as described in the earlier code samples) instead of assuming that a particular method is supported.

Here's a rough outline of support for this feature. For detailed information, follow either of the support links.

navigator.canShare()

Browser support

Chrome 89, Supported 89
Firefox 96, Behind a flag
Edge 93, Supported 93
Safari 14, Supported 14

Source

navigator.share()

Browser support

Chrome 89, Supported 89
Firefox 71, Behind a flag
Edge 93, Supported 93
Safari 12.1, Supported 12.1

Source

Show support for the API #

Are you planning to use the Web Share API? Your public support helps the Chromium team prioritize features and shows other browser vendors how critical it is to support them.

Send a tweet to @ChromiumDev using the hashtag #WebShare and let us know where and how you're using it.

Helpful Links #

Virtual reality comes to the web

2019-10-31T00:00:00Z

Immersive experiences came to the web in Chrome 79. The WebXR Device API brings virtual reality brought virtual reality, while support for augmented reality arrives in Chrome 81. While an update to the GamePad API extends the advanced use of controlls to VR. Other browsers will be supporting these specs soon, including Firefox Reality, Oculus Browser, Edge and Magic Leap's Helio browser, among others.

This article begins a series on the immersive web. This installment covers setting up a basic WebXR application as well as entering and exiting an XR session. Later articles will cover the frame loop (the workhorse of WebXR experience), the specifics of augmented reality, and the WebXR Hit Test API, a means of detecting surfaces in an AR session. Unless stated otherwise, everything I cover in this and succeeding articles applies equally to both AR and VR.

What is the immersive web? #

Though we use two terms to describe immersive experiences—augmented reality and virtual reality—many think of them on a spectrum from complete reality to completely virtual, with degrees of immersion in between. The 'X' in XR is intended to reflect that thinking by being a sort of algebraic variable that stands for anything in the spectrum of immersive experiences.

The spectrum of immersive experiences

Examples of immersive experiences include:

Games
360° videos
Traditional 2D (or 3D) videos presented in immersive surroundings
Home buying
Viewing products in your home before you buy them
Immersive art
Something cool nobody's thought of yet

Concepts and usage #

I'll explain a few basics of using the WebXR Device API. If you need more depth than I've provided, check out the Immersive Web Working Group's WebXR samples or MDN's growing reference materials. If you're familiar with early versions of the WebXR Device API, you should glance over all of this material. There have been changes.

The code in this article is based on the Immersive Web Working Group's barebones sample (demo, source), but is edited for clarity and simplicity.

Part of creating the WebXR specification has been fleshing out security and privacy measures to protect users. Consequently, implementations must adhere to certain requirements. A web page or app must be active and focused before it can request anything sensitive from the viewer. Web pages or apps must be served over HTTPS. The API itself is designed to protect information obtained from sensors and cameras, which it needs in order to function.

Request a session #

Entering an XR session requires a user gesture. To get that, use feature detection to test for XRSystem (via navigator.xr) and make a call to XRSystem.isSessionSupported(). Be aware that in Chrome versions 79 and 80 the XRSystem object was called XR.

In the example below, I've indicated that I want a virtual reality session with the 'immersive-vr' session type. The other session types are 'immersive-ar' and 'inline'. An inline session is for presenting content within HTML and is mainly used for teaser content. The Immersive AR Session sample demonstrates this. I'll explain that in a later article.

Once I know that virtual reality sessions are supported, I enable a button that lets me acquire a user gesture.

if (navigator.xr) {
  const supported = await navigator.xr.isSessionSupported('immersive-vr');
  if (supported) {
    xrButton.addEventListener('click', onButtonClicked);
    xrButton.textContent = 'Enter VR';
    xrButton.enabled = supported; // supported is Boolean
  }
}

After enabling the button, I wait for a click event then request a session.

let xrSession = null;
function onButtonClicked() {
  if (!xrSession) {
    navigator.xr.requestSession('immersive-vr')
    .then((session) => {
      xrSession = session;
      xrButton.textContent = 'Exit XR';
      onSessionStarted(xrSession);
    });
  } else {
    xrSession.end();
  }
}

Notice the object hierarchy in this code. It moves from navigator to xr to an XRSession instance. In early versions of the API, a script had to request a device before requesting a session. Now, the device is acquired implicitly.

Enter a session #

After getting a session, I need to start it and enter it. But first, I need to set up a few things. A session needs an onend event handler so that the app or web page can be reset when the user exits.

I'll also need a <canvas> element to draw my scene on. It needs to be an XR-compatible WebGLRenderingContext or WebGL2RenderingContext. All drawing is done using them or a WebGL-based framework such as Three.js.

Now that I have a place to draw, I need a source of content to draw on it. For that, I create an instance of XRWebGLLayer. I associate it with the canvas by calling XRSession.updateRenderState().

Once I'm in a session, I need a way of determining where things are in virtual reality. I'll need a reference space. A 'local-floor' reference space is one where the origin is located near the viewer and the y-axis is 0 at floor level and is not expected to move. There are other types of reference spaces, but that is a more complicated topic than I can go into here. I save the reference space to a variable because I'll need it when I draw to the screen.

function onSessionStarted(xrSession) {
  xrSession.addEventListener('end', onSessionEnded);

  let canvas = document.createElement('canvas');
  webGLRenContext = canvas.getContext('webgl', { xrCompatible: true });

  xrSession.updateRenderState({
    baseLayer: new XRWebGLLayer(xrSession, webGLRenContext)
  });

  xrSession.requestReferenceSpace('local-floor')
  .then((refSpace) => {
    xrRefSpace = refSpace;
    xrSession.requestAnimationFrame(onXRFrame);
  });
}

After getting a reference space, I call XRSession.requestAnimationFrame(). This is the start of presenting virtual content, which is done in the frame loop.

Run a frame loop #

The frame loop is a user-agent controlled infinite loop in which content is repeatedly drawn to the screen. Content is drawn in discrete blocks called frames. The succession of frames creates the illusion of movement. For VR applications the frames per second can be anywhere from 60 to 144. AR for Android runs at 30 frames per second. Your code should not assume any particular frame rate.

The basic process for the frame loop is:

Call XRSession.requestAnimationFrame(). In response, the user agent invokes the XRFrameRequestCallback, which is defined by you.
Inside your callback function:
1. Call XRSession.requestAnimationFrame() again.
2. Get the viewer's pose.
3. Pass ('bind') the WebGLFramebuffer from the XRWebGLLayer to the WebGLRenderingContext.
4. Iterate over each XRView object, retrieving its XRViewport from the XRWebGLLayer and passing it to the WebGLRenderingContext.
5. Draw something to the framebuffer.

The remainder of this article describes step 1 and part of step 2, setting up and calling the XRFrameRequestCallback. The remaining items of step 2 are covered in part II.

The XRFrameRequestCallback #

The XRFrameRequestCallback is defined by you. It takes two parameters: a DOMHighResTimeStamp and an XRFrame instance. The XRFrame object provides the information needed to render a single frame to the display. The DOMHighResTimeStamp argument is for future use.

Before doing anything else, I'm going to request the next animation frame. As previously stated, the timing of frames is determined by the user agent based on the underlying hardware. Requesting the next frame first ensures that the frame loop continues if something during the callback throws an error.

function onXRFrame(hrTime, xrFrame) {
  let xrSession = xrFrame.session;
  xrSession.requestAnimationFrame(onXRFrame);
  // Render a frame.
}

At this point, it's time to draw something for the viewer. That's a discussion for part II. Before going there, let me show you how to end a session.

End the session #

An immersive session may end for several reasons including ending by your own code through a call to XRSession.end(). Other causes include the headset being disconnected or another application taking control of it. This is why a well-behaved application should monitor the end event. When it occurs, discard the session and its related render objects. An ended immersive session cannot be resumed. To reenter the immersive experience, my app needs to start a new session.

Recall from Entering a session that during setup, I added an onend event handler.

function onSessionStarted(xrSession) {
  xrSession.addEventListener('end', onSessionEnded);
  // More setup…
}

Inside the event handler, restore the state of the app before the user entered a session.

function onSessionEnded(event) {
  xrSession = null;
  xrButton.textContent = 'Enter VR';
}

Conclusion #

I haven't explained everthing you need to write a Web XR or AR application. Hopefull, I've give you enought to start making sense of the code for yourself, and enough to start experimenting. In the next article, I'll explain the frame loop, which is where content is drawn to the screen.

Photo by JESHOOTS.COM on Unsplash

Create OS-style backgrounds with backdrop-filter

2019-07-26T00:00:00Z

Translucence, blurring, and other effects are useful ways of creating depth while keeping the context of the background content. They support a host of use cases such as frosted glass, video overlays, translucent navigation headers, inappropriate image censoring, image loading, and so on. You may recognize these effects from two popular operating systems: Windows 10 and iOS.

An example of a frosted glass effect. Source.

Historically, these techniques were difficult to implement on the web, requiring less than perfect hacks or workarounds. In recent years both Safari and Edge have provided these capabilities through the background-filter (and alternatively, the -webkit-backdrop-filter) property, which dynamically blends foreground and background colors based on filter functions. Now Chrome supports background-filter, starting in version 76.

A demonstration of the filter functions for backdrop-filter. Try the example on CodePen.

Browser support #

Browser support

Chrome 76, Supported 76
Firefox 103, Supported 103
Edge 17, Supported 17
Safari 9, Supported 9

Source

For performance reasons, fall back to an image instead of a polyfill when backdrop-filter isn't supported. The example below shows this.

@supports (backdrop-filter: none) {
	.background {
		backdrop-filter: blur(10px);
	}
}

@supports not (backdrop-filter: none) {
  .background {
    background-image: blurred-hero.png;
  }
}

## Basics

- The `backdrop-filter` property applies one or more filters to an element, changing the appearance of anything behind the element.
- The overlaying element must be at least partially transparent.
- The overlaying element will get a new stacking context.

<aside class="aside flow bg-state-bad-bg color-state-bad-text"><p class="cluster color-state-bad-text"><span class="aside__icon box-block "><svg width="24" height="24" viewBox="0 0 24 24" fill="currentColor" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Error sign">   <path fill-rule="evenodd" clip-rule="evenodd" d="M12 2C6.48 2 2 6.48 2 12s4.48 10 10 10 10-4.48 10-10S17.52 2 12 2zm-1 15v-2h2v2h-2zm0-10v6h2V7h-2z" /> </svg></span><strong>Caution</strong></p><div class=" flow"> <code>backdrop-filter</code> may harm performance. Test it before deploying. </div></aside>

CSS `backdrop-filter` applies one or more effects to an element that is translucent or transparent. To understand that, consider the images below.

<div class="switcher">
<figure class="compare flow" data-type="worse" data-size="full"><p class="compare__label">No foreground transparency</p>

<img     alt="A triangle superimposed on a circle. The circle can&#x27;t be seen through the triangle."          decoding="async"          height="283"          loading="lazy"     sizes="(min-width: 480px) 480px, calc(100vw - 48px)"     src="https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format"     srcset="https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=200 200w,     https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=228 228w,     https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=260 260w,     https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=296 296w,     https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=338 338w,     https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=385 385w,     https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=439 439w,     https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=500 500w,     https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=571 571w,     https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=650 650w,     https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=741 741w,     https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=845 845w,     https://web-dev.imgix.net/image/admin/LOqxvB3qqVkbZBmxMmKS.png?auto=format&w=960 960w"          width="480"   />

```css
.frosty-glass-pane {
  backdrop-filter: blur(2px);
}

Foreground transparency

.frosty-glass-pane {
  opacity: .9;
  backdrop-filter: blur(2px);
}

The image on the left shows how overlapping elements would be rendered if backdrop-filter were not used or supported. The image on the right applies a blurring effect using backdrop-filter. Notice that it uses opacity in addition to backdrop-filter. Without opacity, there would be nothing to apply blurring to. It almost goes without saying that if opacity is set to 1 (fully opaque) there will be no effect on the background.

The backdrop-filter property is like CSS filters in that all your favorite filter functions are supported: blur(), brightness(), contrast(), opacity(), drop-shadow(), and so on. It also supports the url() function if you want to use an external image as the filter, as well as the keywords none, inherit, initial, and unset. There are explanations for all of this on MDN, including descriptions of syntax, filters, and values.

When backdrop-filter is set to anything other than none, the browser creates a new stacking context. A containing block may also be created, but only if the element has absolute and fixed position descendants.

You can combine filters for rich and clever effects, or use just one filter for more subtle or precise effects. You can even combine them with SVG filters.

Examples #

Design techniques and styles previously reserved for operating systems are now performant and achievable with a single CSS declaration. Let's look at some examples.

Single filter #

In the following example, the frosted effect is achieved by combining color and blur. The blur is supplied by backdrop-filter, while the color comes from the element's semi-transparent background color.

.blur-behind-me {
  background-color: rgba(255, 255, 255, 0.3);
  backdrop-filter: blur(.5rem);
}

Try this example for yourself in CodePen.

Multiple filters #

Sometimes you'll need multiple filters to achieve the desired effect. To do this, provide a list of filters separated by a space. For example:

.brighten-saturate-and-blur-behind-me {
  backdrop-filter: brightness(150%) saturate(150%) blur(1rem);
}

In the following example, each of the four panes has a different combination of backdrop filters while the same set of shapes are animated behind them.

Try this example for yourself in CodePen.

Overlays #

This example shows how to blur a semi-transparent background to make text readable while stylistically blending with a page's background.

.modal {
  backdrop-filter: blur(10px);
  background-color: rgba(255, 255, 255, 0.5);
}

Try this example for yourself.

Text contrast on dynamic backgrounds #

As stated earlier, backdrop-filter allows performant effects that would be difficult or impossible on the web. An example of this is changing a background in response to an animation. In this example, backdrop-filter maintains the high contrast between the text and its background in spite of what's going on behind the text. It starts with the default background color darkslategray and uses backdrop-filter to invert the colors after the transformation.

.container::before {
  z-index: 1;
  background-color: darkslategray;
  filter: invert(1);
}

.container::after {
	backdrop-filter: invert(1);
  z-index: 3;
}

Try this example from Chen Hui Jing in Codrops.

Conclusion #

More than 560 of you have upvoted the Chromium bug over the past few years, clearly marking this as a long awaited CSS feature. Chrome's release of backdrop-filter in version 76 brings the web a step closer to truly OS-like UI presentation.

Additional resources #

Visual searching with the Web Perception Toolkit

2019-05-07T00:00:00Z

Wouldn't it be great if users could search your site using their camera? Imagine this. Your site is Razor McShaveyface. Your customers tell you they have trouble finding the right cartridges for their razor when they reorder. They don't know the right keywords for your product search. And let's be honest, they probably never will.

What if they never need to? What if they could point their phone's camera at the UPC code on package and your site could present them with the right cartridge and a big red "reorder" button?

Think of other ways you can use a camera on a site. Imagine a site that supports in-store price checking. Imagine getting information about a museum exhibit or historical marker. Imagine identifying real-world landmarks in games like geocaching or scavenger hunts.

The Web Perception Toolkit makes these camera-based scenarios possible. In some cases you can even create an experience without writing code.

How does it work? #

The open-source Web Perception Toolkit helps you add visual search to your website. It passes a device camera stream through a set of detectors that map real-world objects, here called "targets", to content on your site. This mapping is defined using Structured Data (JSON-LD) on your site. With this data, you can present the right information in a customizable UI.

I'll show you enough of this to give you a taste of how it works. For a complete explanation, check out the Getting Started guide, the toolkit reference, the I/O Sandbox demo and the sample demos.

Structured data #

The toolkit can't find just any target in the camera's view. You must provide it with linked JSON data for the targets you want it to recognize. This data also contains information about those targets that will be shown to the user.

The data is all you need to create a user experience like the one in the image below. If you do nothing else, the Web Perception Toolkit can identify targets, then show and hide cards based on the information provided in the data. Try this for yourself using our artifact-map demo.

The default interface.

Add data to your site with a JSON linked data file, included using a <script> tag and the "application/ld+json" MIME type.

<script type="application/ld+json" src="//path/to/your/sitemap.jsonld">

The file itself looks something like this:

[
  {
    "@context": "https://schema.googleapis.com/",
    "@type": "ARArtifact",
    "arTarget": {
      "@type": "Barcode",
      "text": "012345678912"
    },
    "arContent": {
      "@type": "WebPage",
      "url": "http://localhost:8080/demo/artifact-map/products/product1.html",
      "name": "Product 1",
      "description": "This is a product with a barcode",
      "image": "http://localhost:8080/demo/artifact-map/products/product1.png"
    }
  }
]

The user experience #

What if you want more than the default user experience? The toolkit gives you lifecycle events, Card and Button objects for crafting the user experience around those events, and an easy way to style the cards. I'm going to show a little of this with code based losely on our Getting Started guide.

The most important lifecycle event is PerceivedResults, which is fired every time a target is found. A target can be a real-world object or a marker such as a bar code or QR code.

The process for responding to this event is the same as for any other event with an exception already alluded to. If you don't implement the event, a user interface is automatically created using structured data. To override this behavior start your event handler by callingevent.preventDefault().

const container = document.querySelector('.container');
async function onPerceivedResults(event) {
  // preventDefault() to stop default result Card from showing.
  event.preventDefault();
  // Process the event.
}
window.addEventListener(PerceptionToolkit.Events.PerceivedResults, onPerceivedResults);

Let's look at the event more closely. The event itself contains arrays of markers and targets that it has both found and lost. When targets are found in the world, the even fires and passes found objects in event.found. Similarly, when targets pass from the camera view the event fires again, passing lost objects in event.lost. This helps account for hand and marker movements: cameras not held steadily enough, dropped markers, that kind of thing.

async function onPerceivedResults(event) {
  // preventDefault() to stop default result Card from showing
  event.preventDefault();
  if (container.childNodes.length > 0) { return; }
  const { found, lost } = event.detail;
  // Deal with lost and found objects.
}

Next, you show an appropriate card based on what the toolkit found.

async function onPerceivedResults(event) {
  event.preventDefault();
  if (container.childNodes.length > 0) { return; }
  const { found, lost } = event.detail;
  if (found.length === 0 && lost.length === 0) {
    // Object not found.
    // Show a card with an offer to show the catalog.
  } else if (found.length > 0) {
    // Object found.
    // Show a card with a reorder button.
  }
}

Adding cards and buttons is simply a matter of instantiating them and appending them to a parent object. For example:

const { Card } = PerceptionToolkit.Elements;
const card = new Card();
card.src = 'Your message here.'
container.appendChild(card)'

Finally, here's what the whole thing looks like. Notice the conveniences I've added to the user experience. Whether the marker is found or not, I provide one-click access to what I think is most useful in the circumstances.

async function onPerceivedResults(event) {
  // preventDefault() to stop default result Card from showing
  event.preventDefault();
  if (container.childNodes.length > 0) { return; }
  const { found, lost } = event.detail;
  const { ActionButton, Card } = PerceptionToolkit.Elements;
  if (found.length === 0 && lost.length === 0) {
    //Make a view catalog button.
    const button =  new ActionButton();
    button.label = 'View catalog';
    button.addEventListener('click', () => {
      card.close();
      //Run code to launch a catalog.
    });
    //Make a card for the button.
    const card = new Card();
    card.src = 'We wish we could help, but that\'s not our razor. Would you like to see our catalog?';
    card.appendChild(button);
    //Tell the toolkit it does not keep the card around
    // if it finds something it recognizes.
    card.dataset.notRecognized = true;
    container.appendChild(card);
  } else if (found.length > 0) {
    //Make a reorder button.
    const button = new ActionButton();
    button.label = 'Reorder';
    botton.addEventListener('click', () => {
      card.close();
      //Run code to reorder.
    })
    const card = new Card();
    card.src = found[0].content;
    card.appendChild(button);
    container.appendChild(card);
  }
}

Formatting cards #

The Web Perception Toolkit provides built-in formatting for cards and buttons with the default stylesheet. But you can easily add your own. The provided Card and ActionButton objects contain style properties (among many others) that let you put your organizational stamp on the look and feel. To include the default stylesheet, add a <link> element to your page.

<link rel="stylesheet" href="//path/to/toolkit/styles/perception-toolkit.css">

Conclusion #

As I said at the top, this is not an exhaustive look at the Web Perception Toolkit. Hopefully it gives you a sense of how easy it is to add visual searching to a website. Learn more with the Getting Started guide and the sample demos. Dig in to the toolkit documentation to learn what it can do.

The web component

2019-02-06T00:00:00Z

Adding 3D models to a website is tricky. 3D models ideally will be shown in a viewer that can work responsively on all browsers including smartphones, desktop, or even new head-mounted displays. The viewer should support progressive enhancement for performance and rendering quality. It should support use cases on all devices ranging from older, lower-powered smartphones to newer devices that support augmented reality. It should stay up to date with current technologies. It should be performant and accessible. However, building such a viewer requires specialty 3D programming skills, and can be a challenge for web developers that want to host their own models instead of using a third-party hosting service.

To help with that, the <model-viewer> web component, which just released version 1.1, lets you declaratively add a 3D model to a web page, while hosting the model on your own site. The web component supports responsive design and use cases like augmented reality on some devices, and it includes features for accessibility, rendering quality, and interactivity. The goal of the component is to enable adding 3D models to your website without understanding the underlying technology and platforms.

What is a web component? #

A web component is a custom HTML element built from standard web platform features. A web component behaves for all intents and purposes like a standard element. It has a unique tag, it can have properties and methods, and it can fire and respond to events. In short, you don't need to know anything special to use any web component, much less <model-viewer>.

In this article, I will show you things that are particular to <model-viewer>. If you're interested in learning more about web components, webcomponents.org is a good place to start.

What can <model-viewer> do? #

I'll show you a few current capabilities of <model-viewer>. You'll get a great experience today, and <model-viewer> will get better over time as Google adds new features and improves rendering quality. The examples I'm provided are just to give you a sense of what it does. If you want to try them there are installation and usage instructions in its GitHub repo.

Basic 3D models #

Embedding a 3D model is as simple as the markup below. By using glb files, we've ensured that this component will work on any major browser.

<!-- Import the component -->
<script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script>
<script nomodule src="https://unpkg.com/@google/model-viewer/dist/model-viewer-legacy.js"></script>
<!-- Use it like any other HTML element -->
<model-viewer id="mv-demo" shadow-intensity="1" src="./spacesuit.glb"
alt="A 3D model of an astronaut" auto-rotate camera-controls
poster="./spacesuit.jpg"></model-viewer>

That code renders like this:

on Glitch" allow="geolocation; microphone; camera; midi; vr; encrypted-media; clipboard; clipboard-read; clipboard-write" style="height: 100%; width: 100%; border: 0;">

Adding motion and interactivity #

The auto-rotate and camera-controls attributes provide motion and user control. Those aren't the only possible attributes. See the documentation for a complete list of attributes.

<model-viewer src="assets/Astronaut.glb" auto-rotate camera-controls>

Delayed loading with poster images #

Some 3D models can be very large, so you might want to hold off loading them until the user has requested the model. For this, the component has a built-in means of delaying loading until the user wants it. That's what the poster attribute does.

<model-viewer src="assets/Astronaut.glb" controls auto-rotate
poster="./spacesuit.jpg">

To show your users that it's a 3D model, and not just an image, you can provide some preload animation by using script to switch between multiple posters.

<model-viewer id="toggle-poster" src="assets/Astronaut.glb" controls
auto-rotate poster="assets/poster2.png"></model-viewer>
<script>
    const posters = ['poster.png', 'poster2.png'];
    let i = 0;
    setInterval(() =>
        document.querySelector('#toggle-poster').setAttribute('poster',
            `assets/${posters[i++ % 2]}`), 2000);
</script>

Responsive design #

The component handles some types of responsive design, scaling for both mobile and desktop. It can also manage multiple instances on a page and uses Intersection Observer to conserve battery power and GPU cycles when a model isn't visible.

Multiple spacesuit images representing responsiveness.

Looking Forward #

Install <model-viewer> and give it a try The project team wants <model-viewer> to be useful to you, and wants your input on its future. That's not to say they don't have ideas. So give it a try and let us know what you think by filing an issue in GitHub.

Media conversion

2018-09-20T00:00:00Z

In this article we are going to learn some common commands for converting and manipulating specific characteristics of media files. Although we've tried to show equivalent operations for all procedures, not all operations are possible in both applications.

In many cases, the commands we're showing may be combined in a single command line operation, and would be when actually used. For example, there's nothing preventing you from setting an output file's bitrate in the same operation as a file conversion. For this article, we often show these operations as separate commands for the sake of clarity.

Conversion is done with these applications:

Display characteristics #

Both Shaka Packager and FFmpeg can be used to inspect the content of a media file and then display the characteristics of a stream. However, both provide different output for the same media.

Characteristics using Shaka Packager #

packager input=glocken.mp4 --dump_stream_info

The output looks like:

File "glocken.mp4":
Found 2 stream(s).
Stream [0] type: Video
 codec_string: avc1.640028
 time_scale: 30000
 duration: 300300 (10.0 seconds)
 is_encrypted: false
 codec: H264
 width: 1920
 height: 1080
 pixel_aspect_ratio: 1:1
 trick_play_factor: 0
 nalu_length_size: 4

Stream [1] type: Audio
 codec_string: mp4a.40.2
 time_scale: 48000
 duration: 481280 (10.0 seconds)
 is_encrypted: false
 codec: AAC
 sample_bits: 16
 num_channels: 2
 sampling_frequency: 48000
 language: eng
 seek_preroll_ns: 20833

Characteristics using FFmpeg #

ffmpeg -i glocken.mp4

The output looks like:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'glocken.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.83.100
  Duration: 00:00:10.03, start: 0.000000, bitrate: 8063 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc), 1920x1080, 7939 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
At least one output file must be specified

Demux (separate) the audio and video streams #

Shaka Packager requires demuxing when converting files. This is also required for using media frameworks.

Shaka Packager demuxing #

MP4

packager input=myvideo.mp4,stream=video,output=myvideo_video.mp4
packager input=myvideo.mp4,stream=audio,output=myvideo_audio.m4a

Or:

packager \
  input=myvideo.mp4,stream=video,output=myvideo_video.mp4 \
  input=myvideo.mp4,stream=audio,output=myvideo_audio.m4a

WebM

packager \
  input=myvideo.webm,stream=video,output=myvideo_video.webm \
  input=myvideo.webm,stream=audio,output=myvideo_audio.webm

FFmpeg demuxing #

MP4

ffmpeg -i myvideo.mp4 -vcodec copy -an myvideo_video.mp4
ffmpeg -i myvideo.mp4 -acodec copy -vn myvideo_audio.m4a

WebM

ffmpeg -i myvideo.webm -vcodec copy -an myvideo_video.webm
ffmpeg -i myvideo.webm -acodec copy -vn myvideo_audio.webm

Remux (combine) the audio and video streams #

In some situation you will need to combine the audio and video back into a single container. Especially when not using a media framework. This is something FFmpeg can handle quite well and is something Shaka Packager does not currently support.

ffmpeg -i myvideo_video.webm -i myvideo_audio.webm -c copy myvideo.webm

Change characteristics #

Bitrate #

For FFmpeg, we can do this while converting to .mp4 or .webm.

ffmpeg -i myvideo.mov -b:v 350K myvideo.mp4
ffmpeg -i myvideo.mov -vf setsar=1:1 -b:v 350K myvideo.webm

Dimensions (resolution) #

ffmpeg -i myvideo.webm -s 1920x1080 myvideo_1920x1080.webm

File type #

Shaka Packager cannot process .mov files and hence cannot be used to convert files from that format.

.mov to .mp4

ffmpeg -i myvideo.mov myvideo.mp4

.mov to .webm

ffmpeg -i myvideo.mov myvideo.webm

Synchronize audio and video #

To ensure that audio and video synchronize during playback, insert keyframes.

ffmpeg -i myvideo.mp4 -keyint_min 150 -g 150 -f webm -vf setsar=1:1 out.webm

MP4/H.264

ffmpeg -i myvideo.mp4 -c:v libx264 -c:a copy myvideo.mp4

Audio for an MP4

ffmpeg -i myvideo.mp4 -c:v copy -c:a aac myvideo.mp4

WebM/VP9

ffmpeg -i myvideo.webm -v:c libvpx-vp9 -v:a copy myvideo.webm

Audio for a WebM

ffmpeg -i myvideo.webm -v:c copy -v:a libvorbis myvideo.webm
ffmpeg -i myvideo.webm -v:c copy -v:a libopus myvideo.webm

Video-on-demand and live-streaming #

There are two types of streaming protocols we are going to demonstrate in this article. The first is Dynamic Adaptive Streaming over HTTP (DASH), which is an adaptive bitrate streaming technique and web-standards-based method of presenting video-on-demand. The second is HTTP Live Streaming (HLS), which is Apple's standard for live-streaming and video-on-demand for the web.

DASH/MPD #

This example generates the Media Presentation Description (MPD) output file from the audio and video streams.

packager \
  input=myvideo.mp4,stream=audio,output=myvideo_audio.mp4 \
  input=myvideo.mp4,stream=video,output=myvideo_video.mp4 \
  --mpd_output myvideo_vod.mpd

HLS #

These examples generate an M3U8 output file from the audio and video streams, which is a UTF-8 encoded multimedia playlist.

ffmpeg -i myvideo.mp4 -c:a copy -b:v 8M -c:v copy -f hls \
  -hls_time 10 -hls_list_size 0 myvideo.m3u8

OR:

packager \
  'input=myvideo.mp4,stream=video,segment_template=output$Number$.ts,playlist_name=video_playlist.m3u8' \
  'input=myvideo.mp4,stream=audio,segment_template=output_audio$Number$.ts,playlist_name=audio_playlist.m3u8,hls_group_id=audio,hls_name=ENGLISH' \
  --hls_master_playlist_output="master_playlist.m3u8"

Now that we hopefully have a good grasp on how to convert files, we can build on what we've learned in this article and go learn about Media encryption next.

Welcome to the immersive web

2018-05-08T00:00:00Z

The immersive web means virtual world experiences hosted through the browser. This covers entire virtual reality (VR) experiences surfaced in the browser or in VR enabled headsets like Google's Daydream, Oculus Rift, Samsung Gear VR, HTC Vive, and Windows Mixed Reality Headsets, as well as augmented reality experiences developed for AR-enabled mobile devices.

Though we use two terms to describe immersive experiences, they should be thought of as a spectrum from complete reality to a completely immersive VR environment, with various levels of AR in between.

Examples of immersive experiences include:

Immersive 360° videos
Traditional 2D (or 3D) videos presented in immersive surroundings
Data visualizations
Home shopping
Art
Something cool nobody's thought of yet

How do I get there? #

The immersive web has been available for nearly a year now in embryonic form. This was done through the WebVR 1.1 API, which has been available in an origin trial since Chrome 62. That API is also supported by Firefox and Edge as well as a polyfill for Safari.

But it's time to move on.

The origin trial ended on July 24, 2018, and the spec has been superseded by the WebXR Device API and a new origin trial.

What happened to WebVR 1.1? #

We learned a lot from WebVR 1.1, but over time, it became clear that some major changes were needed to support the types of applications developers want to build. The full list of lessons learned is too long to go into here, but includes issues like the API being explicitly tied to the main JavaScript thread, too many opportunities for developers to set up obviously wrong configurations, and common uses like magic window being a side effect rather than an intentional feature. (Magic window is a technique for viewing immersive content without a headset wherein the app renders a single view based on the device's orientation sensor.)

The new design facilitates simpler implementations and large performance improvements. At the same time, AR and other use cases were emerging and it became important that the API be extensible to support those in the future.

The WebXR Device API was designed and named with these expanded use cases in mind and provides a better path forward. Implementors of WebVR have committed to migrating to the WebXR Device API.

What is the WebXR Device API? #

Like the WebVR spec before it, the WebXR Device API is a product of the Immersive Web Community Group which has contributors from Google, Microsoft, Mozilla, and others. The 'X in XR is intended as a sort of algebraic variable that stands for anything in the spectrum of immersive experiences. It's available in the previously mentioned origin trial as well as through a polyfill.

When this article was originally published during the Chrome 67 beta period, only VR capabilities were enabled. Augmented reality arrived in Chrome 69. Read about it in Augmented reality for the web.

There's more to this new API than I can go to in an article like this. I want to give you enough to start making sense of the WebXR samples. You can find more information in both the original explainer and our Immersive Web Early Adopters Guide. I'll be expanding the latter as the origin trial progresses. Feel free to open issues or submit pull requests.

For this article, I'm going to discuss starting, stopping and running an XR session, plus a few basics about processing input.

What I'm not going to cover is how to draw AR/VR content to the screen. The WebXR Device API does not provide image rendering features. That's up to you. Drawing is done using WebGL APIs. You can do that if you're really ambitious. Though, we recommend using a framework. The immersive web samples use one created just for the demos called Cottontail. Three.js has supported WebXR since May. I've heard nothing about A-Frame.

Starting and running an app #

The basic process is this:

Request an XR device.
If it's available, request an XR session. If you want the user to put their phone in a headset, it's called an immersive session and requires a user gesture to enter.
Use the session to run a render loop which provides 60 image frames per second. Draw appropriate content to the screen in each frame.
Run the render loop until the user decides to exit.
End the XR session.

Let's look at this in a little more detail and include some code. You won't be able to run an app from what I'm about to show you. But again, this is just to give a sense of it.

Request an XR device #

Here, you'll recognize the standard feature detection code. You could wrap this in a function called something like checkForXR().

If you're not using an immersive session you can skip advertising the functionality and getting a user gesture and go straight to requesting a session. An immersive session is one that requires a headset. A non-immersive session simply shows content on the device screen. The former is what most people think of when you refer to virtual reality or augmented reality. The latter is sometimes called a 'magic window'.

if (navigator.xr) {
    navigator.xr.requestDevice()
    .then(xrDevice => {
    // Advertise the AR/VR functionality to get a user gesture.
    })
    .catch(err => {
    if (err.name === 'NotFoundError') {
        // No XRDevices available.
        console.error('No XR devices available:', err);
    } else {
        // An error occurred while requesting an XRDevice.
        console.error('Requesting XR device failed:', err);
    }
    })
} else{
    console.log("This browser does not support the WebXR API.");
}

Request an XR session #

Now that we have our device and our user gesture, it's time to get a session. To create a session, the browser needs a canvas on which to draw.

xrPresentationContext = htmlCanvasElement.getContext('xrpresent');
let sessionOptions = {
    // The immersive option is optional for non-immersive sessions; the value
    //   defaults to false.
    immersive: false,
    outputContext: xrPresentationContext
}
xrDevice.requestSession(sessionOptions)
.then(xrSession => {
    // Use a WebGL context as a base layer.
    xrSession.baseLayer = new XRWebGLLayer(session, gl);
    // Start the render loop
})

Run the render loop #

The code for this step takes a bit of untangling. To untangle it, I'm about to throw a bunch of words at you. If you want a peek at the final code, jump ahead to have a quick look then come back for the full explanation. There's quite a bit that you may not be able to infer.

The basic process for a render loop is this:

Request an animation frame.
Query for the position of the device.
Draw content to the position of the device based on it's position.
Do work needed for the input devices.
Repeat 60 times a second until the user decides to quit.

Request a presentation frame #

The word 'frame' has several meanings in a Web XR context. The first is the frame of reference which defines where the origin of the coordinate system is calculated from, and what happens to that origin when the device moves. (Does the view stay the same when the user moves or does it shift as it would in real life?)

The second type of frame is the presentation frame, represented by an XRFrame object. This object contains the information needed to render a single frame of an AR/VR scene to the device. This is a bit confusing because a presentation frame is retrieved by calling requestAnimationFrame(). This makes it compatible with window.requestAnimationFrame().

Before I give you any more to digest, I'll offer some code. The sample below shows how the render loop is started and maintained. Notice the dual use of the word frame. And notice the recursive call to requestAnimationFrame(). This function will be called 60 times a second.

xrSession.requestFrameOfReference('eye-level')
.then(xrFrameOfRef => {
    xrSession.requestAnimationFrame(onFrame(time, xrFrame) {
    // The time argument is for future use and not implemented at this time.
    // Process the frame.
    xrFrame.session.requestAnimationFrame(onFrame);
    }
});

Poses #

Before drawing anything to the screen, you need to know where the display device is pointing and you need access to the screen. In general, the position and orientation of a thing in AR/VR is called a pose. Both viewers and input devices have a pose. (I cover input devices later.) Both viewer and input device poses are defined as a 4 by 4 matrix stored in a Float32Array in column major order. You get the viewer's pose by calling XRFrame.getDevicePose() on the current animation frame object. Always test to see if you got a pose back. If something went wrong you don't want to draw to the screen.

let pose = xrFrame.getDevicePose(xrFrameOfRef);
if (pose) {
    // Draw something to the screen.
}

Views #

After checking the pose, it's time to draw something. The object you draw to is called a view (XRView). This is where the session type becomes important. Views are retrieved from the XRFrame object as an array. If you're in a non-immersive session the array has one view. If you're in an immersive session, the array has two, one for each eye.

for (let view of xrFrame.views) {
    // Draw something to the screen.
}

This is an important difference between WebXR and other immersive systems. Though it may seem pointless to iterate through one view, doing so allows you to have a single rendering path for a variety of devices.

The whole render loop #

If I put all this together, I get the code below. I've left a placeholder for the input devices, which I'll cover in a later section.

xrSession.requestFrameOfReference('eye-level')
.then(xrFrameOfRef => {
    xrSession.requestAnimationFrame(onFrame(time, xrFrame) {
    // The time argument is for future use and not implemented at this time.
    let pose = xrFrame.getDevicePose(xrFrameOfRef);
    if (pose) {
        for (let view of xrFrame.views) {
        // Draw something to the screen.
        }
    }
    // Input device code will go here.
    frame.session.requestAnimationFrame(onFrame);
    }
}

End the XR session #

An XR session may end for several reasons, including ending by your own code through a call to XRSession.end(). Other causes include the headset being disconnected or another application taking control of it. This is why a well-behaved application should monitor the end event and when it occurs, discard the session and renderer objects. An XR session once ended cannot be resumed.

xrDevice.requestSession(sessionOptions)
.then(xrSession => {
    // Create a WebGL layer and initialize the render loop.
    xrSession.addEventListener('end', onSessionEnd);
});

// Restore the page to normal after immersive access has been released.
function onSessionEnd() {
    xrSession = null;

    // Ending the session stops executing callbacks passed to the XRSession's
    // requestAnimationFrame(). To continue rendering, use the window's
    // requestAnimationFrame() function.
    window.requestAnimationFrame(onDrawFrame);
}

How does interaction work? #

As with the application lifetime, I'm just going to give you a taste for how to interact with objects in AR or VR.

The WebXR Device API adopts a "point and click" approach to user input. With this approach every input source has a defined pointer ray to indicate where an input device is pointing and events to indicate when something was selected. Your app draws the pointer ray and shows where it's pointed. When the user clicks the input device, events are fired - select, selectStart, and selectEnd, specifically. Your app determines what was clicked and responds appropriately.

The input device and the pointer ray #

To users, the pointer ray is just a faint line between the controller and whatever they're pointing at. But your app has to draw it. That means getting the pose of the input device and drawing a line from its location to an object in AR/VR space. That process looks roughly like this:

let inputSources = xrSession.getInputSources();
for (let xrInputSource of inputSources) {
    let inputPose = frame.getInputPose(inputSource, xrFrameOfRef);
    if (!inputPose) {
    continue;
    }
    if (inputPose.gripMatrix) {
    // Render a virtual version of the input device
    //   at the correct position and orientation.
    }
    if (inputPose.pointerMatrix) {
    // Draw a ray from the gripMatrix to the pointerMatrix.
    }
}

This is a stripped down version of the Input Tracking sample from the Immersive Web Community Group. As with frame rendering, drawing the pointer ray and the device is up to you. As alluded to earlier, this code must be run as part of the render loop.

Selecting items in virtual space #

Merely pointing at things in AR/VR is pretty useless. To do anything useful, users need the ability to select things. The WebXR Device API provides three events for responding to user interactions: select, selectStart, and selectEnd. They have a quirk I didn't expect: they only tell you that an input device was clicked. They don't tell you what item in the environment was clicked. Event handlers are added to the XRSession object and should be added as soon as its available.

xrDevice.requestSession(sessionOptions)
.then(xrSession => {
    // Create a WebGL layer and initialize the render loop.
    xrSession.addEventListener('selectstart', onSelectStart);
    xrSession.addEventListener('selectend', onSelectEnd);
    xrSession.addEventListener('select', onSelect);
});

This code is based on an Input Selection example , in case you want more context.

To figure out what was clicked you use a pose. (Are you surprised? I didn't think so.) The details of that are specific to your app or whatever framework you're using, and hence beyond the scope of this article. Cottontail's approach is in the Input Selection example.

function onSelect(ev) {
    let inputPose = ev.frame.getInputPose(ev.inputSource, xrFrameOfRef);
    if (!inputPose) {
    return;
    }
    if (inputPose.pointerMatrix) {
    // Figure out what was clicked and respond.
    }
}

Conclusion: looking ahead #

As I said earlier, augmented reality is expected in Chrome 69 (Canary some time in June 2018). Nevertheless, I encourage you try what we've got so far. We need feedback to make it better. Follow it's progress by watching ChromeStatus.com for WebXR Hit Test. You can also follow WebXR Anchors which will improve pose tracking.

Bitrate

2017-06-30T00:00:00Z

In the previous Containers and codecs article, you learned how to change the container (extension) and codec of a media file. In this article, we'll show you how to change bitrate before explaining resolution.

Bitrate and resolution correlate to the amount of data in a media file. It probably goes without saying, but we're going to say it anyway. You can always lower bitrate and resolution, but increasing them is a problem. Without special software and algorithms, quality is going to take a hit.

So always start your conversion process with the highest quality source file you can get your hands on. Before doing anything, even before changing the codec or container, check the file's display characteristics and verify that your source file has a higher bitrate or resolution than your desired result.

Bitrate is the maximum number of bits used to encode one second of a media stream. The more bits used to encode a second of stream, the higher the fidelity.

Unsurprisingly, the different bitrates the web can handle are low. The table below shows you what bitrate you should target for common network conditions. For the sake of comparison, we've thrown in values for Blu-rays and DVDs.

Delivery method	Bitrate
Blu-ray	20Mbs
DVD	6 Mbs
Desktop web	2 Mbs
4G mobile	0.7 Mbs
3G mobile	0.35 Mbs
2G mobile	Depends on network type. EDGE: 0.4 Mbs GPRS: 0.04Mbs

Which value should I use for video on my web pages? The short answer is at least: desktop, 4G, and 3G. If you're serving video in one of the markets referred to as "the next billion users", say India, for example, you'll want to include 2G as well. For demonstration purposes, we're going to target 3G.

Using FFmpeg you set the bitrate with the (surprise!) bitrate (-b) flag.

If you don't have FFmpeg installed read Media application basics to get it set up with Docker.

MP4

/media # ffmpeg -i glocken.mov -b:v 350k -b:a 64k glocken_3g.mp4

WebM

/media # ffmpeg -i glocken.mov -b:v 350k -b:a 64k glocken_3g.webm

Notice that there are two bitrate flags, -b:a and -b:v. One is for the audio stream, and the other is for the video stream.

/media # ls -l
-rw-r--r-- 1 root root  12080306 Mar  7 12:16 glocken.mov
-rwx------ 1 root root    531117 Mar  7 13:42 glocken_3g.mp4
-rwx------ 1 root root    706119 Mar  7 13:46 glocken_3g.webm

Now that your files are prepared, it's time to adjust their resolutions.

Containers and codecs

2017-06-30T00:00:00Z

To support multiple browsers, you'll need to use FFmpeg to convert your .mov file to two different containers: an MP4 container and a WebM container. In actual practice, you would likely specify a codec at the same time. For now, we're letting FFmpeg use its defaults.

If these concepts are new to you, you should read Media file basics before going further. Additionally, if you don't have FFmpeg installed read Media application basics to get it set up with Docker.

We are using the suggested Docker install and the glocken.mov file from Prepare media files for the web added inside the media directory. We used FFmpeg version 4.3.2 for all the commands in this section.

Containers #

First, we need to create our two containers from the .mov file with the .mp4 and .webm file extensions with both an audio and video stream inside the file. Review Media file basics for more about containers and streams if you don't know the different between them.

MP4

/media # ffmpeg -i glocken.mov glocken.mp4

WebM

/media # ffmpeg -i glocken.mov glocken.webm

WebM takes longer to create than MP4. This isn't surprising when you look at the results. While MP4 compresses to about 83% of the original file's size, WebM is down to 78% of the original's size, but can be much smaller. Your results will vary. It's important to call out that FFmpeg 4.2.2 set the default video bitrate to 200k and in 4.3.2 it does not set a default bitrate. So the video is no longer a mere4% of the original. You can see this for yourself using the ls -a bash command in the folder where your media files are located.

For example:

/media # ls -l
-rw-r--r-- 1 root  root  12080306 Mar 7 12:16 glocken.mov
-rwx------ 1 root  root  10106446 Mar 7 12:33 glocken.mp4
-rwx------ 1 root  root   9503301 Mar 7 18:30 glocken.webm

To get a tiny file, you would do this instead:

/media # ffmpeg -i glocken.mov -b:v 200k glocken.webm
...
frame=  300 fps=3.6 q=0.0 Lsize=     483kB time=00:00:10.01 bitrate= 395.0kbits/s speed=0.121x
video:359kB audio:117kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.356068%
/media # ls -l
-rw-r--r-- 1 root  root  12080306 Mar 7 12:16 glocken.mov
-rwx------ 1 root  root  10106446 Mar 7 12:33 glocken.mp4
-rwx------ 1 root  root    494497 Mar 7 18:45 glocken.webm

Check your work #

To verify your results, use FFmpeg and Shaka Packager as already shown in Media Application basics:

/media # packager input=glocken.mp4 --dump_stream_info

/media # ffmpeg -i glocken.mp4

Codecs #

Next, the codec. As stated in Media file basics, a codec is not the same thing as a container (file type). Two files of the same container type could hold data compressed using different codecs. The WebM format for example allows audio to be encoded using either Vorbis or Opus. To change the codec we use FFmpeg. For example, this command outputs an .mkv file with a vorbis audio codec and an av1 video codec.

/media # ffmpeg -i glocken.mov -c:a vorbis -c:v av1 glocken.mkv

In this example, the -c:a flag and the -c:v are for specifying the audio and video codecs respectively.

The Media conversion page lists commands needed to convert codecs. The tables below summarize the libraries used in FFmpeg to perform the codec conversions for WebM and MP4 files. These are the formats recommended for DASH and HLS respectively.

Video #

Codec	Extension	Library
av1	WebM, mkv	libaom-av1
h264	MP4	libx264
vp9	WebM	libvpx-vp9

Audio #

Codec	Extension	Library
aac	MP4	aac
opus	WebM	libopus
vorbis	WebM	libvorbis

Next, we'll show you how to change the bitrate of your newly created files.

What is a media experience?

2017-06-30T00:00:00Z

Users like media, especially videos; they can be fun and informative. On mobile devices, videos can be an easier way to consume information than text. For a good user experience, videos should not need more than the available bandwidth. Users should be able to use them no matter what device they're viewing them with. Users should never need to wait for media to download. Nobody likes it when they press play and nothing happens.

You've no doubt consumed video on your own device, which means that nothing in that last paragraph surprises you. Now you need to learn how to put a video or other media file on your own website. The technical requirements of adding media should be in service to the user experience.

The technical requirements #

Versions of a media file are in common web-friendly formats containing both audio and video streams.
The resolution is appropriate for your users' devices.
The bitrate doesn't overload your users' network bandwidth.
The result is viewable on all major browsers using appropriate technologies.
The file is encrypted (Optional).

The media section of this website will help you achieve these technical requirements. Don't worry if these concepts are still a bit abstract. We'll explain them progressively throughout all the articles. In the first section you will learn about the basic concepts of media, then how to add media to the web in the second section, and in the final section the practical applications, with some advanced techniques, of using media on the web.

Displaying video on the web #

There are four approaches you can take when displaying video on a site.

Upload your videos to a media hosting provider such as YouTube or Vimeo. These options require you to embed their players within your site.
Basic self-hosted embedding using the HTML <video> and <audio> elements.
More full-featured embedding using a video library such as Shaka Player, JW Player, or Video.js.
Building your own media server and streaming application.

This site mainly covers the basics of embedding media. However, we do cover some more advanced topics to get you started on the path towards building your own media streaming application. The effort to do this is not trivial, so we've built a Media PWA with offline support to use as a reference, which should both show you ways this can be accomplished and just how much effort it requires. The application is by no means a production ready offering or competitor to services like YouTube or Vimeo, but it will provide you with a starting point to learn something challenging and new.

Frankly, building a competitor to hosted media services would require a team of expert engineers and thousands of human-hours of work. Unless your goal is to enter that market as a competitor, you're better off using one of the other methods. It's good to understand the technology and while you may not roll out your own application or video player, there is utility in exploring and experimenting on the cutting edge of browser support, or at the very least using one of the existing video libraries.

What we're going to learn #

The media collection has three parts. In this first section, we'll provide concepts and prerequisite information to adding media to your site. This includes explaining how media files are put together, basics about the applications you'll need to prepare your files for the web, and streaming concepts. The second section explains how to prepare your files and convert them to various formats and optionally add encryption. In the last section, we'll show you how to embed a media file in a web page, discuss autoplay best practices, using media frameworks, taking videos offline, and making your videos accessible.

There's a lot of ground to cover, so let's get started with Media file basics.

Prepare media files for the web

2017-06-30T00:00:00Z

Now that we've introduced you to the applications we use when manipulating media files, over the next few pages, we're going to take a raw video file from a camera and transform it into a resource that you can embed in a web page. We're going to show you how to format your video for mobile web playback, and how to create multiple files to cover a range of browsers. Specifically, we'll create a WebM file for use on Chrome and an MP4 file for use on other browsers.

Plus we will dive deeper into common commands used for Media conversion and Media encryption that will serve as a good reference for later.

By "appropriate technologies" we mean Dynamic Adaptive Streaming over HTTP (DASH) or HTTP Live Streaming (HLS), which are the two primary means of providing video in HTML on the major browsers. By the end of this section, you'll be able to create media files that are ready for use in DASH and HLS.

If you want to play along at home, you'll need a raw video file off a camera, preferably one that contains both audio and video. If you don't have one handy, then here's ten seconds of an .mov file named glocken.mov that was taken of the Rathaus-Glockenspiel in Munich's MarienPlatz.

Next, let's get started with Containers and codecs.

Resolution

2017-06-30T00:00:00Z

In the previous articles you learned how to change the containers, codecs, and bitrate of the glocken.mov media file. This article focuses on changing the resolution.

Resolution is the amount of information in a single frame of video, given as the number of logical pixels in each dimension. For example, a resolution of 1920 by 1080 works out to 1080 stacked horizontal lines, each of which is one logical pixel high and 1920 logical pixels wide. This resolution is frequently abbreviated 1080p because technically the width can vary. The dimensions 1080 by 1920 produce an aspect ratio of 16:9, which is the ratio of movie screens and modern television sets. By the way this is the resolution defined as full HD.

YouTube recommends the following resolutions for video uploads, all in the 16:9 aspect ratio. There's nothing specific to YouTube about this list. It's just a list of common 16:9 video resolutions.

Abbreviation	Dimensions
2160p	3840 x 2160
1440p	2560 x 1440
1080p	1920 x 1080
720p	1280 x 720
480p	854 x 480
360p	640 x 360
240p	426 x 240

Which one should you use? That depends on your application. For simple embedding you may decide to chose only a single resolution. If you're preparing files for DASH or HLS, you may chose one, several, or all. Fortunately, this is one of the simplest transformations you'll make with FFmpeg.

If you don't have FFmpeg installed read Media application basics to get it set up with Docker.

MP4

/media # ffmpeg -i glocken.mov -b:v 350k -b:a 64k -s 1280x720 glocken_3g_720p.mp4

WebM

/media # ffmpeg -i glocken.mov -b:v 350k -b:a 64k -s 1280x720 glocken_3g_720p.webm

The following files should now exist:

/media # ls -l
-rw-r--r-- 1 root root  12080306 Mar  7 12:16 glocken.mov
-rwx------ 1 root root    531117 Mar  7 13:42 glocken_3g.mp4
-rwx------ 1 root root    706119 Mar  7 13:46 glocken_3g.webm
-rwx------ 1 root root    539414 Mar  7 14:15 glocken_3g_720p.mp4
-rwx------ 1 root root    735930 Mar  7 14:19 glocken_3g_720p.webm

It's worth reiterating that you should start from the highest resolution and bitrate file you have available. If you're upgrading an older site, you'll want to find your original camera or other high resolution sources and convert from that rather than from older web files.

Now that your files are prepared, you can either add them to a web page as they are now or dive a bit deeper and continue to learn more command line options by reading the Media conversion page, and then close out the section with Media encryption.

Media application basics

2017-06-09T00:00:00Z

Working with media often requires changing the characteristics of media files, such as bitrate or resolution. Finding a straightforward way to get started can be pretty intimidating. On this page, you will learn about the tools used and how to install them quickly.

First, we describe the basic usage for two common command-line media utilities: Shaka Packager and FFmpeg and then we help you quickly get the tools installed. Why cover two applications? While both are powerful and useful by themselves, neither does everything needed to prepare media for the web. We also created the Media conversion and Media encryption pages which show many more common operations with these two applications.

These applications aren't the only options available for file manipulation tasks, but they are two of the most common and powerful. Other options include the GUI applications Miro, HandBrake, and VLC. There are also encoding/transcoding services such as Zencoder, Amazon Elastic Encoder, and Google Transcoder API (currently in beta).

Shaka Packager #

Shaka Packager is a free media packaging SDK. If you were using a media player on your site, Shaka Packager is what you would use to prepare the files. It supports conversion for the two most common video streaming protocols: Dynamic Adaptive Streaming over HTTP (DASH) or HTTP Live Streaming (HLS). Shaka Packager supports key security features: common encryption and Widevine digital rights management (DRM). It can also handle live-streaming, and video-on-demand.

Despite what it says on the package, this utility is for more than C++ developers. You can use it as both a library for building media software and as a command-line utility for preparing media files for web playback. It's the latter capacity that's useful for us here. In fact, for web media creators, Shaka Packager is the only way to do some tasks without spending money on expensive commercial applications.

Here's the basic pattern for a Shaka Packager command:

packager stream_descriptor [stream_descriptor-2 [stream_descriptor-n]] [flags]

This isn't quite what you get if you type packager -help. This example is easier to reason about, and this reflects the examples in the Shaka Packager documentation. Note that there are multiple stream_descriptor items in the pattern. Though we don't show it, you could manipulate the video and audio streams of a file separately in a single command.

Compare this basic pattern with a simple use that displays file characteristics. In the example, We've lined up equivalent parts.

packager stream_descriptor [stream_descriptor-n] [flags]

packager input=glocken.mp4                       --dump_stream_info

The command outputs this:

File "glocken.mp4":
Found 2 stream(s).
Stream [0] type: Video
 codec_string: avc1.640028
 time_scale: 30000
 duration: 300300 (10.0 seconds)
 is_encrypted: false
 codec: H264
 width: 1920
 height: 1080
 pixel_aspect_ratio: 1:1
 trick_play_factor: 0
 nalu_length_size: 4

Stream [1] type: Audio
 codec_string: mp4a.40.2
 time_scale: 48000
 duration: 481280 (10.0 seconds)
 is_encrypted: false
 codec: AAC
 sample_bits: 16
 num_channels: 2
 sampling_frequency: 48000
 language: eng
 seek_preroll_ns: 20833

Look for the characteristics discussed in Media file basics and notice a few things. The height and width are correct for full HD, and the audio and video codecs are among the preferred codecs for their container types, AAC for audio and H264 for video. Notice also that streams are identified with numbers. These are useful for operations that manipulate the audio and video separately.

Notice that the output above doesn't show the bitrate. Despite what's missing, this output is easier to read, so we use it whenever we can. When we need information that Shaka Packager can't get, such as the bitrate, we use FFmpeg.

FFmpeg #

FFmpeg is also a free application for recording, converting, and streaming media files. Its capabilities aren't better or worse than Shaka Packager's. They're just different.

The basic pattern for an FFmpeg command looks like this:

ffmpeg [GeneralOptions] [InputFileOptions] -i input [OutputFileOptions] output

Like Shaka Packager this application can handle multiple streams. Some of its options are used in multiple locations and manipulate the file output differently depending on where they are in the command. Be aware of this as you look at FFmpeg questions on Stack Overflow and similar sites.

We'll again compare the basic pattern to the example for displaying file characteristics.

    ffmpeg [GeneralOptions] [InputFileOptions] -i input        [OutputFileOptions] output

    ffmpeg                                     -i glocken.mp4

In addition to the information we requested, this also prints an error message as shown in the example below. That's because this is technically an incorrect usage of FFmpeg. We use it because it displays information we care about.

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'glocken.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.17.100
  Duration: 00:01:47.53, start: 0.000000, bitrate: 10715 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc), 1920x1080, 10579 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
At least one output file must be specified

Installing applications with Docker #

If you plan to follow along and try our commands you could either install the required tools manually, or take the easy path and use Docker. We suggest using Docker, because this is going to save you a lot of time. On top of that we've provided the instructions to get you set up quickly.

Start by creating a new directory somewhere on your computer named media-tools; you can use whatever name you like, just note that the following instructions assume you are using media-tools as the directory name.
Create a docker and media directory inside of media-tools. This will keep your media directory out of the build context. This is important because media is where files are stored that we plan to do operations on, and some of them could be quite large. Putting the Dockerfile directly in media-tools would slow down building the image if you ever rebuild it down the road—perhaps to change the versions installed.

Create /media-tools/docker/Dockerfile, and add the following build instructions:

FROM google/shaka-packager:release-v2.4.3 as packager
FROM jrottenberg/ffmpeg:4.3.2-alpine38
COPY --from=packager /usr/bin /usr/bin
ENTRYPOINT  ["sh"]

Build the image:
```
docker build -t media-tools ./docker
```

Run the image as an interactive shell. On Linux:

docker run -w /media -v ${PWD}/media:/media -it --rm media-tools
/media #

On Windows:

docker run -w /media -v %cd%/media:/media -it --rm media-tools
/media #

While running the image you can check versions for both FFmpeg and Shaka Packager to validate everything was successful by running ffmpeg -version and packager --version. The output should look like this:

/media # ffmpeg -version
ffmpeg version 4.3.2 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 6.4.0 (Alpine 6.4.0)
configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libass --enable-fontconfig --enable-libfreetype --enable-libvidstab --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libbluray --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-libsrt --enable-libaribb24 --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100

/media # packager --version
packager version v2.4.3-dd9870075f-release

Now that you've tried your hand at using Shaka Packager and FFmpeg, you can continue learning the basic concepts, next up Media streaming basics.

Media Source Extensions

2017-02-08T00:00:00Z

Media Source Extensions (MSE) is a JavaScript API that lets you build streams for playback from segments of audio or video. Although not covered in this article, understanding MSE is needed if you want to embed videos in your site that do things like:

Adaptive streaming, which is another way of saying adapting to device capabilities and network conditions
Adaptive splicing, such as ad insertion
Time shifting
Control of performance and download size

Figure 1: Basic MSE data flow

You can almost think of MSE as a chain. As illustrated in the figure, between the downloaded file and the media elements are several layers.

An <audio> or <video> element to play the media.
A MediaSource instance with a SourceBuffer to feed the media element.
A fetch() or XHR call to retrieve media data in a Response object.
A call to Response.arrayBuffer() to feed MediaSource.SourceBuffer.

In practice, the chain looks like this:

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  mediaSource.addEventListener('sourceopen', sourceOpen);
} else {
  console.log('The Media Source Extensions API is not supported.');
}

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  fetch(videoUrl)
    .then(function (response) {
      return response.arrayBuffer();
    })
    .then(function (arrayBuffer) {
      sourceBuffer.addEventListener('updateend', function (e) {
        if (!sourceBuffer.updating && mediaSource.readyState === 'open') {
          mediaSource.endOfStream();
        }
      });
      sourceBuffer.appendBuffer(arrayBuffer);
    });
}

If you can sort things out from the explanations so far, feel free to stop reading now. If you want a more detailed explanation, then please keep reading. I'm going to walk through this chain by building a basic MSE example. Each of the build steps will add code to the previous step.

A note about clarity #

Will this article tell you everything you need to know about playing media on a web page? No, it's only intended to help you understand more complicated code you might find elsewhere. For the sake of clarity, this document simplifies and excludes many things. We think we can get away with this because we also recommend using a library such as Google's Shaka Player. I will note throughout where I'm deliberately simplifying.

A few things not covered #

Here, in no particular order, are a few things I won't cover.

Playback controls. We get those for free by virtue of using the HTML5 <audio> and <video> elements.
Error handling.

For use in production environments #

Here are some things I'd recommend in a production usage of MSE related APIs:

Before making calls on these APIs, handle any error events or API exceptions, and check HTMLMediaElement.readyState and MediaSource.readyState. These values can change before associated events are delivered.
Make sure previous appendBuffer() and remove() calls are not still in progress by checking the SourceBuffer.updating boolean value before updating the SourceBuffer's mode, timestampOffset, appendWindowStart, appendWindowEnd, or calling appendBuffer() or remove() on the SourceBuffer.
For all SourceBuffer instances added to your MediaSource, ensure none of their updating values are true before calling MediaSource.endOfStream() or updating the MediaSource.duration.
If MediaSource.readyState value is ended, calls like appendBuffer() and remove(), or setting SourceBuffer.mode or SourceBuffer.timestampOffset will cause this value to transition to open. This means you should be prepared to handle multiple sourceopen events.
When handling HTMLMediaElement error events, the contents of MediaError.message can be useful to determine the root cause of the failure, especially for errors that are hard to reproduce in test environments.

Attach a MediaSource instance to a media element #

As with many things in web development these days, you start with feature detection. Next, get a media element, either an <audio> or <video> element. Finally create an instance of MediaSource. It gets turned into a URL and passed to the media element's source attribute.

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  // Is the MediaSource instance ready?
} else {
  console.log('The Media Source Extensions API is not supported.');
}

Figure 1: A source attribute as a blob

That a MediaSource object can be passed to a src attribute might seem a bit odd. They're usually strings, but they can also be blobs. If you inspect a page with embedded media and examine its media element, you'll see what I mean.

Is the MediaSource instance ready? #

URL.createObjectURL() is itself synchronous; however, it processes the attachment asynchronously. This causes a slight delay before you can do anything with the MediaSource instance. Fortunately, there are ways to test for this. The simplest way is with a MediaSource property called readyState. The readyState property describes the relation between a MediaSource instance and a media element. It can have one of the following values:

closed - The MediaSource instance is not attached to a media element.
open - The MediaSource instance is attached to a media element and is ready to receive data or is receiving data.
ended - The MediaSource instance is attached to a media element and all of its data has been passed to that element.

Querying these options directly can negatively affect performance. Fortunately, MediaSource also fires events when readyState changes, specifically sourceopen, sourceclosed, sourceended. For the example I'm building, I'm going to use the sourceopen event to tell me when to fetch and buffer the video.

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  <strong>mediaSource.addEventListener('sourceopen', sourceOpen);</strong>
} else {
  console.log("The Media Source Extensions API is not supported.")
}

<strong>function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  // Create a SourceBuffer and get the media file.
}</strong>

Notice that I've also called revokeObjectURL(). I know this seems premature, but I can do this any time after the media element's src attribute is connected to a MediaSource instance. Calling this method doesn't destroy any objects. It does allow the platform to handle garbage collection at an appropriate time, which is why I'm calling it immediately.

Create a SourceBuffer #

Now it's time to create the SourceBuffer, which is the object that actually does the work of shuttling data between media sources and media elements. A SourceBuffer has to be specific to the type of media file you're loading.

In practice you can do this by calling addSourceBuffer() with the appropriate value. Notice that in the example below the mime type string contains a mime type and two codecs. This is a mime string for a video file, but it uses separate codecs for the video and audio portions of the file.

Version 1 of the MSE spec allows user agents to differ on whether to require both a mime type and a codec. Some user agents don't require, but do allow just the mime type. Some user agents, Chrome for example, require a codec for mime types that don't self-describe their codecs. Rather than trying to sort all this out, it's better to just include both.

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  mediaSource.addEventListener('sourceopen', sourceOpen);
} else {
  console.log('The Media Source Extensions API is not supported.');
}

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  <strong>
    var mime = 'video/webm; codecs="opus, vp09.00.10.08"'; // e.target refers to
    the mediaSource instance. // Store it in a variable so it can be used in a
    closure. var mediaSource = e.target; var sourceBuffer =
    mediaSource.addSourceBuffer(mime); // Fetch and process the video.
  </strong>;
}

Get the media file #

If you do an internet search for MSE examples, you'll find plenty that retrieve media files using XHR. To be more cutting edge, I'm going to use the Fetch API and the Promise it returns. If you're trying to do this in Safari, it won't work without a fetch() polyfill.

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  <strong>
    fetch(videoUrl) .then(function(response){' '}
    {
      // Process the response object.
    }
    );
  </strong>;
}

A production quality player would have the same file in multiple versions to support different browsers. It could use separate files for audio and video to allow audio to be selected based on language settings.

Real world code would also have multiple copies of media files at different resolutions so that it could adapt to different device capabilities and network conditions. Such an application is able to load and play videos in chunks either using range requests or segments. This allows for adaption to network conditions while media are playing. You may have heard the terms DASH or HLS, which are two methods of accomplishing this. A full discussion of this topic is beyond the scope of this introduction.

Process the response object #

The code looks almost done, but the media doesn't play. We need to get media data from the Response object to the SourceBuffer.

The typical way to pass data from the response object to the MediaSource instance is to get an ArrayBuffer from the response object and pass it to the SourceBuffer. Start by calling response.arrayBuffer(), which returns a promise to the buffer. In my code, I've passed this promise to a second then() clause where I append it to the SourceBuffer.

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  fetch(videoUrl)
    .then(function(response) {
      <strong>return response.arrayBuffer();</strong>
    })
    <strong>.then(function(arrayBuffer) {
      sourceBuffer.appendBuffer(arrayBuffer);
    });</strong>
}

Call endOfStream() #

After all ArrayBuffers are appended, and no further media data is expected, call MediaSource.endOfStream(). This will change MediaSource.readyState to ended and fire the sourceended event.

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  fetch(videoUrl)
    .then(function(response) {
      return response.arrayBuffer();
    })
    .then(function(arrayBuffer) {
      <strong>sourceBuffer.addEventListener('updateend', function(e) {
        if (!sourceBuffer.updating && mediaSource.readyState === 'open') {
          mediaSource.endOfStream();
        }
      });</strong>
      sourceBuffer.appendBuffer(arrayBuffer);
    });
}

The final version #

Here's the complete code example. I hope you have learned something about Media Source Extensions.

var vidElement = document.querySelector('video');

if (window.MediaSource) {
  var mediaSource = new MediaSource();
  vidElement.src = URL.createObjectURL(mediaSource);
  mediaSource.addEventListener('sourceopen', sourceOpen);
} else {
  console.log('The Media Source Extensions API is not supported.');
}

function sourceOpen(e) {
  URL.revokeObjectURL(vidElement.src);
  var mime = 'video/webm; codecs="opus, vp09.00.10.08"';
  var mediaSource = e.target;
  var sourceBuffer = mediaSource.addSourceBuffer(mime);
  var videoUrl = 'droid.webm';
  fetch(videoUrl)
    .then(function (response) {
      return response.arrayBuffer();
    })
    .then(function (arrayBuffer) {
      sourceBuffer.addEventListener('updateend', function (e) {
        if (!sourceBuffer.updating && mediaSource.readyState === 'open') {
          mediaSource.endOfStream();
        }
      });
      sourceBuffer.appendBuffer(arrayBuffer);
    });
}

Feedback #

ResizeObserver: it’s like document.onresize for elements

2016-10-07T00:00:00Z

Before ResizeObserver, you had to attach a listener to the document's resize event to get notified of any change of the viewport's dimensions. In the event handler, you would then have to figure out which elements have been affected by that change and call a specific routine to react appropriately. If you needed the new dimensions of an element after a resize, you had to call getBoundingClientRect() or getComputedStyle(), which can cause layout thrashing if you don't take care of batching all your reads and all your writes.

This didn't even cover cases where elements change their size without the main window having been resized. For example, appending new children, setting an element's display style to none, or similar actions can change the size of an element, its siblings, or its ancestors.

This is why ResizeObserver is a useful primitive. It reacts to changes in size of any of the observed elements, independent of what caused the change. It provides access to the new size of the observed elements too.

Browser support

Chrome 64, Supported 64
Firefox 69, Supported 69
Edge 79, Supported 79
Safari 13.1, Supported 13.1

Source

API #

All the APIs with the Observer suffix we mentioned above share a simple API design. ResizeObserver is no exception. You create a ResizeObserver object and pass a callback to the constructor. The callback is passed an array of ResizeObserverEntry objects—one entry per observed element—which contains the new dimensions for the element.

var ro = new ResizeObserver(entries => {
  for (let entry of entries) {
    const cr = entry.contentRect;
    console.log('Element:', entry.target);
    console.log(`Element size: ${cr.width}px x ${cr.height}px`);
    console.log(`Element padding: ${cr.top}px ; ${cr.left}px`);
  }
});

// Observe one or multiple elements
ro.observe(someElement);

Some details #

What is being reported? #

Generally, a ResizeObserverEntry reports the content box of an element through a property called contentRect, which returns a DOMRectReadOnly object. The content box is the box in which content can be placed. It is the border box minus the padding.

It's important to note that while ResizeObserver reports both the dimensions of the contentRect and the padding, it only watches the contentRect. Don't confuse contentRect with the bounding box of the element. The bounding box, as reported by getBoundingClientRect(), is the box that contains the entire element and its descendants. SVGs are an exception to the rule, where ResizeObserver will report the dimensions of the bounding box.

As of Chrome 84, ResizeObserverEntry has three new properties to provide more detailed information. Each of these properties returns a ResizeObserverSize object containing a blockSize property and an inlineSize property. This information is about the observered element at the time the callback is invoked.

borderBoxSize
contentBoxSize
devicePixelContentBoxSize

All of these items return read-only arrays because in the future it's hoped that they can support elements that have multiple fragments, which occur in multi-column scenarios. For now, these arrays will only contain one element.

Platform support for these properties is limited, but Firefox already supports the first two.

When is it being reported? #

The spec proscribes that ResizeObserver should process all resize events before paint and after layout. This makes the callback of a ResizeObserver the ideal place to make changes to your page's layout. Because ResizeObserver processing happens between layout and paint, doing so will only invalidate layout, not paint.

Gotcha #

You might be asking yourself: what happens if I change the size of an observed element inside the callback to ResizeObserver? The answer is: you will trigger another call to the callback right away. Fortunately, ResizeObserver has a mechanism to avoid infinite callback loops and cyclic dependencies. Changes will only be processed in the same frame if the resized element is deeper in the DOM tree than the shallowest element processed in the previous callback. Otherwise, they'll get deferred to the next frame.

Application #

One thing that ResizeObserver allows you to do is to implement per-element media queries. By observing elements, you can imperatively define your design breakpoints and change an element's styles. In the following example, the second box will change its border radius according to its width.

const ro = new ResizeObserver(entries => {
  for (let entry of entries) {
    entry.target.style.borderRadius =
        Math.max(0, 250 - entry.contentRect.width) + 'px';
  }
});
// Only observe the second box
ro.observe(document.querySelector('.box:nth-child(2)'));

Another interesting example to look at is a chat window. The problem that arises in a typical top-to-bottom conversation layout is scroll positioning. To avoid confusing the user, it is helpful if the window sticks to the bottom of the conversation, where the newest messages appear. Additionally, any kind of layout change (think of a phone going from landscape to portrait or vice versa) should achieve the same.

ResizeObserver allows you to write a single piece of code that takes care of both scenarios. Resizing the window is an event that a ResizeObserver can capture by definition, but calling appendChild() also resizes that element (unlessoverflow: hidden is set), because it needs to make space for the new elements. With this in mind, it takes very few lines to achieve the desired effect:

const ro = new ResizeObserver(entries => {
  document.scrollingElement.scrollTop =
    document.scrollingElement.scrollHeight;
});

// Observe the scrollingElement for when the window gets resized
ro.observe(document.scrollingElement);
// Observe the timeline to process new messages
ro.observe(timeline);

Pretty neat, huh?

From here, I could add more code to handle the case where the user has scrolled up manually and wants scrolling to stick to that message when a new message comes in.

Another use case is for any kind of custom element that is doing its own layout. Until ResizeObserver, there was no reliable way to get notified when its dimensions change so its children can be laid out again.

Conclusion #

ResizeObserver is available in most major browsers. In some cases, that availability is quite recent. There are some polyfills available but it is not possible to completely duplicate the functionality of ResizeObserver. Current implementations either rely on polling or on adding sentinel elements to the DOM. The former will drain your battery on mobile by keeping the CPU busy while the latter modifies your DOM and might mess up styling and other DOM-reliant code.

Photo by Markus Spiske on Unsplash.

The

2014-02-15T00:00:00Z

You've properly prepared a video file for the web. You've given it correct dimensions and the correct resolution. You've even created separate WebM and MP4 files for different browsers.

For anyone to see your video, you still need to add it to a web page. Doing so properly requires adding two HTML elements: the <video> element and the <source> element. In addition to basics about these tags, this article explains attributes you should add to those tags to craft a good user experience.

Specify a single file #

Although it's not recommended, you can use the video element by itself. Always use the type attribute as shown below. The browser uses this to determine if it can play the provided video file. If it can't, the enclosed text displays.

<video src="chrome.webm" type="video/webm">
    <p>Your browser cannot play the provided video file.</p>
</video>

Specify multiple file formats #

Recall from Media file basics that not all browsers support the same video formats. The <source> element lets you specify multiple formats as a fallback in case the user's browser doesn't support one of them.

The example below produces the embedded video that is used as an example later in this article.

<video controls>
  <source src="https://storage.googleapis.com/web-dev-assets/video-and-source-tags/chrome.webm" type="video/webm">
  <source src="https://storage.googleapis.com/web-dev-assets/video-and-source-tags/chrome.mp4" type="video/mp4">
  <p>Your browser cannot play the provided video file.</p>
</video>

Try it on Glitch (source)

You should always add a type attribute to the <source> tags event though it is optional. This ensures that the browser only downloads the file that it is capable of playing.

This approach has several advantages over serving different HTML or server-side scripting, especially on mobile:

You can list formats in order of preference.
Client-side switching reduces latency; only one request is made to get content.
Letting the browser choose a format is simpler, quicker, and potentially more reliable than using a server-side support database with user-agent detection.
Specifying each file source's type improves network performance; the browser can select a video source without having to download part of the video to "sniff" the format.

These issues are especially important in mobile contexts, where bandwidth and latency are at a premium, and the user's patience is likely limited. Omitting the type attribute can affect performance when there are multiple sources with unsupported types.

There are a few ways you can dig into the details. Check out A Digital Media Primer for Geeks to find out more about how video and audio work on the web. You can also use remote debugging in DevTools to compare network activity with type attributes and without type attributes.

Specify start and end times #

Save bandwidth and make your site feel more responsive: use media fragments to add start and end times to the video element.

This browser does not support the video element.

To use a media fragment, add #t=[start_time][,end_time] to the media URL. For example, to play the video from seconds 5 to 10, specify:

<source src="video/chrome.webm#t=5,10" type="video/webm">

You can also specify the times in <hours>:<minutes>:<seconds>. For example, #t=00:01:05 starts the video at one minute, five seconds. To play only the first minute of video, specify #t=,00:01:00.

You can use this feature to deliver multiple views on the same video—like cue points in a DVD–without having to encode and serve multiple files.

For this feature to work, your server must support range requests and that capability must be enabled. Most servers enable range requests by default. Because some hosting services turn them off, you should confirm that range requests are available for using fragments on your site.

Fortunately, you can do this in your browser developer tools. In Chrome, for instance, it's in the Network panel. Look for the Accept-Ranges header and verify that it says bytes. In the image, I've drawn a red box around this header. If you do not see bytes as the value, you'll need to contact your hosting provider.

Chrome DevTools screenshot: Accept-Ranges: bytes.

Include a poster image #

Add a poster attribute to the video element so that viewers have an idea of the content as soon as the element loads, without needing to download the video or start playback.

<video poster="poster.jpg" ...>
  …
</video>

A poster can also be a fallback if the video src is broken or if none of the supplied video formats are supported. The only downside to a poster images is an additional file request, which consumes some bandwidth and requires rendering. For more information see Efficiently encode images.

Don't

Without a fallback poster, the video just looks broken.

A fallback poster makes it seem as if the first frame has been captured.

Ensure videos don't overflow containers #

When video elements are too big for the viewport, they may overflow their container, making it impossible for the user to see the content or use the controls.

Android Chrome screenshot, portrait: unstyled video element overflows viewport.

Android Chrome screenshot, landscape: unstyled video element overflows viewport.

You can control video dimensions using CSS. If CSS does not meet all of your needs, JavaScript libraries and plugins such as FitVids (outside the scope of this article) can help, even for videos from YouTube and other sources. Unfortunately, these resources can increase your network payload sizes with negative consequences for your revenues and your users' wallets.

For simple uses like the ones I'm describing here, use CSS media queries to specify the size of elements depending on the viewport dimensions; max-width: 100% is your friend.

For media content in iframes (such as YouTube videos), try a responsive approach (like the one proposed by John Surdakowski).

CSS #

.video-container {
    position: relative;
    padding-bottom: 56.25%;
    padding-top: 0;
    height: 0;
    overflow: hidden;
}

.video-container iframe,
.video-container object,
.video-container embed {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
}

HTML #

<div class="video-container">
  <iframe
    src="//www.youtube.com/embed/l-BA9Ee2XuM"
    frameborder="0"
    width="560"
    height="315"
  ></iframe>
</div>

Try it

Compare the responsive sample to the unresponsive version. As you can see, the unresponsive version isn't a great user experience.

Device orientation #

Device orientation isn't an issue for desktop monitors or laptops, but it's hugely important when considering web page design for mobile devices and tablets.

Safari on iPhone does a good job of switching between portrait and landscape orientation:

Screenshot of video playing in Safari on iPhone, portrait.

Screenshot of video playing in Safari on iPhone, landscape.

Device orientation on an iPad and Chrome on Android can be problematic. For example, without any customization a video playing on an iPad in landscape orientation looks like this:

Screenshot of video playing in Safari on iPad, landscape.

Setting the video width: 100% or max-width: 100% with CSS can resolve many device orientation layout problems.

Autoplay #

The autoplay attribute controls whether the browser downloads and plays a video immediately. The precise way it works depends on the platform and browser.

Chrome: Depends on multiple factors including but not limited to whether the viewing is on desktop and whether the mobile user has added your site or app to their homescreen. For details, see Autoplay best practices.
Firefox: Blocks all video and sound, but gives users the ability to relax these restrictions for either all sites or particular sites. For details, see Allow or block media autoplay in Firefox
Safari: Has historically required a user gesture, but has been relaxing that requirement in recent versions. For details, see New <video> Policies for iOS.

Even on platforms where autoplay is possible, you need to consider whether it's a good idea to enable it:

Data usage can be expensive.
Playing media before the user wants it can hog bandwidth and CPU, and thereby delay page rendering.
Users may be in a context where playing video or audio is intrusive.

Preload #

The preload attribute provides a hint to the browser as to how much information or content to preload.

Value	Description
`none`	The user might chose not to watch the video, so don't preload anything.
`metadata`	Metadata (duration, dimensions, text tracks) should be preloaded, but with minimal video.
`auto`	Downloading the entire video right away is considered desirable. An empty string produces the same result.

The preload attribute has different effects on different platforms. For example, Chrome buffers 25 seconds of video on desktop but none on iOS or Android. This means that on mobile, there may be playback startup delays that don't happen on desktop. See Fast playback with audio and video preload or Steve Souders' blog for more details.

Now that you know how to add media to your web page it's time to learn about Media accessibility where you will add captions to your video for hearing impaired, or when playing the audio is not a viable option.

Content security policy

2012-06-15T00:00:00Z

The web's security model is rooted in the same-origin policy. Code from https://mybank.com should only have access to https://mybank.com's data, and https://evil.example.com should certainly never be allowed access. Each origin is kept isolated from the rest of the web, giving developers a safe sandbox in which to build and play. In theory, this is perfectly brilliant. In practice, attackers have found clever ways to subvert the system.

Cross-site scripting (XSS) attacks, for example, bypass the same origin policy by tricking a site into delivering malicious code along with the intended content. This is a huge problem, as browsers trust all of the code that shows up on a page as being legitimately part of that page's security origin. The XSS Cheat Sheet is an old but representative cross-section of the methods an attacker might use to violate this trust by injecting malicious code. If an attacker successfully injects any code at all, it's pretty much game over: user session data is compromised and information that should be kept secret is exfiltrated to The Bad Guys. We'd obviously like to prevent that if possible.

This overview highlights a defense that can significantly reduce the risk and impact of XSS attacks in modern browsers: Content Security Policy (CSP).

Summary #

Use allowlists to tell the client what's allowed and what isn't.
Learn what directives are available.
Learn the keywords they take.
Inline code and eval() are considered harmful.
Report policy violations to your server before enforcing them.

Source allowlists #

The issue exploited by XSS attacks is the browser's inability to distinguish between script that's part of your application and script that's been maliciously injected by a third-party. For example, the Google +1 button at the bottom of this page loads and executes code from https://apis.google.com/js/plusone.js in the context of this page's origin. We trust that code, but we can't expect the browser to figure out on its own that code from apis.google.com is awesome, while code from apis.evil.example.com probably isn't. The browser happily downloads and executes any code a page requests, regardless of source.

Instead of blindly trusting everything that a server delivers, CSP defines the Content-Security-Policy HTTP header, which allows you to create an allowlist of sources of trusted content, and instructs the browser to only execute or render resources from those sources. Even if an attacker can find a hole through which to inject script, the script won't match the allowlist, and therefore won't be executed.

Since we trust apis.google.com to deliver valid code, and we trust ourselves to do the same, let's define a policy that only allows script to execute when it comes from one of those two sources:

Content-Security-Policy: script-src 'self' https://apis.google.com

Simple, right? As you probably guessed, script-src is a directive that controls a set of script-related privileges for a specific page. We've specified 'self' as one valid source of script, and https://apis.google.com as another. The browser dutifully downloads and executes JavaScript from apis.google.com over HTTPS, as well as from the current page's origin.

With this policy defined, the browser simply throws an error instead of loading script from any other source. When a clever attacker manages to inject code into your site, they'll run headlong into an error message rather than the success they were expecting.

Policy applies to a wide variety of resources #

While script resources are the most obvious security risks, CSP provides a rich set of policy directives that enable fairly granular control over the resources that a page is allowed to load. You've already seen script-src, so the concept should be clear.

Let's quickly walk through the rest of the resource directives. The list below represents the state of the directives as of level 2. A level 3 spec has been published, but is largely unimplemented in the major browsers.

base-uri restricts the URLs that can appear in a page's <base> element.
child-src lists the URLs for workers and embedded frame contents. For example: child-src https://youtube.com would enable embedding videos from YouTube but not from other origins.
connect-src limits the origins that you can connect to (via XHR, WebSockets, and EventSource).
font-src specifies the origins that can serve web fonts. Google's web fonts could be enabled via font-src https://themes.googleusercontent.com.
form-action lists valid endpoints for submission from <form> tags.
frame-ancestors specifies the sources that can embed the current page. This directive applies to <frame>, <iframe>, <embed>, and <applet> tags. This directive can't be used in <meta> tags and applies only to non-HTML resources.
frame-src was deprecated in level 2, but is restored in level 3. If not present it still falls back to child-src as before.
img-src defines the origins from which images can be loaded.
media-src restricts the origins allowed to deliver video and audio.
object-src allows control over Flash and other plugins.
plugin-types limits the kinds of plugins a page may invoke.
report-uri specifies a URL where a browser will send reports when a content security policy is violated. This directive can't be used in <meta> tags.
style-src is script-src's counterpart for stylesheets.
upgrade-insecure-requests instructs user agents to rewrite URL schemes, changing HTTP to HTTPS. This directive is for websites with large numbers of old URL's that need to be rewritten.
worker-src is a CSP Level 3 directive that restricts the URLs that may be loaded as a worker, shared worker, or service worker. As of July 2017, this directive has limited implementations.

By default, directives are wide open. If you don't set a specific policy for a directive, let's say font-src, then that directive behaves by default as though you'd specified * as the valid source (for example, you could load fonts from anywhere, without restriction).

You can override this default behavior by specifying a default-src directive. This directive defines the defaults for most directives that you leave unspecified. Generally, this applies to any directive that ends with -src. If default-src is set to https://example.com, and you fail to specify a font-src directive, then you can load fonts from https://example.com, and nowhere else. We specified only script-src in our earlier examples, which means that images, fonts, and so on can be loaded from any origin.

The following directives don't use default-src as a fallback. Remember that failing to set them is the same as allowing anything.

base-uri
form-action
frame-ancestors
plugin-types
report-uri
sandbox

You can use as many or as few of these directives as makes sense for your specific application, simply listing each in the HTTP header, separating directives with semicolons. Make sure that you list all required resources of a specific type in a single directive. If you wrote something like script-src https://host1.com; script-src https://host2.com the second directive would simply be ignored. Something like the following would correctly specify both origins as valid:

script-src https://host1.com https://host2.com

If, for example, you have an application that loads all of its resources from a content delivery network (say, https://cdn.example.net), and know that you don't need any framed content or plugins, then your policy might look something like the following:

Content-Security-Policy: default-src https://cdn.example.net; child-src 'none'; object-src 'none'

Implementation details #

You will see X-WebKit-CSP and X-Content-Security-Policy headers in various tutorials on the web. Going forward, you should ignore these prefixed headers. Modern browsers (with the exception of IE) support the unprefixed Content-Security-Policy header. That's the header you should use.

Regardless of the header you use, policy is defined on a page-by-page basis: you'll need to send the HTTP header along with every response that you'd like to ensure is protected. This provides a lot of flexibility, as you can fine-tune the policy for specific pages based on their specific needs. Perhaps one set of pages in your site has a +1 button, while others don't: you could allow the button code to be loaded only when necessary.

The source list in each directive is flexible. You can specify sources by scheme (data:, https:), or ranging in specificity from hostname-only (example.com, which matches any origin on that host: any scheme, any port) to a fully qualified URI (https://example.com:443, which matches only HTTPS, only example.com, and only port 443). Wildcards are accepted, but only as a scheme, a port, or in the leftmost position of the hostname: *://*.example.com:* would match all subdomains of example.com (but not example.com itself), using any scheme, on any port.

The source list also accepts four keywords:

'none', as you might expect, matches nothing.
'self' matches the current origin, but not its subdomains.
'unsafe-inline' allows inline JavaScript and CSS. (We'll touch on this in more detail in a bit.)
'unsafe-eval' allows text-to-JavaScript mechanisms like eval. (We'll get to this too.)

These keywords require single-quotes. For example, script-src 'self' (with quotes) authorizes the execution of JavaScript from the current host; script-src self (no quotes) allows JavaScript from a server named "self" (and not from the current host), which probably isn't what you meant.

Sandboxing #

There's one more directive worth talking about: sandbox. It's a bit different from the others we've looked at, as it places restrictions on actions that the page can take rather than on resources that the page can load. If the sandbox directive is present, the page is treated as though it was loaded inside of an <iframe> with a sandbox attribute. This can have a wide range of effects on the page: forcing the page into a unique origin, and preventing form submission, among others. It's a bit beyond the scope of this article, but you can find full details on valid sandboxing attributes in the "Sandboxing" section of the HTML5 spec.

The meta tag #

CSPs preferred delivery mechanism is an HTTP header. It can be useful, however, to set a policy on a page directly in the markup. Do that using a <meta> tag with an http-equiv attribute:

<meta http-equiv="Content-Security-Policy" content="default-src https://cdn.example.net; child-src 'none'; object-src 'none'">

This can't be used for frame-ancestors, report-uri, or sandbox.

Inline code is considered harmful #

It should be clear that CSP is based on allowlist origins, as that's an unambiguous way of instructing the browser to treat specific sets of resources as acceptable and to reject the rest. Origin-based allowlists don't, however, solve the biggest threat posed by XSS attacks: inline script injection. If an attacker can inject a script tag that directly contains some malicious payload (<script>sendMyDataToEvilDotCom()</script>), the browser has no mechanism by which to distinguish it from a legitimate inline script tag. CSP solves this problem by banning inline script entirely: it's the only way to be sure.

This ban includes not only scripts embedded directly in script tags, but also inline event handlers and javascript: URLs. You'll need to move the content of script tags into an external file, and replace javascript: URLs and <a ... onclick="[JAVASCRIPT]"> with appropriate addEventListener() calls. For example, you might rewrite the following from:

<script>
    function doAmazingThings() {
    alert('YOU AM AMAZING!');
    }
</script>
<button onclick='doAmazingThings();'>Am I amazing?</button>

to something more like:

<!-- amazing.html -->
<script src='amazing.js'></script>
<button id='amazing'>Am I amazing?</button>

// amazing.js
function doAmazingThings() {
    alert('YOU AM AMAZING!');
}
document.addEventListener('DOMContentLoaded', function () {
    document.getElementById('amazing')
    .addEventListener('click', doAmazingThings);
});

The rewritten code has a number of advantages above and beyond working well with CSP; it's already best practice, regardless of your use of CSP. Inline JavaScript mixes structure and behavior in exactly the way you shouldn't. External resources are easier for browsers to cache, more understandable for developers, and conducive to compilation and minification. You'll write better code if you do the work to move code into external resources.

Inline style is treated in the same way: both the style attribute and style tags should be consolidated into external stylesheets to protect against a variety of surprisingly clever data exfiltration methods that CSS enables.

If you must have inline script and style, you can enable it by adding 'unsafe-inline' as an allowed source in a script-src or style-src directive. You can also use a nonce or a hash (see below), but you really shouldn't. Banning inline script is the biggest security win CSP provides, and banning inline style likewise hardens your application. It's a little bit of effort up front to ensure that things work correctly after moving all the code out-of-line, but that's a tradeoff that's well worth making.

If you absolutely must use it #

CSP Level 2 offers backward compatibility for inline scripts by allowing you to add specific inline scripts to the allowlist using either a cryptographic nonce (number used once) or a hash. Although this may be cumbersome, it is useful in a pinch.

To use a nonce, give your script tag a nonce attribute. Its value must match one in the list of trusted sources. For example:

<script nonce="EDNnf03nceIOfn39fn3e9h3sdfa">
    // Some inline code I can't remove yet, but need to asap.
</script>

Now, add the nonce to your script-src directive appended to the nonce- keyword.

Content-Security-Policy: script-src 'nonce-EDNnf03nceIOfn39fn3e9h3sdfa'

Remember that nonces must be regenerated for every page request and they must be unguessable.

Hashes work in much the same way. Instead of adding code to the script tag, create a SHA hash of the script itself and add it to the script-src directive. For example, let's say your page contained this:

<script>alert('Hello, world.');</script>

Your policy would contain this:

Content-Security-Policy: script-src 'sha256-qznLcsROx4GACP2dm0UCKCzCG-HiZ1guq6ZZDob_Tng='

There are a few things to note here. The sha*- prefix specifies the algorithm that generates the hash. In the example above, sha256- is used. CSP also supports sha384- and sha512-. When generating the hash, do not include the <script> tags. Also capitalization and whitespace matter, including leading or trailing whitespace.

A Google search on generating SHA hashes will lead you to solutions in any number of languages. Using Chrome 40 or later, you can open DevTools and then reload your page. The Console tab will contain error messages with the correct sha256 hash for each of your inline scripts.

Eval too #

Even when an attacker can't inject script directly, they might be able to trick your application into converting otherwise inert text into executable JavaScript and executing it on their behalf. eval(), new Function(), setTimeout([string], …), and setInterval([string], ...) are all vectors through which injected text might end up executing something unexpectedly malicious. CSP's default response to this risk is to completely block all of these vectors.

This has more than a few impacts on the way you build applications:

You must parse JSON via the built-in JSON.parse, rather than relying on eval. Native JSON operations are available in every browser since IE8, and they're completely safe.
Rewrite any setTimeout or setInterval calls you're currently making with inline functions rather than strings. For example:

setTimeout("document.querySelector('a').style.display = 'none';", 10);

would be better written as:

setTimeout(function () {
    document.querySelector('a').style.display = 'none';
}, 10);

Avoid inline templating at runtime: Many templating libraries use new Function() liberally to speed up template generation at runtime. It's a nifty application of dynamic programming, but comes at the risk of evaluating malicious text. Some frameworks support CSP out of the box, falling back to a robust parser in the absence of eval. AngularJS's ng-csp directive is a good example of this.

However, a better choice would be a templating language that offers precompilation (Handlebars does, for instance). Precompiling your templates can make the user experience even faster than the fastest runtime implementation, and it's safer too. If eval and its text-to-JavaScript brethren are essential to your application, you can enable them by adding 'unsafe-eval' as an allowed source in a script-src directive, but we strongly discourage this. Banning the ability to execute strings makes it much more difficult for an attacker to execute unauthorized code on your site.

Reporting #

CSP's ability to block untrusted resources client-side is a huge win for your users, but it would be quite helpful to have some sort of notification sent back to the server so that you can identify and squash any bugs that allow malicious injection in the first place. To this end, you can instruct the browser to POST JSON-formatted violation reports to a location specified in a report-uri directive.

Content-Security-Policy: default-src 'self'; ...; report-uri /my_amazing_csp_report_parser;

Those reports will look something like the following:

{
    "csp-report": {
    "document-uri": "http://example.org/page.html",
    "referrer": "http://evil.example.com/",
    "blocked-uri": "http://evil.example.com/evil.js",
    "violated-directive": "script-src 'self' https://apis.google.com",
    "original-policy": "script-src 'self' https://apis.google.com; report-uri http://example.org/my_amazing_csp_report_parser"
    }
}

This contains a good chunk of information that will help you track down the specific cause of the violation, including the page on which the violation occurred (document-uri), that page's referrer (note that unlike the HTTP header field, the key is not misspelled), the resource that violated the page's policy (blocked-uri), the specific directive it violated (violated-directive), and the page's complete policy (original-policy).

Report-Only #

If you're just starting out with CSP, it makes sense to evaluate the current state of your application before rolling out a draconian policy to your users. As a stepping stone to a complete deployment, you can ask the browser to monitor a policy, reporting violations but not enforcing the restrictions. Instead of sending a Content-Security-Policy header, send a Content-Security-Policy-Report-Only header.

Content-Security-Policy-Report-Only: default-src 'self'; ...; report-uri /my_amazing_csp_report_parser;

The policy specified in report-only mode won't block restricted resources, but it will send violation reports to the location you specify. You can even send both headers, enforcing one policy while monitoring another. This is a great way to evaluate the effect of changes to your application's CSP: turn on reporting for a new policy, monitor the violation reports and fix any bugs that turn up; when you're satisfied with its effect, start enforcing the new policy.

Real World Usage #

CSP 1 is quite usable in Chrome, Safari, and Firefox, but has very limited support in IE 10. You can view specifics at caniuse.com. CSP Level 2 has been available in Chrome since version 40. Massive sites like Twitter and Facebook have deployed the header (Twitter's case study is worth a read), and the standard is very much ready for you to start deploying on your own sites.

The first step towards crafting a policy for your application is to evaluate the resources you're actually loading. Once you think you have a handle on how things are put together in your app, set up a policy based on those requirements. Let's walk through a few common use cases and determine how we'd best be able to support them within the protective confines of CSP.

Facebook's Like button has a number of implementation options. We recommend sticking with the <iframe> version as it's safely sandboxed from the rest of your site. It requires a child-src https://facebook.com directive to function properly. Note that, by default, the <iframe> code that Facebook provides loads a relative URL, //facebook.com. Change that to explicitly specify HTTPS: https://facebook.com. There's no reason to use HTTP if you don't have to.
Twitter's Tweet button relies on access to a script and a frame, both hosted at https://platform.twitter.com. (Twitter likewise provides a relative URL by default; edit the code to specify HTTPS when copy/pasting it locally.) You'll be all set with script-src https://platform.twitter.com; child-src https://platform.twitter.com, as long as you move the JavaScript snippet that Twitter provides out into an external JavaScript file.
Other platforms have similar requirements, and can be addressed similarly. We suggest just setting a default-src of 'none', and watching your console to determine which resources you'll need to enable to make the widgets work.

Including multiple widgets is straightforward: simply combine the policy directives, remembering to merge all resources of a single type into a single directive. If you wanted all three social media widgets, the policy would look like this:

script-src https://apis.google.com https://platform.twitter.com; child-src https://plusone.google.com https://facebook.com https://platform.twitter.com

Use case #2: lockdown #

Assume for a moment that you run a banking site and want to make sure that only those resources you've written yourself can be loaded. In this scenario, start with a default policy that blocks absolutely everything (default-src 'none'), and build up from there.

Let's say the bank loads all images, style, and script from a CDN at https://cdn.mybank.net, and connects via XHR to https://api.mybank.com/ to pull various bits of data down. Frames are used, but only for pages local to the site (no third-party origins). There's no Flash on the site, no fonts, no extras. The most restrictive CSP header that we could send is this:

Content-Security-Policy: default-src 'none'; script-src https://cdn.mybank.net; style-src https://cdn.mybank.net; img-src https://cdn.mybank.net; connect-src https://api.mybank.com; child-src 'self'

Use case #3: SSL only #

A wedding-ring discussion forum admin wants to ensure that all resources are only loaded via secure channels, but doesn't really write much code; rewriting large chunks of the third-party forum software that's filled to the brim with inline script and style is beyond his abilities. The following policy would be effective:

Content-Security-Policy: default-src https:; script-src https: 'unsafe-inline'; style-src https: 'unsafe-inline'

Even though https: is specified in default-src, the script and style directives don't automatically inherit that source. Each directive completely overwrites the default for that specific type of resource.

The future #

Content Security Policy Level 2 is a Candidate Recommendation. The W3C's Web Application Security Working Group has already begun work on the specification's next iteration, Content Security Policy Level 3.

If you're interested in the discussion around these upcoming features, skim the public-webappsec@ mailing list archives, or join in yourself.

Joe Medley on web.dev

How to copy text

The modern way #

Using the Async Clipboard API #

The classic way #

Using document.execCommand() #

Progressive enhancement #

Further reading #

Demo #

Media encryption

Clear Key encryption #

Create a key #

Create an IV #

Encrypt with Clear Key #

Create a key information file #

Encrypt for HLS #

Widevine encryption #

Encrypt with Widevine #

Media conversion sequence #

DASH/WebM #

DASH/MP4 #

HLS/MP4 #

Introducing 1.1

What's new since last year? #

Augmented reality #

Camera controls #

Annotations #

Editor #

Rendering and performance improvements #

Stability #

What's next? #

Media accessibility

Add <track> tags #

WebVTT file structure #

Kinds of text tracks #

Add media to a web page

Media file basics

Serving video files #

Containers and codecs, and streams? #

Codec formats #

Bitrate and resolution #

Positioning virtual objects in real-world views

The immersive AR session sample #

Create the reticle #

Request a session #

Entering a session #

Multiple reference spaces #

Running a frame loop #

Placing an object #

Conclusion #

Virtual reality comes to the web, part II

What this article is not #

The players and the game #

The players #

XRViewerPose #

XRView #

XRWebGLLayer #

WebGLFramebuffer #

XRViewport #

WebGLRenderingContext #

The game #

Get the viewer's pose #

A short detour #

Pass ('bind') the WebGLFramebuffer #

Iterate over each XRView object #

Pass the XRViewport object to the WebGLRenderingContext #

The webGLRenContext #

Draw something to the framebuffer #

Conclusion #

Augmented reality: You may already know it

What's it useful for? #

Requesting a session #

A convenience property #

Entering a session #

Reference spaces #

Running a frame loop #

Conclusion #

Integrate PWAs into built-in sharing UIs with Workbox

On the same page #

Service worker share targets with Workbox #

Using `document.execCommand()` #

Add `<track>` tags #