Sam Dutton on web.dev

2021 Scroll Survey Report

2021-08-02T00:00:00Z

In April, the Chrome team released a scroll and touch-action survey based on top reported issues from the 2019 MDN Web DNA Report. The 2021 Scroll Survey Report is ready, and the Chrome team would like to share some thoughts and action items we've gleaned from the survey results. We hope these results will help browser vendors and standards groups understand how to improve web scrolling.

View the 2021 Scroll Survey Report.

Noteworthy results #

The survey anonymously collected 880 submissions, with 366 answering every question.

While getting started with scrolling is one line of CSS, like overflow-x: scroll;, the surface area of scroll APIs and options is large, spanning JavaScript to CSS. The following results help to highlight the issues web developers are encountering.

Overall satisfaction with web scrolling #

Question 27

45_%

are somewhat or extremely dissatisfied overall
with web scrolling.

This question was placed near the end of the survey intentionally, after questions on 26 scroll use cases and features. From the response, it's clear that the web community struggles with scroll. Almost half of the respondents report a level of overall dissatisfaction.

We believe overall sentiment about working with scroll should not be this low. This metric needs to be changed; it's a clear signal there's work to be done.

Difficulty working with scroll #

Question 2

43_%

reported it's somewhat or
extremely difficult
to work with scrolling.

From our research, these difficulties come from the multitude of use cases for scroll. When we talk about scrolling, that might include:

Missing browser features, complex JavaScript, and the need to support input modes including touch, keyboard, and gamepads, make all of these things harder.

Importance of touch interactions #

Question 3

51_%

report touch interactions as
very or extremely important
to their work.

With mobile web users still on the rise in visit statistics, it wasn't surprising to see half of the respondents report that touch is very important to their work on the web. This signaled that web features like CSS scroll snap and touch-action need extra attention so the web can deliver on high-quality touch interaction.

Question 5a

44_%

report somewhat or extremely difficult
to do gamepad and tab navigation.

Scrolling includes navigation methods such as keyboard arrows, tab keys, spacebar presses, and gamepads, and it can be difficult to include these when doing custom scroll work. Almost half of the respondents report it's somewhat or extremely difficult to include these inputs.

Learning `touch-action` #

Question 9

50_%

report learning about
`touch-action: manipulation`
from the survey.

Some of the survey questions asked about using certain APIs with a possible answer of Yes, No, or "today I learned." One notable piece of feedback was the number of people who reported learning about touch-action from the survey, as it's a critical property when building custom touch gestures that need to interact within scroll.

Cyclical scrolling #

Question 27

58_%

report sometimes, often or on every project
using cyclical scrolling.

The video shows cyclical seconds scrolling,
after 60 seconds it begins at 0 again.

Those numbers are high for a scrolling feature with little or no support provided by the web platform. The feature often incurs high amounts of technical debt because of this, with duplication or JavaScript injected to force the effect. It's popular for product carousels and when selecting time in seconds or minutes to offer cyclical scrolling.

Are scrollable areas important #

Question 2

55_%

very or
extremely important

16_%

report not at all
or slightly important

Respondents felt strongly about the importance of scrollable areas, giving another signal about the struggles required to deliver high-quality scrolling.

Carousels #

Question 20

87_%

have used carousels.

24_%

report they're
easy to manage.

Nearly every respondent delivers carousels in their web work, while only 25% find them easy to manage. Off-the-shelf carousels were popular during our research, but this statistic surprised us, as it doesn't sound very solved.

Infinite scroll #

Question 22

65_%

use it sometimes
to every project

60_%

somewhat or
extremely difficult.

Two-thirds of respondents deliver infinite scroll in their web work, and an equal amount report it's difficult to do. Another example of high usage paired with high difficulty, which indicates to us an area needing attention.

While content-visibility and contain-intrinsic-size can be combined to reduce render costs for long scrollable areas, it doesn't seem to be helping with "load more" infinite scroll UX.

Scroll-linked or scroll-triggered animations #

Question 24

47_%

use it sometimes
to every project

56_%

report somewhat or
extremely difficult

Almost half of all respondents use scroll-orchestrated animations and half the respondents find it difficult, once again linking high usage with difficulty.

Compete with built-in scrolling #

Question 26

32_%

always or
most of the time

50_%

sometimes

The built-in scroll and touch interactions of phone and tablet applications are often touted as a clear place where the web can catch up. The features include scroll-linked animations, programmatic interfaces, voice integration, scroll hints, and pull-to-refresh APIs.

Just half of the respondents felt it was only sometimes possible to match the experience of built-in scrolling.

Overall satisfaction building scroll interactions on the web #

Question 27

Survey Takeaways #

The survey results are segmented into four categories: compatibility, education, APIs, and features.

Compatibility #

The Chrome team has declared a goal to decrease the number of web compatibility issues, including scroll compatibility.

The first three compatibility issues to focus on:

Horizontal scrolling compatibility.
overscroll-behavior cross browser.
Removing prefixes from -webkit-scrollbar and following the standard.

Education #

The survey results showed that there needs to be more education around touch-action and logical properties. The browser is at the forefront of international layout, and it's apparent it's underutilized or misunderstood.

Areas to focus on:

APIs #

Usage of scroll snapping is growing, and developers have responded that they want to use features interoperably with popular libraries and plugins. Shrinking this gap between CSS and plugin libraries will help the satisfaction of scroll snap developer and user experience.

We'll focus API work on scroll-snap:

API availability and compatibility across browsers.
Begin work on new CSS APIs like scroll-start.
Begin work on new JS events like snapChanged().

Features #

The survey results showed that users struggle with some specific types of scroll-related components on the web, as the platform doesn't provide the primitives they need to build them without plugins or a high level of effort. This is an area that we hope to explore more deeply.

The features developers struggle to build include:

Carousels
Virtual scroll
Infinite scroll

Resources #

Thumbnail image: photo by Taylor Wilcox on Unsplash.

Take the 2021 scroll survey to help improve scrolling on the web

2021-04-15T00:00:00Z

We've been analyzing the results of the 2019 Mozilla Developer Network Web DNA Report for action items and follow up strategies to improve the top reported issues. A small sub-group on the Chrome team identified a group of issues related to scrolling, scroll-snap, and touch-action. Those issues were researched, evaluated, and synthesized into a new, follow-up survey with more precise questions about issues related to these topics.

2021 Scroll Survey #

We believe that there are many ways to improve scrolling on the web for developers, designers, and users alike. We've created a survey which we hope is respectful of your time. It should take no more than 10 minutes to complete. The results will help browser vendors and standards groups understand how to make scrolling better.

You can take part in the survey at 2021 Web Scrolling Survey.

What is FLoC?

2021-03-30T00:00:00Z

FLoC provides a privacy-preserving mechanism for interest-based ad selection.

As a user moves around the web, their browser uses the FLoC algorithm to work out its "interest cohort", which will be the same for thousands of browsers with a similar recent browsing history. The browser recalculates its cohort periodically, on the user's device, without sharing individual browsing data with the browser vendor or anyone else.

Advertisers (sites that pay for advertisements) can include code on their own websites in order to gather and provide cohort data to their adtech platforms (companies that provide software and tools to deliver advertising). For example, an adtech platform might learn from an online shoe store that browsers from cohorts 1101 and 1354 seem interested in the store's hiking gear. From other advertisers, the adtech platform learns about other interests of those cohorts.

Subsequently, the ad platform can use this data to select relevant ads (such as an ad for hiking boots from the shoe store) when a browser from one of those cohorts requests a page from a site that displays ads, such as a news website.

The Privacy Sandbox is a series of proposals to satisfy third-party use cases without third-party cookies or other tracking mechanisms. See Digging into the Privacy Sandbox for an overview of all the proposals.

If you have comments on this proposal, create an issue on the FLoC Explainer repository. If you have feedback on Chrome's experiment with this proposal, post a reply on the Intent to Experiment.

Why do we need FLoC? #

Many businesses rely on advertising to drive traffic to their sites, and many publisher websites fund content by selling advertising inventory. People generally prefer to see ads that are relevant and useful to them, and relevant ads also bring more business to advertisers and more revenue to the websites that host them. In other words, ad space is more valuable when it displays relevant ads. Thus, selecting relevant ads increases revenue for ad-supported websites. That, in turn, means that relevant ads help fund content creation that benefits users.

However, people are concerned about the privacy implications of tailored advertising, which currently relies on techniques such as tracking cookies and device fingerprinting which can reveal your browsing history across sites to advertisers or ad platforms. The FLoC proposal aims to allow ad selection in a way that better protects privacy.

What can FLoC be used for? #

Show ads to people whose browsers belong to a cohort that has been observed to frequently visit an advertiser's site or shows interest in relevant topics.
Use machine learning models to predict the probability a user will convert based on their cohort, in order to inform ad auction bidding behavior.
Recommend content to users. For example, suppose a news site observes that their sports podcast page has become especially popular with visitors from cohorts 1234 and 7. They can recommend that content to other visitors from those cohorts.

How does FLoC work? #

The example below describes the different roles in selecting an ad using FLoC.

The advertiser (a company that pays for advertising) in this example is an online shoe retailer:
shoestore.example
The publisher (a site that sells ad space) in the example is a news site:
dailynews.example
The adtech platform (which provides software and tools to deliver advertising) is:
adnetwork.example

In this example we've called the users Yoshi and Alex. Initially their browsers both belong to the same cohort, 1354.

1. FLoC service #

The FLoC service used by the browser creates a mathematical model with thousands of "cohorts", each of which will correspond to thousands of web browsers with similar recent browsing histories. More about how this works below.
Each cohort is given a number.

2. Browser #

From the FLoC service, Yoshi's browser gets data describing the FLoC model.
Yoshi's browser works out its cohort by using the FLoC model's algorithm to calculate which cohort corresponds most closely to its own browsing history. In this example, that will be the cohort 1354. Note that Yoshi's browser does not share any data with the FLoC service.
In the same way, Alex's browser calculates its cohort ID. Alex's browsing history is different from Yoshi's, but similar enough that their browsers both belong to cohort 1354.

3. Advertiser: shoestore.example #

Yoshi visits shoestore.example.
The site asks Yoshi's browser for its cohort: 1354.
Yoshi looks at hiking boots.
The site records that a browser from cohort 1354 showed interest in hiking boots.
The site later records additional interest in its products from cohort 1354, as well as from other cohorts.
The site periodically aggregates and shares information about cohorts and product interests with its adtech platform adnetwork.example.

Now it's Alex's turn.

4. Publisher: dailynews.example #

Alex visits dailynews.example.
The site asks Alex's browser for its cohort.
The site then makes a request for an ad to its adtech platform, adnetwork.example, including Alex's browser's cohort: 1354.

5. Adtech platform: adnetwork.example #

adnetwork.example can select an ad suitable for Alex by combining the data it has from the publisher dailynews.example and the advertiser shoestore.example:

Alex's browser's cohort (1354) provided by dailynews.example.
Data about cohorts and product interests from shoestore.example: "Browsers from cohort 1354 might be interested in hiking boots."

adnetwork.example selects an ad appropriate to Alex: an ad for hiking boots on shoestore.example.
dailynews.example displays the ad 🥾.

Who runs the back-end service that creates the FLoC model? #

Every browser vendor will need to make their own choice of how to group browsers into cohorts. Chrome is running its own FLoC service; other browsers might choose to implement FLoC with a different clustering approach, and would run their own service to do so.

How does the FLoC service enable the browser to work out its cohort? #

The FLoC service used by the browser creates a multi-dimensional mathematical representation of all potential web browsing histories. We'll call this model "cohort space".
The service divides up this space into thousands of segments. Each segment represents a cluster of thousands of similar browsing histories. These groupings aren't based on knowing any actual browsing histories; they're simply based on picking random centers in "cohort space" or cutting up the space with random lines.
Each segment is given a cohort number.
The web browser gets this data describing "cohort space" from its FLoC service.
As a user moves around the web, their browser uses an algorithm to periodically calculate the region in "cohort space" that corresponds most closely to its own browsing history.

The FLoC service divides up "cohort space" into thousands of segments (only a few are shown here).

Can a browser's cohort change? #

Yes! A browser's cohort definitely can change! You probably don't visit the same websites every week, and your browser's cohort will reflect that.

A cohort represents a cluster of browsing activity, not a collection of people. The activity characteristics of a cohort are generally consistent over time, and cohorts are useful for ad selection because they group similar recent browsing behavior. Individual people's browsers will float in and out of a cohort as their browsing behavior changes. Initially, we expect the browser to recalculate its cohort every seven days.

In the example above, both Yoshi and Alex's browser's cohort is 1354. In the future, Yoshi's browser and Alex's browser may move to a different cohort if their interests change. In the example below, Yoshi's browser moves to cohort 1101 and Alex's browser moves to cohort 1378. Other people's browsers will move into and out of cohorts as their browsing interests change.

Yoshi's and Alex's browser cohort may change if their interests change.

How does the browser work out its cohort? #

As described above, the user's browser gets data from its FLoC service that describes the mathematical model for cohorts: a multi-dimensional space that represents the browsing activity of all users. The browser then uses an algorithm to work out which region of this "cohort space" (that is, which cohort) most closely matches its own recent browsing behavior.

How does FLoC work out the right size of cohort? #

There will be thousands of browsers in each cohort.

A smaller cohort size might be more useful for personalizing ads, but is less likely to stop user tracking—and vice versa. A mechanism for assigning browsers to cohorts needs to make a trade off between privacy and utility. The Privacy Sandbox uses k-anonymity to allow a user to "hide in a crowd". A cohort is k-anonymous if it is shared by at least k users. The higher the k number, the more privacy-preserving the cohort.

Can FLoC be used to group people based on sensitive categories? #

The clustering algorithm used to construct the FLoC cohort model is designed to evaluate whether a cohort may be correlated with sensitive categories, without learning why a category is sensitive. Cohorts that might reveal sensitive categories such as race, sexuality, or medical history will be blocked. In other words, when working out its cohort, a browser will only be choosing between cohorts that won't reveal sensitive categories.

Is FLoC just another way of categorizing people online? #

With FLoC, a user's browser will belong to one of thousands of cohorts, along with thousands of other users' browsers. Unlike with third-party cookies and other targeting mechanisms, FLoC only reveals the cohort a user's browser is in, and not an individual user ID. It does not enable others to distinguish an individual within a cohort. In addition, the information about browsing activity that is used to work out a browser's cohort is kept local on the browser or device, and is not uploaded elsewhere. The browser may further leverage other anonymization methods, such as differential privacy.

Websites will have the ability to opt in or out of FLoC, so sites about sensitive topics will be able to prevent visits to their site from being included in the FLoC calculation. As additional protection, analysis by the FLoC service will evaluate whether a cohort may reveal sensitive information about users without learning why that cohort is sensitive. If a cohort might represent a greater-than-typical number of people who visit sites in a sensitive category, that entire cohort is removed. Negative financial status and mental health are among the sensitive categories covered by this analysis.

Websites can exclude a page from the FLoC calculation by setting a Permissions-Policy header interest-cohort=() for that page. For pages that haven't been excluded, a page visit will be included in the browser's FLoC calculation if document.interestCohort() is used on the page. During the current FLoC origin trial, a page will also be included in the calculation if Chrome detects that the page loads ads or ads-related resources. (Ad Tagging in Chromium explains how Chrome's ad detection mechanism works.)

Pages served from private IP addresses, such as intranet pages, won't be part of the FLoC computation.

How does the FLoC JavaScript API work? #

The FLoC API is very simple: just a single method that returns a promise that resolves to an object providing the cohort id and version:

const { id, version } = await document.interestCohort();
console.log('FLoC ID:', id);
console.log('FLoC version:', version);

The cohort data made available looks like this:

{
  id: "14159",
  version: "chrome.2.1"
}

The version value enables sites using FLoC to know which browser and which FLoC model the cohort ID refers to. As described below, the promise returned by document.interestCohort() will reject for any frame that is not allowed the interest-cohort permission.

Can websites opt out of being included in the FLoC computation? #

The interest-cohort permissions policy enables a site to declare that it does not want to be included in the user's list of sites for cohort calculation. The policy will be allow by default. The promise returned by document.interestCohort() will reject for any frame that is not allowed interest-cohort permission. If the main frame does not have the interest-cohort permission, then the page visit will not be included in the interest cohort calculation.

For example, a site can opt out of all FLoC cohort calculation by sending the following HTTP response header:

  Permissions-Policy: interest-cohort=()

Can a user stop sites from getting their browser's FLoC cohort? #

If a user disables Privacy Sandbox in chrome://settings/privacySandbox, the browser will not provide the user's cohort when asked for it via JavaScript: the promise returned by document.interestCohort() will reject.

How can I make suggestions or provide feedback? #

If you have comments on the API, create an issue on the FLoC Explainer repository.

Find out more #

Photo by Rhys Kentish on Unsplash.

Payment and address form best practices

2020-12-09T00:00:00Z

Well-designed forms help users and increase conversion rates. One small fix can make a big difference!

Here is an example of a simple payment form that demonstrates all of the best practices:

Here is an example of a simple address form that demonstrates all of the best practices:

Checklist #

Use meaningful HTML elements: <form>, <input>, <label>, and <button>.
Label each form field with a <label>.
Use HTML element attributes to access built-in browser features, in particular type and autocomplete with appropriate values.
Avoid using type="number" for numbers that aren't meant to be incremented, such as payment card numbers. Use type="text" and inputmode="numeric" instead.
If an appropriate autocomplete value is available for an input, select, or textarea, you should use it.
To help browsers autofill forms, give input name and id attributes stable values that don't change between page loads or website deployments.
Disable submit buttons once they've been tapped or clicked.
Validate data during entry—not just on form submission.
Make guest checkout the default and make account creation simple once checkout is complete.
Show progress through the checkout process in clear steps with clear calls to action.
Limit potential checkout exit points by removing clutter and distractions.
Show full order details at checkout and make order adjustments easy.
Don't ask for data you don't need.
Ask for names with a single input unless you have a good reason not to.
Don't enforce Latin-only characters for names and usernames.
Allow for a variety of address formats.
Consider using a single textarea for address.
Use autocomplete for billing address.
Internationalize and localize where necessary.
Consider avoiding postal code address lookup.
Use appropriate payment card autocomplete values.
Use a single input for payment card numbers.
Avoid using custom elements if they break the autofill experience.
Test in the field as well as the lab: page analytics, interaction analytics, and real-user performance measurement.
Test on a range of browsers, devices, and platforms.

Use meaningful HTML #

Use the elements and attributes built for the job:

<form>, <input>, <label>, and <button>
type, autocomplete, and inputmode

These enable built-in browser functionality, improve accessibility, and add meaning to your markup.

Use HTML elements as intended #

Put your form in a <form> #

You might be tempted not to bother wrapping your <input> elements in a <form>, and to handle data submission purely with JavaScript.

Don't do it!

An HTML <form> gives you access to a powerful set of built-in features across all modern browsers, and can help make your site accessible to screen readers and other assistive devices. A <form> also makes it simpler to build basic functionality for older browsers with limited JavaScript support, and to enable form submission even if there's a glitch with your code—and for the small number of users who actually disable JavaScript.

If you have more than one page component for user input, make sure to put each in its own <form> element. For example, if you have search and sign-up on the same page, put each in its own <form>.

Use `<label>` to label elements #

To label an <input>, <select>, or <textarea>, use a <label>.

Associate a label with an input by giving the label's for attribute the same value as the input's id.

<label for="address-line1">Address line 1</label>
<input id="address-line1" …>

Use a single label for a single input: don't try to label multiple inputs with only one label. This works best for browsers, and best for screenreaders. A tap or click on a label moves focus to the input it's associated with, and screenreaders announce label text when the label or the label's input gets focus.

Make buttons helpful #

Use <button> for buttons! You can also use <input type="submit">, but don't use a div or some other random element acting as a button. Button elements provide accessible behaviour, built-in form submission functionality, and can easily be styled.

Give each form submit button a value that says what it does. For each step towards checkout, use a descriptive call-to-action that shows progress and makes the next step obvious. For example, label the submit button on your delivery address form Proceed to Payment rather than Continue or Save.

Consider disabling a submit button once the user has tapped or clicked it—especially when the user is making a payment or placing an order. Many users click buttons repeatedly, even if they're working fine. That can mess up checkout and add to server load.

On the other hand, don't disable a submit button waiting on complete and valid user input. For example, don't just leave a Save Address button disabled because something is missing or invalid. That doesn't help the user—they may continue to tap or click the button and assume that it's broken. Instead, if users attempt to submit a form with invalid data, explain to them what's gone wrong and what to do to fix it. This is particularly important on mobile, where data entry is more difficult and missing or invalid form data may not be visible on the user's screen by the time they attempt to submit a form.

Make the most of HTML attributes #

Make it easy for users to enter data #

Use the appropriate input type attribute to provide the right keyboard on mobile and enable basic built-in validation by the browser.

For example, use type="email" for email addresses and type="tel" for phone numbers.

Keyboards appropriate for email and telephone.

For dates, try to avoid using custom select elements. They break the autofill experience if not properly implemented and don't work on older browsers. For numbers such as birth year, consider using an input element rather than a select, since entering digits manually can be easier and less error prone than selecting from a long drop-down list—especially on mobile. Use inputmode="numeric" to ensure the right keyboard on mobile and add validation and format hints with text or a placeholder to make sure the user enters data in the appropriate format.

Use autocomplete to improve accessibility and help users avoid re-entering data #

Using appropriate autocomplete values enables browsers to help users by securely storing data and autofilling input, select, and textarea values. This is particularly important on mobile, and crucial for avoiding high form abandonment rates. Autocomplete also provides multiple accessibility benefits.

If an appropriate autocomplete value is available for a form field, you should use it. MDN web docs has a full list of values and explanations of how to use them correctly.

By default, set the billing address to be the same as the delivery address. Reduce visual clutter by providing a link to edit the billing address (or use summary and details elements) rather than displaying the billing address in a form.

Add a link to review billing.

Use appropriate autocomplete values for the billing address, just as you do for shipping address, so the user doesn't have to enter data more than once. Add a prefix word to autocomplete attributes if you have different values for inputs with the same name in different sections.

<input autocomplete="shipping address-line-1" ...>
...
<input autocomplete="billing address-line-1" ...>

Help users enter the right data #

Try to avoid "telling off" customers because they "did something wrong". Instead, help users complete forms more quickly and easily by helping them fix problems as they happen. Through the checkout process, customers are trying to give your company money for a product or service—your job is to assist them, not to punish them!

You can add constraint attributes to form elements to specify acceptable values, including min, max, and pattern. The validity state of the element is set automatically depending on whether the element's value is valid, as are the :valid and :invalid CSS pseudo-classes which can be used to style elements with valid or invalid values.

For example, the following HTML specifies input for a birth year between 1900 and 2020. Using type="number" constrains input values to numbers only, within the range specified by min and max. If you attempt to enter a number outside the range, the input will be set to have an invalid state.

The following example uses pattern="[\d ]{10,30}" to ensure a valid payment card number, while allowing spaces:

Modern browsers also do basic validation for inputs with type email or url.

On form submission, browsers automatically set focus on fields with problematic or missing required values. No JavaScript required!

Basic built-in validation by the browser.

Validate inline and provide feedback to the user as they enter data, rather than providing a list of errors when they click the submit button. If you need to validate data on your server after form submission, list all problems that are found and clearly highlight all form fields with invalid values, as well as displaying a message inline next to each problematic field explaining what needs to be fixed. Check server logs and analytics data for common errors—you may need to redesign your form.

You should also use JavaScript to do more robust validation while users are entering data and on form submission. Use the Constraint Validation API (which is widely supported) to add custom validation using built-in browser UI to set focus and display prompts.

Find out more in Use JavaScript for more complex real-time validation.

Help users avoid missing required data #

Use the required attribute on inputs for mandatory values.

When a form is submitted modern browsers automatically prompt and set focus on required fields with missing data, and you can use the :required pseudo-class to highlight required fields. No JavaScript required!

Add an asterisk to the label for every required field, and add a note at the start of the form to explain what the asterisk means.

Simplify checkout #

Mind the mobile commerce gap! #

Imagine that your users have a fatigue budget. Use it up, and your users will leave.

You need to reduce friction and maintain focus, especially on mobile. Many sites get more traffic on mobile but more conversions on desktop—a phenomenon known as the mobile commerce gap. Customers may simply prefer to complete a purchase on desktop, but lower mobile conversion rates are also a result of poor user experience. Your job is to minimize lost conversions on mobile and maximize conversions on desktop. Research has shown that there's a huge opportunity to provide a better mobile form experience.

Most of all, users are more likely to abandon forms that look long, that are complex, and without a sense of direction. This is especially true when users are on smaller screens, distracted, or in a rush. Ask for as little data as possible.

Make guest checkout the default #

For an online store, the simplest way to reduce form friction is to make guest checkout the default. Don't force users to create an account before making a purchase. Not allowing guest checkout is cited as a major reason for shopping cart abandonment.

From baymard.com/checkout-usability

You can offer account sign-up after checkout. At that point, you already have most of the data you need to set up an account, so account creation should be quick and easy for the user.

Show checkout progress #

You can make your checkout process feel less complex by showing progress and making it clear what needs to be done next. The video below shows how UK retailer johnlewis.com achieves this.

Show checkout progress.

You need to maintain momentum! For each step towards payment, use page headings and descriptive button values that make it clear what needs to be done now, and what checkout step is next.

Give form buttons meaningful names that show what's next.

Use the enterkeyhint attribute on form inputs to set the mobile keyboard enter key label. For example, use enterkeyhint="previous" and enterkeyhint="next" within a multi-page form, enterkeyhint="done" for the final input in the form, and enterkeyhint="search" for a search input.

Enter key buttons on Android: 'next' and 'done'.

The enterkeyhint attribute is supported on Android and iOS. You can find out more from the enterkeyhint explainer.

Make it easy for users to go back and forth within the checkout process, to easily adjust their order, even when they're at the final payment step. Show full details of the order, not just a limited summary. Enable users to easily adjust item quantities from the payment page. Your priority at checkout is to avoid interrupting progress towards conversion.

Remove distractions #

Limit potential exit points by removing visual clutter and distractions such as product promotions. Many successful retailers even remove navigation and search from checkout.

Search, navigation and other distractions removed for checkout.

Keep the journey focused. This is not the time to tempt users to do something else!

Don't distract customers from completing their purchase.

For returning users you can simplify the checkout flow even more, by hiding data they don't need to see. For example: display the delivery address in plain text (not in a form) and allow users to change it via a link.

Hide data customers don't need to see.

Make it easy to enter name and address #

Only ask for the data you need #

Before you start coding your name and address forms, make sure to understand what data is required. Don't ask for data you don't need! The simplest way to reduce form complexity is to remove unnecessary fields. That's also good for customer privacy and can reduce back-end data cost and liability.

Use a single name input #

Allow your users to enter their name using a single input, unless you have a good reason for separately storing given names, family names, honorifics, or other name parts. Using a single name input makes forms less complex, enables cut-and-paste, and makes autofill simpler.

In particular, unless you have good reason not to, don't bother adding a separate input for a prefix or title (like Mrs, Dr or Lord). Users can type that in with their name if they want to. Also, honorific-prefix autocomplete currently doesn't work in most browsers, and so adding a field for name prefix or title will break the address form autofill experience for most users.

Enable name autofill #

Use name for a full name:

<input autocomplete="name" ...>

If you really do have a good reason to split out name parts, make sure to use appropriate autocomplete values:

honorific-prefix
given-name
nickname
additional-name-initial
additional-name
family-name
honorific-suffix

Allow international names #

You might want to validate your name inputs, or restrict the characters allowed for name data. However, you need to be as unrestrictive as possible with alphabets. It's rude to be told your name is "invalid"!

For validation, avoid using regular expressions that only match Latin characters. Latin-only excludes users with names or addresses that include characters that aren't in the Latin alphabet. Allow Unicode letter matching instead—and ensure your back-end supports Unicode securely as both input and output. Unicode in regular expressions is well supported by modern browsers.

Don't

<!-- Names with non-Latin characters (such as Françoise or Jörg) are 'invalid'. -->
<input pattern="[\w \-]+" ...>

<!-- Accepts Unicode letters. -->
<input pattern="[\p{L} \-]+" ...>

Unicode letter matching compared to Latin-only letter matching.

Allow for a variety of address formats #

When you're designing an address form, bear in mind the bewildering variety of address formats, even within a single country. Be careful not to make assumptions about "normal" addresses. (Take a look at UK Address Oddities! if you're not convinced!)

Make address forms flexible #

Don't force users to try to squeeze their address into form fields that don't fit.

For example, don't insist on a house number and street name in separate inputs, since many addresses don't use that format, and incomplete data can break browser autofill.

Be especially careful with required address fields. For example, addresses in large cities in the UK do not have a county, but many sites still force users to enter one.

Using two flexible address lines can work well enough for a variety of address formats.

<input autocomplete="address-line-1" id="address-line1" ...>
<input autocomplete="address-line-2" id="address-line2" ...>

Add labels to match:

<label for="address-line-1">
Address line 1 (or company name)
</label>
<input autocomplete="address-line-1" id="address-line1" ...>

<label for="address-line-2">
Address line 2 (optional)
</label>
<input autocomplete="address-line-2" id="address-line2" ...>

You can try this out by remixing and editing the demo embedded below.

Consider using a single textarea for address #

The most flexible option for addresses is to provide a single textarea.

The textarea approach fits any address format, and it's great for cutting and pasting—but bear in mind that it may not fit your data requirements, and users may miss out on autofill if they previously only used forms with address-line1 and address-line2.

For a textarea, use street-address as the autocomplete value.

Here is an example of a form that demonstrates the use of a single textarea for address:

Internationalize and localize your address forms #

It's especially important for address forms to consider internationalization and localization, depending on where your users are located.

Be aware that the naming of address parts varies as well as address formats, even within the same language.

    ZIP code: US
 Postal code: Canada
    Postcode: UK
     Eircode: Ireland
         PIN: India

It can be irritating or puzzling to be presented with a form that doesn't fit your address or that doesn't use the words you expect.

Customizing address forms for multiple locales may be necessary for your site, but using techniques to maximize form flexibility (as described above) may be adequate. If you don't localize your address forms, make sure to understand the key priorities to cope with a range of address formats:

Avoid being over-specific about address parts, such as insisting on a street name or house number.
Where possible avoid making fields required. For example, addresses in many countries don't have a postal code, and rural addresses may not have a street or road name.
Use inclusive naming: 'Country/region' not 'Country'; 'ZIP/postal code' not 'ZIP'.

Keep it flexible! The simple address form example above can be adapted to work 'well enough' for many locales.

Consider avoiding postal code address lookup #

Some websites use a service to look up addresses based on postal code or ZIP. This may be sensible for some use cases, but you should be aware of the potential downsides.

Postal code address suggestion doesn't work for all countries—and in some regions, post codes can include a large number of potential addresses.

ZIP or postal codes may include a lot of addresses!

It's difficult for users to select from a long list of addresses—especially on mobile if they're rushed or stressed. It can be easier and less error prone to let users take advantage of autofill, and enter their complete address filled with a single tap or click.

A single name input enables one-tap (one-click) address entry.

Simplify payment forms #

Payment forms are the single most critical part of the checkout process. Poor payment form design is a common cause of shopping cart abandonment. The devil's in the details: small glitches can tip users towards abandoning a purchase, particularly on mobile. Your job is to design forms to make it as easy as possible for users to enter data.

Help users avoid re-entering payment data #

Make sure to add appropriate autocomplete values in payment card forms, including the payment card number, name on the card, and the expiry month and year:

cc-number
cc-name
cc-exp-month
cc-exp-year

This enables browsers to help users by securely storing payment card details and correctly entering form data. Without autocomplete, users may be more likely to keep a physical record of payment card details, or store payment card data insecurely on their device.

Avoid using custom elements for payment card dates #

If not properly designed, custom elements can interrupt payment flow by breaking autofill, and won't work on older browsers. If all other payment card details are available from autocomplete but a user is forced to find their physical payment card to look up an expiry date because autofill didn't work for a custom element, you're likely to lose a sale. Consider using standard HTML elements instead, and style them accordingly.

Autocomplete filled all the fields—except the expiry date!

Use a single input for payment card and phone numbers #

For payment card and phone numbers use a single input: don't split the number into parts. That makes it easier for users to enter data, makes validation simpler, and enables browsers to autofill. Consider doing the same for other numeric data such as PIN and bank codes.

Don't use multiple inputs for a credit card number.

Validate carefully #

You should validate data entry both in realtime and before form submission. One way to do this is by adding a pattern attribute to a payment card input. If the user attempts to submit the payment form with an invalid value, the browser displays a warning message and sets focus on the input. No JavaScript required!

However, your pattern regular expression must be flexible enough to handle the range of payment card number lengths: from 14 digits (or possibly less) to 20 (or more). You can find out more about payment card number structuring from LDAPwiki.

Allow users to include spaces when they're entering a new payment card number, since this is how numbers are displayed on physical cards. That's friendlier to the user (you won't have to tell them "they did something wrong"), less likely to interrupt conversion flow, and it's straightforward to remove spaces in numbers before processing.

Test on a range of devices, platforms, browsers and versions #

It's particularly important to test address and payment forms on the platforms most common for your users, since form element functionality and appearance may vary, and differences in viewport size can lead to problematic positioning. BrowserStack enables free testing for open source projects on a range of devices and browsers.

The same page on iPhone 7 and iPhone 11.
Reduce padding for smaller mobile viewports to ensure the Complete payment button isn't hidden.

Implement analytics and RUM #

Testing usability and performance locally can be helpful, but you need real-world data to properly understand how users experience your payment and address forms.

For that you need analytics and Real User Monitoring—data for the experience of actual users, such as how long checkout pages take to load or how long payment takes to complete:

Page analytics: page views, bounce rates and exits for every page with a form.
Interaction analytics: goal funnels and events indicate where users abandon your checkout flow and what actions do they take when interacting with your forms.
Website performance: user-centric metrics can tell you if your checkout pages are slow to load and, if so—what's the cause.

Page analytics, interaction analytics, and real user performance measurement become especially valuable when combined with server logs, conversion data, and A/B testing, enabling you to answer questions such as whether discount codes increase revenue, or whether a change in form layout improves conversions.

That, in turn, gives you a solid basis for prioritizing effort, making changes, and rewarding success.

Keep learning #

Sign-in form best practices
Sign-up form best practices
Verify phone numbers on the web with the WebOTP API
Create Amazing Forms
Best Practices For Mobile Form Design
More capable form controls
Creating Accessible Forms
Streamlining the Sign-up Flow Using Credential Management API
Frank's Compulsive Guide to Postal Addresses provides useful links and extensive guidance for address formats in over 200 countries.
Countries Lists has a tool for downloading country codes and names in multiple languages, in multiple formats.

Photo by @rupixen on Unsplash.

Sign-up form best practices

2020-12-09T00:00:00Z

If users ever need to log in to your site, then good sign-up form design is critical. This is especially true for people on poor connections, on mobile, in a hurry, or under stress. Poorly designed sign-up forms get high bounce rates. Each bounce could mean a lost and disgruntled user—not just a missed sign-up opportunity.

Here is an example of a very simple sign-up form that demonstrates all of the best practices:

Checklist #

Before you implement a sign-up form and ask users to create an account on your site, consider whether you really need to. Wherever possible you should avoid gating features behind login.

The best sign-up form is no sign-up form!

By asking a user to create an account, you come between them and what they're trying to achieve. You're asking a favor, and asking the user to trust you with personal data. Every password and item of data you store carries privacy and security "data debt", becoming a cost and liability for your site.

If the main reason you ask users to create an account is to save information between navigations or browsing sessions, consider using client-side storage instead. For shopping sites, forcing users to create an account to make a purchase is cited as a major reason for shopping cart abandonment. You should make guest checkout the default.

Make sign-in obvious #

Make it obvious how to create an account on your site, for example with a Login or Sign in button at the top right of the page. Avoid using an ambiguous icon or vague wording ("Get on board!", "Join us") and don't hide login in a navigational menu. The usability expert Steve Krug summed up this approach to website usability: Don't make me think! If you need to convince others on your web team, use analytics to show the impact of different options.

Make sign-in obvious. An icon may be ambiguous, but a Sign in button or link is obvious.

The Gmail sign-in page has a link to create an account.
At window sizes larger than shown here, Gmail displays a Sign in link and a Create an account button.

Make sure to link accounts for users who sign up via an identity provider such as Google and who also sign up using email and password. That's easy to do if you can access a user's email address from the profile data from the identity provider, and match the two accounts. The code below shows how to access email data for a Google Sign-in user.

// auth2 is initialized with gapi.auth2.init()
if (auth2.isSignedIn.get()) {
  var profile = auth2.currentUser.get().getBasicProfile();
  console.log(`Email: ${profile.getEmail()}`);
}

Once a user has signed in, make it clear how to access account details. In particular, make it obvious how to change or reset passwords.

Cut form clutter #

In the sign-up flow, your job is to minimize complexity and keep the user focused. Cut the clutter. This is not the time for distractions and temptations!

Don't distract users from completing sign-up.

On sign-up, ask for as little as possible. Collect additional user data (such as name and address) only when you need to, and when the user sees a clear benefit from providing that data. Bear in mind that every item of data you communicate and store incurs cost and liability.

Don't double up your inputs just to make sure users get their contact details right. That slows down form completion and doesn't make sense if form fields are autofilled. Instead, send a confirmation code to the user once they've entered their contact details, then continue with account creation once they respond. This is a common sign-up pattern: users are used to it.

You may want to consider password-free sign-in by sending users a code every time they sign in on a new device or browser. Sites such as Slack and Medium use a version of this.

Password-free sign-in on medium.com.

As with federated login, this has the added benefit that you don't have to manage user passwords.

Consider session length #

Whatever approach you take to user identity, you'll need to make a careful decision about session length: how long the user remains logged in, and what might cause you to log them out.

Consider whether your users are on mobile or desktop, and whether they are sharing on desktop, or sharing devices.

Help password managers securely suggest and store passwords #

You can help third party and built-in browser password managers suggest and store passwords, so that users don't need to choose, remember or type passwords themselves. Password managers work well in modern browsers, synchronizing accounts across devices, across platform-specific and web apps—and for new devices.

This makes it extremely important to code sign-up forms correctly, in particular to use the correct autocomplete values. For sign-up forms use autocomplete="new-password" for new passwords, and add correct autocomplete values to other form fields wherever possible, such as autocomplete="email" and autocomplete="tel". You can also help password managers by using different name and id values in sign-up and sign-in forms, for the form element itself, as well as any input, select and textarea elements.

You should also use the appropriate type attribute to provide the right keyboard on mobile and enable basic built-in validation by the browser. You can find out more from Payment and address form best practices.

Ensure users enter secure passwords #

Enabling password managers to suggest passwords is the best option, and you should encourage users to accept the strong passwords suggested by browsers and third-party browser managers.

However, many users want to enter their own passwords, so you need to implement rules for password strength. The US National Institute of Standards and Technology explains how to avoid insecure passwords.

Don't allow compromised passwords #

Whatever rules you choose for passwords, you should never allow passwords that have been exposed in security breaches.

Once a user has entered a password, you need to check that it's not a password that's already been compromised. The site Have I Been Pwned provides an API for password checking, or you can run this as a service yourself.

Google's Password Manager also allows you to check if any of your existing passwords have been compromised.

If you do reject the password that a user proposes, tell them specifically why it was rejected. Show problems inline and explain how to fix them, as soon as the user has entered a value—not after they've submitted the sign-up form and had to wait for a response from your server.

Be clear why a password is rejected.

Don't prohibit password pasting #

Some sites don't allow text to be pasted into password inputs.

Disallowing password pasting annoys users, encourages passwords that are memorable (and therefore may be easier to compromise) and, according to organizations such as the UK National Cyber Security Centre, may actually reduce security. Users only become aware that pasting is disallowed after they try to paste their password, so disallowing password pasting doesn't avoid clipboard vulnerabilities.

Never store or transmit passwords in plain text #

Make sure to salt and hash passwords—and don't try to invent your own hashing algorithm!

Don't force password updates #

Don't force users to update their passwords arbitrarily.

Forcing password updates can be costly for IT departments, annoying to users, and doesn't have much impact on security. It's also likely to encourage people to use insecure memorable passwords, or to keep a physical record of passwords.

Rather than force password updates, you should monitor for unusual account activity and warn users. If possible you should also monitor for passwords that become compromised because of data breaches.

You should also provide your users with access to their account login history, showing them where and when a login happened.

Gmail account activity page.

Make it simple to change or reset passwords #

Make it obvious to users where and how to update their account password. On some sites, it's surprisingly difficult.

You should, of course, also make it simple for users to reset their password if they forget it. The Open Web Application Security Project provides detailed guidance on how to handle lost passwords.

To keep your business and your users safe, it's especially important to help users change their password if they discover that it's been compromised. To make this easier, you should add a /.well-known/change-password URL to your site that redirects to your password management page. This enables password managers to navigate your users directly to the page where they can change their password for your site. This feature is now implemented in Safari, Chrome, and is coming to other browsers. Help users change passwords easily by adding a well-known URL for changing passwords explains how to implement this.

You should also make it simple for users to delete their account if that's what they want.

Many users prefer to log in to websites using an email address and password sign-up form. However, you should also enable users to log in via a third party identity provider, also known as federated login.

WordPress login page, with Google and Apple login options.

This approach has several advantages. For users who create an account using federated login, you don't need to ask for, communicate, or store passwords.

You may also be able to access additional verified profile information from federated login, such as an email address—which means the user doesn't have to enter that data and you don't need to do the verification yourself. Federated login can also make it much easier for users when they get a new device.

Integrating Google Sign-In into your web app explains how to add federated login to your sign-up options. Many other identity platforms are available.

Make account switching simple #

Many users share devices and swap between accounts using the same browser. Whether users access federated login or not, you should make account switching simple.

Account switching on Gmail.

Consider offering multi-factor authentication #

Multi-factor authentication means ensuring that users provide authentication in more than one way. For example, as well as requiring the user to set a password, you might also enforce verification using a one-time-passcode sent by email or an SMS text message, or by using an app-based one-time code, security key or fingerprint sensor. SMS OTP best practices and Enabling Strong Authentication with WebAuthn explain how to implement multi-factor authentication.

You should certainly offer (or enforce) multi-factor authentication if your site handles personal or sensitive information.

Take care with usernames #

Don't insist on a username unless (or until) you need one. Enable users to sign up and sign in with only an email address (or telephone number) and password—or federated login if they prefer. Don't force them to choose and remember a username.

If your site does require usernames, don't impose unreasonable rules on them, and don't stop users from updating their username. On your backend you should generate a unique ID for every user account, not an identifier based on personal data such as username.

Also make sure to use autocomplete="username" for usernames.

Test on a range of devices, platforms, browsers and versions #

Test sign-up forms on the platforms most common for your users. Form element functionality may vary, and differences in viewport size can cause layout problems. BrowserStack enables free testing for open source projects on a range of devices and browsers.

Implement analytics and Real User Monitoring #

You need field data as well as lab data to understand how users experience your sign-up forms. Analytics and Real User Monitoring (RUM) provide data for the actual experience of your users, such as how long sign-up pages take to load, which UI components users interact with (or not) and how long it takes users to complete sign-up.

Page analytics: page views, bounce rates and exits for every page in your sign-up flow.
Interaction analytics: goal funnels and events indicate where users abandon the sign-up flow and what proportion of users click buttons, links, and other components of your sign-up pages.
Website performance: user-centric metrics can tell you if your sign-up flow is slow to load or visually unstable.

Small changes can make a big difference to completion rates for sign-up forms. Analytics and RUM enable you to optimize and prioritize changes, and monitor your site for problems that aren't exposed by local testing.

Keep learning #

Photo by @ecowarriorprincess on Unsplash.

Address form best practices codelab

2020-12-09T00:00:00Z

How can you design a form that works well for a variety of names and address formats? Minor form glitches irritate users and can cause them to leave your site or give up on completing a purchase or sign-up.

This codelab shows you how to build an easy-to-use, accessible form that works well for most users.

Step 1: Make the most of HTML elements and attributes #

You'll start this part of the codelab with an empty form, just a heading and a button all on their own. Then you'll begin adding inputs. (CSS and a little bit of JavaScript are already included.)

Click Remix to Edit to make the project editable.

Add a name field to your <form> element with the following code:

<section>
  <label for="name">Name</label>
  <input id="name" name="name">
</section>

That may look complicated and repetitive for just one name field, but it already does a lot.

You associated the label with the input by matching the label's for attribute with the input's name or id. A tap or click on a label moves focus to its input, making a much bigger target than the input on its own—which is good for fingers, thumbs and mouse clicks! Screenreaders announce label text when the label or the label's input gets focus.

What about name="name"? This is the name (which happens to be 'name'!) associated with the data from this input which is sent to the server when the form is submitted. Including a name attribute also means that the data from this element can be accessed by the FormData API.

Step 2: Add attributes to help users enter data #

What happens when you tap or click in the Name input in Chrome? You should see autofill suggestions that the browser has stored and guesses are appropriate for this input, given its name and id values.

Now add autocomplete="name" to your input code so it looks like this:

<section>
  <label for="name">Name</label>
  <input id="name" name="name" autocomplete="name">
</section>

Reload the page in Chrome and tap or click in the Name input. What differences do you see?

You should notice a subtle change: with autocomplete="name", the suggestions are now specific values that were used previously in form inputs that also had autocomplete="name". The browser isn't just guessing what might be appropriate: you have control. You'll also see the Manage… option to view and edit the names and addresses stored by your browser.

UI for autofill with guessed values, versus autocomplete.

Now add constraint validation attributes maxlength, pattern and required so your input code looks like this:

<section>
  <label for="name">Name</label>
  <input id="name" name="name" autocomplete="name"
    maxlength="100" pattern="[\p{L} \-\.]+" required>
</section>

maxlength="100" means the browser won't allow any input longer than 100 characters.
pattern="[\p{L} \-\.]+" uses a regular expression that allows Unicode letter characters, hyphens and periods (full stops). That means names such as Françoise or Jörg aren't classed as 'invalid'. That isn't the case if you use the value \w which [only allows characters from the Latin alphabet.
required means… required! The browser will not allow the form to be submitted without data for this field, and will warn and highlight the input if you attempt to submit it. No extra code required!

To test how the form works with and without these attributes, try entering data:

Try entering values that don't fit the pattern attribute.
Try submitting the form with an empty input. You'll see built-in browser functionality warning of the empty required field and setting focus on it.

Step 3: Add a flexible address field to your form #

To add an address field, add the following code to your form:

<section>
  <label for="address">Address</label>
  <textarea id="address" name="address" autocomplete="address"
    maxlength="300" required></textarea>
</section>

A textarea is the most flexible way for your users to enter their address, and it's great for cutting and pasting.

You should avoid splitting your address form into components such as street name and number unless you really need to. Don't force users to try to fit their address into fields that don't make sense.

Now add fields for ZIP or postal code, and Country or region. For simplicity, only the first five countries are included here. A full list is included in the completed address form.

<section>
  <label for="postal-code">ZIP or postal code (optional)</label>
  <input id="postal-code" name="postal-code"
    autocomplete="postal-code" maxlength="20">
</section>

<section id="country-region">
  <label for="">Country or region</label>
  <select id="country" name="country" autocomplete="country"
    required>
      <option selected value="SPACER"> </option>
      <option value="AF">Afghanistan</option>
      <option value="AX">Åland Islands</option>
      <option value="AL">Albania</option>
      <option value="DZ">Algeria</option>
      <option value="AS">American Samoa</option>
  </select>
</section>

You'll see that Postal code is optional: that's because many countries don't use postal codes. (Global Sourcebook provides information about address formats for 194 different countries, including sample addresses.) The label Country or region is used instead of Country, because some options from the full list (such as the United Kingdom) are not single countries (despite the autocomplete value).

Step 4: Enable customers to easily enter shipping and billing addresses #

You've built a highly functional address form, but what if your site requires more than one address, for shipping and billing, for example? Try updating your form to enable customers to enter shipping and billing addresses. How can you make data entry as quick and easy as possible, especially if the two addresses are the same? The article that goes with this codelab explains techniques for handling multiple addresses. Whatever you do, make sure to use the correct autocomplete values!

Step 5: Add a telephone number field #

To add a telephone number input, add the following code to the form:

<section>
  <label for="tel">Telephone</label>
  <input id="tel" name="tel" autocomplete="tel" type="tel"
    maxlength="30" pattern="[\d \-\+]+" enterkeyhint="done"
    required>
</section>

For phone numbers use a single input: don't split the number into parts. That makes it easier for users to enter data or copy and paste, makes validation simpler, and enables browsers to autofill.

There are two attributes that can improve the user experience of entering a telephone number:

type="tel" ensures mobile users get the right keyboard.
enterkeyhint="done" sets the mobile keyboard enter key label to show that this is the last field and the form can now be submitted (the default is next).

Use the enterkeyhint attribute to set the Enter button label: 'next' and 'done'.

Your complete address form should now look like this:

Try out your form on different devices. What devices and browsers are you targeting? How could the form be improved?

There are several ways to test your form on different devices:

Use Chrome DevTools Device Mode to simulate mobile devices.
Send the URL from your computer to your phone.
Use a service such as BrowserStack to test on a range of devices and browsers.

Going further #

Analytics and Real User Monitoring: enable the performance and usability of your form design to be tested and monitored for real users, and to check if changes are successful. You should monitor load performance and other Web Vitals, as well as page analytics (what proportion of users bounce from your address form without completing it? how long do users spend on your address form pages?) and interaction analytics (which page components do users interact with, or not?)
Where are your users located? How do they format their address? What names do they use for address components, such as ZIP or postal code? Frank's Compulsive Guide to Postal Addresses provides useful links and extensive guidance detailing address formats in over 200 countries.
Country selectors are notorious for poor usability. It's best to avoid select elements for a long list of items and the ISO 3166 country-code standard currently lists 249 countries! Instead of a <select> you may want to consider an alternative such as the Baymard Institute country selector.

Can you design a better selector for lists with a lot of items? How would you ensure your design is accessible across a range of devices and platforms? (The <select> element doesn't work well for a large number of items, but at least it's usable on virtually all browsers and assistive devices!)

The blog post <input type="country" /> discusses the complexity of standardizing a country selector. Localization of country names can also be problematic. Countries Lists has a tool for downloading country codes and names in multiple languages, in multiple formats.

Payment form best practices codelab

2020-12-09T00:00:00Z

This codelab shows you how to build a payment form that is secure, accessible and easy to use.

Step 1: Use HTML as intended #

Use elements built for the job:

<form>
<section>
<label>
<input>, <select>, <textarea>
<button>

As you'll see, these elements enable built-in browser functionality, improve accessibility, and add meaning to your markup.

Click Remix to Edit to make the project editable.

Take a look at the HTML for your form in index.html.

<form action="#" method="post">

  <h1>Payment form</h1>

  <section>
    <label>Card number</label>
    <input>
  </section>

  <section>
    <label>Name on card</label>
    <input>
  </section>

  <section id="cc-exp-csc">
    <div>
      <label>Expiry date</label>
      <input>
    </div>
    <div>
      <label>Security code</label>
      <input>
      <div class="explanation">Last 3 digits on back of card</div>
    </div>
  </section>

  <button id="complete-payment">Complete payment</button>

</form>

There are <input> elements for card number, name on card, expiry date and security code. They're all wrapped in <section> elements, and each has a label. The Complete Payment button is an HTML <button>. Later in this codelab you'll learn about the browser features you can access by using these elements.

Click View App to preview your payment form.

Does the form work well enough as it is?
Is there anything you would change to make it work better?
How about on mobile?

Click View Source to return to your source code.

Step 2: Design for mobile and desktop #

The HTML you added is valid, but the default browser styling makes the form hard to use, especially on mobile. It doesn't look too good, either.

You need to ensure your forms work well on a range of devices by adjusting padding, margins, and font sizes.

Copy all the CSS below and paste it into your own css/main.css file.

That's a lot of CSS! The main things to be aware of are the changes to sizes:

padding and margin are added to inputs.
font-size and other values are different for different viewport sizes.

When you're ready, click View App to see the styled form. You'll also notice that borders have been adjusted, and display: block; is used for labels so they go on a line on their own, and inputs can be full width. Sign-in form best practices explains the benefits of this approach in more detail.

The :invalid selector is used to indicate when an input has an invalid value. (You'll use this later in the codelab.)

The CSS is mobile-first:

The default CSS is for viewports less than 400px wide.
Media queries are used to override the default for viewports that are at least 400px wide, and then again for viewports that are at least 500px wide. This should work well for smaller phones, mobile devices with larger screens, and on desktop.

Whenever you build for the web, you need to test on different devices and viewport sizes. That's especially true for forms, because one small glitch can make them unusable. You should always adjust CSS breakpoints to ensure they work well with your content and your target devices.

Is the whole form visible?
Are the form inputs big enough?
Is all the text readable?
Did you notice any differences between using a real mobile device, and viewing the form in Device Mode in Chrome DevTools?
Did you need to adjust breakpoints?

There are several ways to test your form on different devices:

Use Chrome DevTools Device Mode to simulate mobile devices.
Send the URL from your computer to your phone.
Use a service such as BrowserStack to test on a range of devices and browsers.

Step 3: Add attributes to help users enter data #

Enable the browser to store and autofill input values, and provide access to secure built-in payment and validation features.

Add attributes to the form in your index.html file so it looks like this:

<form action="#" method="post">

  <h1>Payment form</h1>

  <section>
    <label for="cc-number">Card number</label>
    <input id="cc-number" name="cc-number" autocomplete="cc-number" inputmode="numeric" pattern="[\d ]{10,30}" required>
  </section>

  <section>
    <label for="cc-name">Name on card</label>
    <input id="cc-name" name="cc-name" autocomplete="cc-name" pattern="[\p{L} \-\.]+" required>
  </section>

  <section id="cc-exp-csc">
    <div>
      <label for="cc-exp">Expiry date</label>
      <input id="cc-exp" name="cc-exp" autocomplete="cc-exp" placeholder="MM/YY" maxlength="5" required>
    </div>
    <div>
      <label for="cc-csc">Security code</label>
      <input id="cc-csc" name="cc-csc" autocomplete="cc-csc" inputmode="numeric" maxlength="3" required>
      <div class="explanation">Back of card, last 3 digits</div>
    </div>
  </section>

  <button id="complete-payment">Complete payment</button>

</form>

View your app again and then tap or click in the Card number field. Depending on the device and platform, you may see a chooser showing payment methods stored for the browser, like the one below.

Built-in browser payment chooser and autofill.

Once you select a payment method and enter your security code, the browser autofills the form using the payment card autocomplete values you added to the form:

cc-number
cc-name
cc-exp
cc-csc

Many browsers also check and confirm the validity of credit card numbers and security codes.

On a mobile device you'll also notice that you get a numeric keyboard as soon as you tap into the Card number field. That's because you used inputmode="numeric". For numeric fields this makes it easier to enter numbers and impossible to enter non-numeric characters, and nudges users to remember the type of data they're entering.

It's extremely important to correctly add all available autocomplete values to payment forms. It's quite common for sites to miss out the autocomplete value for the card expiry date and other fields. If a single autofill value is wrong or missing, users will need to retrieve their actual card to manually enter card data, and you may lose out on a sale. If autofill on payment forms doesn't work properly, users may also decide to keep a record of payment card details on their phone or computer, which is highly insecure.

Try submitting the payment form with an empty field. The browser prompts to complete missing data. Now add a letter to the value in the Card number field and try submitting the form. The browser warns that value is invalid. This happens because you used the pattern attribute to specify valid values for a field. The same works for maxlength and other validation constraints No JavaScript required.

Your payment form should now look this:

Try removing autocomplete values and filling in the payment form. What difficulties do you encounter?
Try out payment forms on online stores. Consider what works well and what goes wrong. Are there any common problems or best practices you should follow?

Step 4: Disable the payment button once the form is submitted #

You should consider disabling a submit button once the user has tapped or clicked it—especially when the user is making payment. Many users tap or click buttons repeatedly, even if they're working fine. That can cause problems with payment processing and add to server load.

Add the following JavaScript to your js/main.js file:

const form = document.querySelector('form');
const completePaymentButton = document.querySelector('button#complete-payment');

form.addEventListener('submit', handleFormSubmission);

function handleFormSubmission(event) {
  event.preventDefault();
  if (form.checkValidity() === false) {
    // Handle invalid form data.
  } else {
    completePaymentButton.textContent = 'Making payment...';
    completePaymentButton.disabled = 'true';
    setTimeout(() => {alert('Made payment!');}, 500);
  }
}

Try submitting the payment form and see what happens.

Here is how your code should look at this point, with the addition of some comments and a validate() function:

You'll notice that the JavaScript includes commented-out code for data validation. This code uses the Constraint Validation API (which is widely supported) to add custom validation, accessing built-in browser UI to set focus and display prompts. Un-comment the code and try it out. You'll need to set appropriate values for someregex and message, and set a value for someField.
What analytics and Real User Monitoring data would you monitor in order to identify ways to improve your forms?

Your complete payment form should now look like this:

Try out your form on different devices. What devices and browsers are you targeting? How could the form be improved?

Going further #

Consider the following crucial form features that are not covered in this codelab:

Link to your Terms of Service and privacy policy documents: make it clear to users how you safeguard their data.
Style and branding: make sure these match the rest of your site. When entering names and addresses and making payment, users need to feel comfortable, trusting that they're still in the right place.
Analytics and Real User Monitoring: enable the performance and usability of your form design to be tested and monitored for real users.

Sign-up form best practices codelab

2020-12-09T00:00:00Z

This codelab shows you how to build a sign-up form that's secure, accessible, and easy to use.

Step 1: Use meaningful HTML #

In this step you'll learn how to use form elements to make the most of built-in browser features.

Click Remix to Edit to make the project editable.

Take a look at the HTML for your form in index.html. You'll see there are inputs for name, email and password. Each is in a section, and each has a label. The Sign up button is… a <button>! Later in this codelab, you'll learn the special powers of all these elements.

<form action="#" method="post">
        
  <h1>Sign up</h1>
  
  <section>
    <label>Full name</label>
    <input>
  </section>
  
  <section>
    <label>Email</label>
    <input>
  </section>
  
  <section>
    <label>Password</label>
    <input>
  </section>
  
  <button id="sign-up">Sign up</button>
  
</form>

Click View App to preview your sign-up form. This shows you what a form looks like with no CSS other than the default browser styles.

Do the default styles look OK? What would you change to make the form look better?
Do the default styles work OK? What problems might be encountered using your form as it is? What about on mobile? What about for screenreaders or other assistive technologies?
Who are your users, and what devices and browsers are you targeting?

Test your form #

You could acquire a lot of hardware and set up a device lab, but there are cheaper and simpler ways to try out your form on a range of browsers, platforms and devices:

Use Chrome DevTools Device Mode to simulate mobile devices.
Send the URL from your computer to your phone.
Use a service such as BrowserStack to test on a range of devices and browsers.
Try out the form using a screenreader tool such as the ChromeVox Chrome extension.

Click View App to preview your sign-up form.

Try out your form on different devices using Chrome DevTools Device Mode.
Now open the form on a phone or other real devices. What differences do you see?

Step 2: Add CSS to make the form work better #

Click View Source to return to your source code.

There's nothing wrong with the HTML so far, but you need to make sure your form works well for a range of users on mobile and desktop.

In this step you'll add CSS to make the form easier to use.

Copy and paste all the following CSS into css/main.css file:

Click View App to see your styled sign-up form. Then click View Source to return to css/main.css.

Does this CSS work for a variety of browsers and screen sizes?
Try adjusting padding, margin, and font-size to suit your test devices.
The CSS is mobile-first. Media queries are used to apply CSS rules for viewports that are at least 400px wide, and then again for viewports that are at least 500px wide. Are these breakpoints adequate? How should you choose breakpoints for forms?

Step 3: Add attributes to help users enter data #

In this step you add attributes to your input elements to enable the browser to store and autofill form field values, and warn of fields with missing or invalid data.

Update your index.html file so the form code looks like this:

<form action="#" method="post">
        
  <h1>Sign up</h1>
  
  <section>        
    <label for="name">Full name</label>
    <input id="name" name="name" autocomplete="name" 
           pattern="[\p{L}\.\- ]+" required>
  </section>

  <section>        
    <label for="email">Email</label>
    <input id="email" name="email" autocomplete="username"
           type="email" required>
  </section>
  
  <section>
    <label for="password">Password</label>
    <input id="password" name="password" autocomplete="new-password" 
           type="password" minlength="8" required>
  </section>
  
  <button id="sign-up">Sign up</button>
  
</form>

The type values do a lot:

type="password" obscures text as it's entered and enables the browser's password manager to suggest a strong password.
type="email" provides basic validation and ensures mobile users get an appropriate keyboard. Try it out!

Click View App and then click the Email label. What happens? Focus moves to the email input because the label has a for value that matches the email input's id. The other labels and inputs work the same way. Screenreaders also announce label text when the label (or the label's associated input) gets focus. You can try that using the ChromeVox extension.

Try submitting the form with an empty field. The browser won't submit the form, and it prompts to complete missing data and sets focus. That's because you added the require attribute to all the inputs. Now try submitting with a password that has less than eight characters. The browser warns that the password isn't long enough and sets focus on the password input because of the minlength="8" attribute. The same works for pattern (used for the name input) and other validation constraints. The browser does all this automatically, without needing any extra code.

Using the autocomplete value name for the Full name input makes sense, but what about the other inputs?

autocomplete="username" for the Email input means the browser's password manager will store the email address as the 'name' for this user (the username!) to go with the password.
autocomplete="new-password" for Password is a hint to the password manager that it should offer to store this value as the password for the current site. You can then use autocomplete="current-password" to enable autofill in a sign-in form (remember, this is sign-up form).

Step 4: Help users enter secure passwords #

With the form as it is, did you notice anything wrong with the password input?

There are two issues:

There's no way to know if there are constraints on the password value.
You can't see the password to check if you got it right.

Don't make users guess!

Update the password section of index.html with the following code:

<section>
  <label for="password">Password</label>
  <button id="toggle-password" type="button" aria-label="Show password as plain text. 
          Warning: this will display your password on the screen.">Show password</button>
  <input id="password" name="password" type="password" autocomplete="new-password" 
         minlength="8" aria-describedby="password-constraints" required>
  <div id="password-constraints">Eight or more characters.</div>
</section>

This adds new features to help users enter passwords:

A button (actually just text) to toggle password display. (The button functionality will be added with JavaScript.)
An aria-label attribute for the password-toggle button.
An aria-describedby attribute for the password input. Screenreaders read the label text, the input type (password), and then the description.

To enable the password-toggle button and show users information about password constraints, copy all the JavaScript below and paste it into your own js/main.js file.

(The CSS is already in place from step 2. Take a look, to see how the password-toggle button is styled and positioned.)

Would an icon work better than text to toggle password display? Try Discount Usability Testing with a small group of friends or colleagues.
To experience how screenreaders work with forms, install the ChromeVox extension and navigate through the form. Could you complete the form using ChromeVox only? If not, what would you change?

Here's how your form should look at this point:

Going further #

This codelab doesn't cover several important features:

Checking for compromised passwords. You should never allow passwords that have been compromised. You can (and should) use a password-checking service to catch compromised passwords. You can use an existing service or run one yourself on your own servers. Try it out! Add password checking to your form.
Link to your Terms of Service and privacy policy documents: make it clear to users how you safeguard their data.
Style and branding: make sure these match the rest of your site. When entering names and addresses and making payment, users need to feel comfortable, trusting that they're still in the right place.
Analytics and Real User Monitoring: enable the performance and usability of your form design to be tested and monitored for real users.

What are third-party origin trials?

2020-10-01T00:00:00Z

Origin trials are a way to test a new or experimental web platform feature.

Origin trials are usually only available on a first-party basis: they only work for a single registered origin. If a developer wants to test an experimental feature on other origins where their content is embedded, those origins all need to be registered for the origin trial, each with a unique trial token. This is not a scalable approach for testing scripts that are embedded across a number of sites.

Third-party origin trials make it possible for providers of embedded content to try out a new feature across multiple sites.

Third-party origin trials don't make sense for all features. Chrome will only make the third-party origin trial option available for features where embedding code on third-party sites is a common use case. Getting started with Chrome's origin trials provides more general information about how to participate in Chrome origin trials.

If you participate in an origin trial as a third-party provider, it will be your responsibility to notify and set expectations with any partners or customers whose sites you intend to include in the origin trial. Experimental features may cause unexpected issues and browser vendors may not be able to provide troubleshooting support.

Check Chrome Platform Status for updates on progress with third-party origin trials.

How to register for a third-party origin trial #

Select a trial from the list of active trials.
On the trial's registration page, enable the option to request a third-party token, if available.
Select one of the choices for restricting usage for a third-party token:
1. Standard Limit: This is the usual limit of 0.5% of Chrome page loads.
2. User Subset: A small percentage of Chrome users will always be excluded from the trial, even when a valid third-party token is provided. The exclusion percentage varies (or might not apply) for each trial, but is typically less than 5%.
Click the Register button to submit your request.
Your third-party token will be issued immediately, unless further review of the request is required. (Depending on the trial, token requests may require review.)
If review is required, you'll be notified by email when the review is complete and your third-party token is ready.

Registration page for the Conversion Measurement trial.

How to provide feedback #

If you're registering for a third-party origin trial and have feedback to share on the process or ideas on how we can improve it, please create an issue on the Origin Trials GitHub code repo.

Find out more #

Photo by Louis Reed on Unsplash.

Sign-in form best practices

2020-06-29T00:00:00Z

If users ever need to log in to your site, then good sign-in form design is critical. This is especially true for people on poor connections, on mobile, in a hurry, or under stress. Poorly designed sign-in forms get high bounce rates. Each bounce could mean a lost and disgruntled user—not just a missed sign-in opportunity.

Here is an example of a simple sign-in form that demonstrates all of the best practices:

Checklist #

Use meaningful HTML elements: <form>, <input>, <label>, and <button>.
Label each input with a <label>.
Use element attributes to access built-in browser features: type, name, autocomplete, required.
Give input name and id attributes stable values that don't change between page loads or website deployments.
Put sign-in in its own <form> element.
Ensure successful form submission.
Use autocomplete="new-password" and id="new-password" for the password input in a sign-up form, and for the new password in a reset-password form.
Use autocomplete="current-password" and id="current-password" for a sign-in password input.
Provide Show password functionality.
Use aria-label and aria-describedby for password inputs.
Don't double-up inputs.
Design forms so the mobile keyboard doesn't obscure inputs or buttons.
Ensure forms are usable on mobile: use legible text, and make sure inputs and buttons are large enough to work as touch targets.
Maintain branding and style on your sign-up and sign-in pages.
Test in the field as well as the lab: build page analytics, interaction analytics, and user-centric performance measurement into your sign-up and sign-in flow.
Test across browsers and devices: form behaviour varies significantly across platforms.

This article is about frontend best practices. It does not explain how to build backend services to authenticate users, store their credentials, or manage their accounts. 12 best practices for user account, authorization and password management outlines core principles for running your own backend. If you have users in different parts of the world, you need to consider localizing your site's use of third-party identity services as well as its content. There are also two relatively new APIs not covered in this article which can help you build a better sign-in experience: * WebOTP: to deliver one-time passcodes or PIN numbers via SMS to mobile phones. This can allow users to select a phone number as an identifier (no need to enter an email address!) and also enables two-step verification for sign-in and one-time codes for payment confirmation. * Credential Management: to enable developers to store and retrieve password credentials and federated credentials programmatically.

Use meaningful HTML #

Use elements built for the job: <form>, <label> and <button>. These enable built-in browser functionality, improve accessibility, and add meaning to your markup.

Use `<form>` #

You might be tempted to wrap inputs in a <div> and handle input data submission purely with JavaScript. It's generally better to use a plain old <form> element. This makes your site accessible to screenreaders and other assistive devices, enables a range of built-in browser features, makes it simpler to build basic functional sign-in for older browsers, and can still work even if JavaScript fails.

Use `<label>` #

To label an input, use a <label>!

<label for="email">Email</label>
<input id="email" …>

Two reasons:

A tap or click on a label moves focus to its input. Associate a label with an input by using the label's for attribute with the input's name or id.
Screenreaders announce label text when the label or the label's input gets focus.

Don't use placeholders as input labels. People are liable to forget what the input was for once they've started entering text, especially if they get distracted ("Was I entering an email address, a phone number, or an account ID?"). There are lots of other potential problems with placeholders: see Don't Use The Placeholder Attribute and Placeholders in Form Fields Are Harmful if you're unconvinced.

It's probably best to put your labels above your inputs. This enables consistent design across mobile and desktop and, according to Google AI research, enables quicker scanning by users. You get full width labels and inputs, and you don't need to adjust label and input width to fit the label text.

Label and input width is limited when both are on the same line.

Open the label-position Glitch on a mobile device to see for yourself.

Use `<button>` #

Use <button> for buttons! Button elements provide accessible behaviour and built-in form submission functionality, and they can easily be styled. There's no point in using a <div> or some other element pretending to be a button.

Ensure that the submit button says what it does. Examples include Create account or Sign in, not Submit or Start.

Ensure successful form submission #

Help password managers understand that a form has been submitted. There are two ways to do that:

Navigate to a different page.
Emulate navigation with History.pushState() or History.replaceState(), and remove the password form.

With an XMLHttpRequest or fetch request, make sure that sign-in success is reported in the response and handled by taking the form out of the DOM as well as indicating success to the user.

Consider disabling the Sign in button once the user has tapped or clicked it. Many users click buttons multiple times even on sites that are fast and responsive. That slows down interactions and adds to server load.

Conversely, don't disable form submission awaiting user input. For example, don't disable the Sign in button if users haven't entered their customer PIN. Users may miss out something in the form, then try repeatedly tapping the (disabled) Sign in button and think it's not working. At the very least, if you must disable form submission, explain to the user what's missing when they click on the disabled button.

Don't double up inputs #

Some sites force users to enter emails or passwords twice. That might reduce errors for a few users, but causes extra work for all users, and increases abandonment rates. Asking twice also makes no sense where browsers autofill email addresses or suggest strong passwords. It's better to enable users to confirm their email address (you'll need to do that anyway) and make it easy for them to reset their password if necessary.

Make the most of element attributes #

This is where the magic really happens! Browsers have multiple helpful built-in features that use input element attributes.

Keep passwords private—but enable users to see them if they want #

Passwords inputs should have type="password" to hide password text and help the browser understand that the input is for passwords. (Note that browsers use a variety of techniques to understand input roles and decide whether or not to offer to save passwords.)

You should add a Show password icon or button to enable users to check the text they've entered—and don't forget to add a Forgot password link. See Enable password display.

Password input from the Google sign-in form: with Show password icon and Forgot password link.

Give mobile users the right keyboard #

Use <input type="email"> to give mobile users an appropriate keyboard and enable basic built-in email address validation by the browser… no JavaScript required!

If you need to use a telephone number instead of an email address, <input type="tel"> enables a telephone keypad on mobile. You can also use the inputmode attribute where necessary: inputmode="numeric" is ideal for PIN numbers. Everything You Ever Wanted to Know About inputmode has more detail.

Prevent mobile keyboard from obstructing the Sign in button #

Unfortunately, if you're not careful, mobile keyboards may cover your form or, worse, partially obstruct the Sign in button. Users may give up before realizing what has happened.

The Sign in button: now you see it, now you don't.

Where possible, avoid this by displaying only the email/phone and password inputs and Sign in button at the top of your sign-in page. Put other content below.

The keyboard doesn't obstruct the Sign in button.

Test on a range of devices #

You'll need to test on a range of devices for your target audience, and adjust accordingly. BrowserStack enables free testing for open source projects on a range of real devices and browsers.

The Sign in button: obscured on iPhone 7 and 8, but not on iPhone 11.

Consider using two pages #

Some sites (including Amazon and eBay) avoid the problem by asking for email/phone and password on two pages. This approach also simplifies the experience: the user is only tasked with one thing at a time.

Two-stage sign-in: email or phone, then password.

Ideally, this should be implemented with a single <form>. Use JavaScript to initially display only the email input, then hide it and show the password input. If you must force the user to navigate to a new page between entering their email and password, the form on the second page should have a hidden input element with the email value, to help enable password managers to store the correct value. Password Form Styles that Chromium Understands provides a code example.

Help users to avoid re-entering data #

You can help browsers store data correctly and autofill inputs, so users don't have to remember to enter email and password values. This is particularly important on mobile, and crucial for email inputs, which get high abandonment rates.

There are two parts to this:

The autocomplete, name, id, and type attributes help browsers understand the role of inputs in order to store data that can later be used for autofill. To allow data to be stored for autofill, modern browsers also require inputs to have a stable name or id value (not randomly generated on each page load or site deployment), and to be in a <form> with a submit button.
The autocomplete attribute helps browsers correctly autofill inputs using stored data.

For email inputs use autocomplete="username", since username is recognized by password managers in modern browsers—even though you should use type="email" and you may want to use id="email" and name="email".

For password inputs, use the appropriate autocomplete and id values to help browsers differentiate between new and current passwords.

Use `autocomplete="new-password"` and `id="new-password"` for a new password #

Use autocomplete="new-password" and id="new-password" for the password input in a sign-up form, or the new password in a change-password form.

Use `autocomplete="current-password"` and `id="current-password"` for an existing password #

Use autocomplete="current-password" and id="current-password" for the password input in a sign-in form, or the input for the user's old password in a change-password form. This tells the browser that you want it to use the current password that it has stored for the site.

For a sign-up form:

<input type="password" autocomplete="new-password" id="new-password" …>

For sign-in:

<input type="password" autocomplete="current-password" id="current-password" …>

Support password managers #

Different browsers handle email autofill and password suggestion somewhat differently, but the effects are much the same. On Safari 11 and above on desktop, for example, the password manager is displayed, and then biometric authentication (fingerprint or facial recognition) is used if available.

Sign-in with autocomplete—no text entry required!

Chrome on desktop displays email suggestions, shows the password manager, and autofills the password.

Autocomplete sign-in flow in Chrome 84.

Browser password and autofill systems are not simple. The algorithms for guessing, storing and displaying values are not standardized, and vary from platform to platform. For example, as pointed out by Hidde de Vries: "Firefox's password manager complements its heuristics with a recipe system."

Autofill: What web devs should know, but don't has a lot more information about using name and autocomplete. The HTML spec lists all 59 possible values.

Enable the browser to suggest a strong password #

Modern browsers use heuristics to decide when to show the password manager UI and suggest a strong password.

Here's how Safari does it on desktop.

Password suggestion flow in Safari.

(Strong unique password suggestion has been available in Safari since version 12.0.)

Built-in browser password generators mean users and developers don't need to work out what a "strong password" is. Since browsers can securely store passwords and autofill them as necessary, there's no need for users to remember or enter passwords. Encouraging users to take advantage of built-in browser password generators also means they're more likely to use a unique, strong password on your site, and less likely to reuse a password that could be compromised elsewhere.

Help save users from accidentally missing inputs #

Add the required attribute to both email and password fields. Modern browsers automatically prompt and set focus for missing data. No JavaScript required!

Prompt and focus for missing data on Firefox for desktop (version 76) and Chrome for Android (version 83).

Design for fingers and thumbs #

The default browser size for just about everything relating to input elements and buttons is too small, especially on mobile. This may seem obvious, but it's a common problem with sign-in forms on many sites.

Make sure inputs and buttons are large enough #

The default size and padding for inputs and buttons is too small on desktop and even worse on mobile.

According to Android accessibility guidance the recommended target size for touchscreen objects is 7–10 mm. Apple interface guidelines suggest 48x48 px, and the W3C suggest at least 44x44 CSS pixels. On that basis, add (at least) about 15 px of padding to input elements and buttons for mobile, and around 10 px on desktop. Try this out with a real mobile device and a real finger or thumb. You should comfortably be able to tap each of your inputs and buttons.

The Tap targets are not sized appropriately Lighthouse audit can help you automate the process of detecting input elements that are too small.

Design for thumbs #

Search for touch target and you'll see lots of pictures of forefingers. However, in the real world, many people use their thumbs to interact with phones. Thumbs are bigger than forefingers, and control is less precise. All the more reason for adequately sized touch targets.

Make text big enough #

As with size and padding, the default browser font size for input elements and buttons is too small, particularly on mobile.

Default styling on desktop and mobile: input text is too small to be legible for many users.

Browsers on different platforms size fonts differently, so it's difficult to specify a particular font size that works well everywhere. A quick survey of popular websites shows sizes of 13–16 pixels on desktop: matching that physical size is a good minimum for text on mobile.

This means you need to use a larger pixel size on mobile: 16px on Chrome for desktop is quite legible, but even with good vision it's difficult to read 16px text on Chrome for Android. You can set different font pixel sizes for different viewport sizes using media queries. 20px is about right on mobile—but you should test this out with friends or colleagues who have low vision.

The Document doesn't use legible font sizes Lighthouse audit can help you automate the process of detecting text that's too small.

Provide enough space between inputs #

Add enough margin to make inputs work well as touch targets. In other words, aim for about a finger width of margin.

Make sure your inputs are clearly visible #

The default border styling for inputs makes them hard to see. They're almost invisible on some platforms such as Chrome for Android.

As well as padding, add a border: on a white background, a good general rule is to use #ccc or darker.

Legible text, visible input borders, adequate padding and margins.

Use built-in browser features to warn of invalid input values #

Browsers have built-in features to do basic form validation for inputs with a type attribute. Browsers warn when you submit a form with an invalid value, and set focus on the problematic input.

Basic built-in validation by the browser.

You can use the :invalid CSS selector to highlight invalid data. Use :not(:placeholder-shown) to avoid selecting inputs with no content.

input[type=email]:not(:placeholder-shown):invalid {
  color: red;
  outline-color: red;
}

Try out different ways of highlighting inputs with invalid values.

Use JavaScript where necessary #

Toggle password display #

You should add a Show password icon or button to enable users to check the text they've entered. Usability suffers when users can't see the text they've entered. Currently there's no built-in way to do this, though there are plans for implementation. You'll need to use JavaScript instead.

Google sign-in form: with Show password icon and Forgot password link.

The following code uses a text button to add Show password functionality.

HTML:

<section>
  <label for="password">Password</label>
  <button id="toggle-password" type="button" aria-label="Show password as plain text. Warning: this will display your password on the screen.">Show password</button>
  <input id="password" name="password" type="password" autocomplete="current-password" required>
</section>

Here's the CSS to make the button look like plain text:

button#toggle-password {
  background: none;
  border: none;
  cursor: pointer;
  /* Media query isn't shown here. */
  font-size: var(--mobile-font-size);
  font-weight: 300;
  padding: 0;
  /* Display at the top right of the container */
  position: absolute;
  top: 0;
  right: 0;
}

And the JavaScript for showing the password:

const passwordInput = document.getElementById('password');
const togglePasswordButton = document.getElementById('toggle-password');

togglePasswordButton.addEventListener('click', togglePassword);

function togglePassword() {
  if (passwordInput.type === 'password') {
    passwordInput.type = 'text';
    togglePasswordButton.textContent = 'Hide password';
    togglePasswordButton.setAttribute('aria-label',
      'Hide password.');
  } else {
    passwordInput.type = 'password';
    togglePasswordButton.textContent = 'Show password';
    togglePasswordButton.setAttribute('aria-label',
      'Show password as plain text. ' +
      'Warning: this will display your password on the screen.');
  }
}

Here's the end result:

Sign-in form with Show password text 'button', in Safari on Mac and iPhone 7.

Make password inputs accessible #

Use aria-describedby to outline password rules by giving it the ID of the element that describes the constraints. Screenreaders provide the label text, the input type (password), and then the description.

<input type="password" aria-describedby="password-constraints" …>
<div id="password-constraints">Eight or more characters with a mix of letters, numbers and symbols.</div>

When you add Show password functionality, make sure to include an aria-label to warn that the password will be displayed. Otherwise users may inadvertently reveal passwords.

<button id="toggle-password"
        aria-label="Show password as plain text.
                    Warning: this will display your password on the screen.">
  Show password
</button>

You can see both ARIA features in action in the following Glitch:

Creating Accessible Forms has more tips to help make forms accessible.

Validate in realtime and before submission #

HTML form elements and attributes have built-in features for basic validation, but you should also use JavaScript to do more robust validation while users are entering data and when they attempt to submit the form.

Step 5 of the sign-in form codelab uses the Constraint Validation API (which is widely supported) to add custom validation using built-in browser UI to set focus and display prompts.

Find out more: Use JavaScript for more complex real-time validation.

Analytics and RUM #

"What you cannot measure, you cannot improve" is particularly true for sign-up and sign-in forms. You need to set goals, measure success, improve your site—and repeat.

Discount usability testing can be helpful for trying out changes, but you'll need real-world data to really understand how your users experience your sign-up and sign-in forms:

Page analytics: sign-up and sign-in page views, bounce rates, and exits.
Interaction analytics: goal funnels (where do users abandon your sign-in or sign-in flow?) and events (what actions do users take when interacting with your forms?)
Website performance: user-centric metrics (are your sign-up and sign-in forms slow for some reason and, if so, what is the cause?).

You may also want to consider implementing A/B testing in order to try out different approaches to sign-up and sign-in, and staged rollouts to validate the changes on a subset of users before releasing changes to all users.

General guidelines #

Well designed UI and UX can reduce sign-in form abandonment:

Don't make users hunt for sign-in! Put a link to the sign-in form at the top of the page, using well-understood wording such as Sign In, Create Account or Register.
Keep it focused! Sign-up forms are not the place to distract people with offers and other site features.
Minimize sign-up complexity. Collect other user data (such as addresses or credit card details) only when users see a clear benefit from providing that data.
Before users start on your sign-up form, make it clear what the value proposition is. How do they benefit from signing in? Give users concrete incentives to complete sign-up.
If possible allow users to identify themselves with a mobile phone number instead of an email address, since some users may not use email.
Make it easy for users to reset their password, and make the Forgot your password? link obvious.
Link to your terms of service and privacy policy documents: make it clear to users from the start how you safeguard their data.
Include the logo and name of your company or organization on your signup and sign-in pages, and make sure that language, fonts and styles match the rest of your site. Some forms don't feel like they belong to the same site as other content, especially if they have a significantly different URL.

Keep learning #

Photo by Meghan Schiereck on Unsplash.

Use cross-platform browser features to build a sign-in form

2020-06-29T00:00:00Z

This codelab teaches you how to build a sign-in form that's secure, accessible, and easy to use.

1. Use meaningful HTML #

Use these elements built for the job:

<form>
<section>
<label>
<button>

As you'll see, these elements enable built-in browser functionality, improve accessibility, and add meaning to your markup.

Click Remix to edit to make the project editable.
Add the following code to the <body> element:

<form action="#" method="post">
  <h1>Sign in</h1>
  <section>
    <label>Email</label>
    <input>
  </section>
  <section>
    <label>Password</label>
    <input>
  </section>
  <button>Sign in</button>
</form>

Here's how your index.html file should look at this point:

Click View App to preview your sign-in form. The HTML that you added is valid and correct, but the default browser styling makes it looks terrible and hard to use, especially on mobile devices.
Click View Source to return to your source code.

2. Design for fingers and thumbs #

Adjust padding, margins, and font sizes to ensure that your inputs work well on mobile.

Copy the following CSS and paste it into your style.css file:

Click View App to see your freshly styled sign-in form.
Click View Source to return to your style.css file.

That's quite a lot of code! The main things to be aware of are the changes to sizes:

padding and margin are added to inputs.
font-size is different for mobile and desktop.

The :invalid selector is used to indicate when an input has an invalid value. This doesn't work yet.

The CSS layout is mobile-first:

The default CSS is for viewports less than 450 pixels wide.
The media query section sets overrides for viewports that are at least 450 pixels wide.

When building your own form like this, it's very important at this point in the process to test your code on real devices on desktop and mobile:

Is label and input text readable, especially for people with low vision?
Are the inputs and Sign in button large enough to use as touch targets for thumbs?

3. Add input attributes to enable built-in browser features #

Enable the browser to store and autofill input values, and provide access to built-in password-management features.

Add attributes to your form HTML so it looks like this:

<form action="#" method="post">
  <h1>Sign in</h1>
  <section>        
    <label for="email">Email</label>
    <input id="email" name="email" type="email" autocomplete="username" required autofocus>
  </section>
  <section>        
    <label for="password">Password</label>
    <input id="password" name="password" type="password" autocomplete="current-password" required>
  </section>
  <button id="sign-in">Sign in</button>
</form>

View your app again and then click Email.

Notice how focus moves to the email input. This is because the label is associated with the input through the for="email" attribute. Screenreaders also announce the label text when the label or the label's associated input gets focus.
Focus the email input on a mobile device.

Notice how the keyboard is optimized for typing an email address. For example, the @ and . characters might be shown on the primary keyboard, and the operating system might show stored emails above the keyboard. All of this happens because the type="email" attribute is applied to an <input> element.

Type some text into the password input.

The text is hidden by default because the type="password" attribute has been applied to the element.

The autocomplete, name, id, and type attributes help browsers understand the role of inputs in order to store data that can be used later for autofill.

Focus the email input on a desktop device and type some text.

You can see the URL of your app when you click Fullscreen . If you stored any email addresses in your browser, you probably see a dialog that lets you select from those stored emails. This happens because the autocomplete="username" attribute applied to the email input.

autocomplete="username" and autocomplete="current-password" help browsers use stored values to autofill the inputs.

Different browsers use different techniques to work out the role of form inputs and provide autofill for a range of different websites.

Add and remove attributes to try this yourself.

It's extremely important to test behavior across platforms. You should enter values and submit the form in different browsers on different devices. It's easy to test on a range of platforms with BrowserStack, which is free for open source projects. Try it!

Here's how your index.html file should look at this point:

4. Add UI to toggle password display #

Usability experts strongly recommend the addition of an icon or button that lets users see the text that they enter in the Password field. There's no built-in way to do this, so you need to implement it yourself with JavaScript.

The code to add this functionality is straightforward. This example uses text, not an icon.

Update the index.html, style.css, and script.js files as follows.

Add a toggle to the password section in the index.html file:

<section>
  <label for="password">Password</label>
  <button id="toggle-password" type="button" aria-label="Show password as plain text. Warning: this will display your password on the screen.">Show password</button>
  <input id="password" name="password" type="password" autocomplete="current-password" required>
</section>

Add the following CSS to the bottom of the style.css file:

button#toggle-password {
  background: none;
  border: none;
  cursor: pointer;
  font-weight: 300;
  padding: 0;
  position: absolute;
  top: -4px;
  right: -2px;
}

This makes the Show password button look like plain text and displays it in the top-right corner of the password section.

Add the following JavaScript to the script.js file to toggle password display and set the appropriate aria-label:

const passwordInput = document.getElementById('password');
const togglePasswordButton = document.getElementById('toggle-password');

togglePasswordButton.addEventListener('click', togglePassword);

function togglePassword() {
  if (passwordInput.type === 'password') {
    passwordInput.type = 'text';
    togglePasswordButton.textContent = 'Hide password';
    togglePasswordButton.setAttribute('aria-label',
      'Hide password.');
  } else {
    passwordInput.type = 'password';
    togglePasswordButton.textContent = 'Show password';
    togglePasswordButton.setAttribute('aria-label',
      'Show password as plain text. ' +
      'Warning: this will display your password on the screen.');
  }
}

Try the show password logic now.

a. View your app. b. Enter some text in the password field. c. Click Show password.
Repeat the fourth step on multiple browsers on different operating systems.

Think about UX design: will users notice Show password and understand it? Is there a better way to provide this functionality? This is a good moment to try discount usability testing with a small group of friends or colleagues.

To understand how this functionality works for screenreaders, install the ChromeVox Classic Extension and navigate through the form. Do the aria-label values work as intended?

Some websites, such as Gmail, use icons, not text, to toggle password display. When you're done with this codelab, implement this with SVG images. Material Design offers high-quality icons that you can download for free.

Here's how your code should look at this point:

5. Add form validation #

You can help users enter their data correctly when you let them validate their data before form submission and show them what they need to change.

HTML form elements and attributes have built-in features for basic validation, but you should also use JavaScript to do more robust validation while users enter data and when they attempt to submit the form.

This step uses the Constraint Validation API (which is widely supported) to add custom validation with built-in browser UI that sets focus and displays prompts.

Tell users the constraints for passwords and any other inputs. Don't make them guess!

Update the password section of the index.html file:

<section>
  <label for="password">Password</label>
  <button id="toggle-password" type="button" aria-label="Show password as plain text. Warning: this will display your password on the screen.">Show password</button>
  <input id="password" name="password" type="password" autocomplete="current-password" aria-describedby="password-constraints" required>
  <div id="password-constraints">At least eight characters, with at least one lowercase and one uppercase letter.</div>
</section>

This adds two new features:

Information about password constraints
An aria-describedby attribute for the password input (Screenreaders read the label text, the input type (password), and then the description.)

Add the following CSS to the bottom of the style.css file:

div#password-constraints {
  margin: 5px 0 0 0;
  font-size: 16px;
}

Add the following JavaScript to script.js file:

passwordInput.addEventListener('input', resetCustomValidity); 
function resetCustomValidity() {
  passwordInput.setCustomValidity('');
}

// A production site would use more stringent password testing. 
function validatePassword() {
  let message= '';
  if (!/.{8,}/.test(passwordInput.value)) {
    message = 'At least eight characters. ';
  }
  if (!/.*[A-Z].*/.test(passwordInput.value)) {
    message += 'At least one uppercase letter. ';
  }
  if (!/.*[a-z].*/.test(passwordInput.value)) {
    message += 'At least one lowercase letter.';
  }
  passwordInput.setCustomValidity(message);
}

const form = document.querySelector('form');
const signinButton = document.querySelector('button#sign-in');

form.addEventListener('submit', handleFormSubmission);                       

function handleFormSubmission(event) {
  event.preventDefault();
  validatePassword();
  form.reportValidity();
  if (form.checkValidity() === false) {
  } else {
    // On a production site do form submission.
    alert('Logging in!')
    signinButton.disabled = 'true';
  }
}

Try it!

All recent browsers have built-in features for form validation and support validation with JavaScript.

a. Enter an invalid email address and click Sign in. The browser displays a warning—no JavaScript required!

b. Enter a valid email address, but then click Sign in without a password value. The browser warns that you missed a required value and sets focus on the password input.

c. Enter an invalid password and click Sign in. Now you see different messages depending on what's wrong.
Try different ways to help users enter email addresses and passwords. Better password form fields offers some clever suggestions.

Here's how your code should look at this point:

Go further #

They're not shown in this codelab, but you still need these four crucial sign-in form features:

Add Forgot your password?, a button that makes it easy for users to reset their passwords.
Link to your terms of service and privacy policy documents so that your users know how you safeguard their data.
Consider style and branding, and ensure that these additional features match the rest of your website.
Add Analytics and RUM so that you can test and monitor the performance and usability of your form design.

Getting started with Trust Tokens

2020-06-22T00:00:00Z

Summary #

Trust tokens enable an origin to issue cryptographic tokens to a user it trusts. The tokens are stored by the user's browser. The browser can then use the tokens in other contexts to evaluate the user's authenticity.

The Trust Token API enables trust of a user in one context to be conveyed to another context without identifying the user or linking the two identities.

You can try out the API with our demo, and inspect tokens in the Chrome DevTools Network and Application tabs.

Trust Tokens in the Chrome DevTools Network tab.

Trust Tokens in the Chrome DevTools Application tab.

Why do we need Trust Tokens? #

The web needs ways to establish trust signals which show that a user is who they say they are, and not a bot pretending to be a human, or a malicious third-party defrauding a real person or service. Fraud protection is particularly important for advertisers, ad providers, and CDNs.

Unfortunately, many existing mechanisms to gauge and propagate trustworthiness—to work out if an interaction with a site is from a real human, for example—take advantage of techniques that can also be used for fingerprinting.

The API must preserve privacy, enabling trust to be propagated across sites without individual user tracking.

What's in the Trust Tokens proposal? #

The web relies on building trust signals to detect fraud and spamming. One way to do this is by tracking browsing with global, cross-site per-user identifiers. For a privacy-preserving API, that's not acceptable.

From the proposal explainer:

This API proposes a new per-origin storage area for "Privacy Pass" style cryptographic tokens, which are accessible in third party contexts. These tokens are non-personalized and cannot be used to track users, but are cryptographically signed so they cannot be forged.

When an origin is in a context where they trust the user, they can issue the browser a batch of tokens, which can be "spent" at a later time in a context where the user would otherwise be unknown or less trusted. Crucially, the tokens are indistinguishable from one another, preventing websites from tracking users through them.

We further propose an extension mechanism for the browser to sign outgoing requests with keys bound to a particular token redemption.

Sample API usage #

The following is adapted from sample code in the API explainer.

Imagine that a user visits a news website (publisher.example) which embeds advertising from a third party ad network (foo.example). The user has previously used a social media site that issues trust tokens (issuer.example).

The sequence below shows how trust tokens work.

1. The user visits issuer.example and performs actions that lead the site to believe they are a real human, such as account activity, or passing a CAPTCHA challenge.

2. issuer.example verifies the user is a human, and runs the following JavaScript to issue a trust token to the user's browser:

fetch('https://issuer.example/trust-token', {
  trustToken: {
    type: 'token-request',
    issuer: 'https://issuer.example'
  }
}).then(...)

3. The user's browser stores the trust token, associating it with issuer.example.

4. Some time later, the user visits publisher.example.

5. publisher.example wants to know if the user is a real human. publisher.example trusts issuer.example, so they check if the user's browser has valid tokens from that origin:

document.hasTrustToken('https://issuer.example');

6. If this returns a promise that resolves to true, that means the user has tokens from issuer.example, so publisher.example can attempt to redeem a token:

fetch('https://issuer.example/trust-token', {
trustToken: {
  type: 'token-redemption',
  issuer: 'https://issuer.example',
  refreshPolicy: {none, refresh}
}
}).then(...)

With this code:

The redeemer publisher.example requests a redemption.
If the redemption is successful, the issuer issuer.example returns a redemption record which indicates that at some point they issued a valid token to this browser.

7. Once the promise returned by fetch() has resolved, the redemption record can be used in subsequent resource requests:

fetch('https://foo.example/get-content', {
  trustToken: {
    type: 'send-redemption-record',
    issuers: ['https://issuer.example', ...]
  }
});

With this code:

Redemption records are included as a request header Sec-Redemption-Record.
foo.example receives the redemption record and can parse the record to determine whether issuer.example thought this user was a human.
foo.example responds accordingly.

How can a website work out whether to trust you?

You might have shopping history with an e-commerce site, check-ins on a location platform, or account history at a bank. Issuers might also look at other factors such as how long you've had an account, or other interactions (such as CAPTCHAs or form submission) that increase the issuer's trust in the likelihood that you're a real human.

Trust token issuance #

If the user is deemed to be trustworthy by a trust token issuer such as issuer.example, the issuer can fetch trust tokens for the user by making a fetch() request with a trustToken parameter:

fetch('issuer.example/trust-token', {
  trustToken: {
    type: 'token-request'
  }
}).then(...)

This invokes an extension of the Privacy Pass issuance protocol using a new cryptographic primitive:

Generate a set of pseudo-random numbers known as nonces.
Blind the nonces (encode them so the issuer can't view their contents) and attach them to the request in a Sec-Trust-Token header.
Send a POST request to the endpoint provided.

The endpoint responds with blinded tokens (signatures on the blind nonces), then the tokens are unblinded and stored internally together with the associated nonces by the browser as trust tokens.

Trust token redemption #

A publisher site (such as publisher.example in the example above) can check if there are trust tokens available for the user:

const userHasTokens = await document.hasTrustToken('issuer.example/trust-token');

If there are tokens available, the publisher site can redeem them to get a redemption record:

fetch('issuer.example/trust-token', {
  ...
  trustToken: {
    type: 'token-redemption',
    refreshPolicy: 'none'
  }
  ...
}).then(...)

The publisher can include redemption records in requests that require a trust token—such as posting a comment, liking a page, or voting in a poll—by using a fetch() call like the following:

fetch('https://foo.example/post-comment', {
  ...
  trustToken: {
    type: 'send-redemption-record',
    issuers: ['issuer.example/trust-token', ...]
  }
  ...
}).then(...);

Redemption records are included as a Sec-Redemption-Record request header.

Privacy considerations #

Tokens are designed to be 'unlinkable'. An issuer can learn aggregate information about which sites its users visit, but can't link issuance with redemption: when a user redeems a token, the issuer can't tell the token apart from other tokens it has created. However, trust tokens currently do not exist in a vacuum: there are other ways an issuer could currently—in theory—join a user's identity across sites, such as third-party cookies and covert tracking techniques. It is important for sites to understand this ecosystem transition as they plan their support. This is a general aspect of the transition for many Privacy Sandbox APIs, so not discussed further here.

Security considerations #

Trust token exhaustion: a malicious site could deliberately deplete a user's supply of tokens from a particular issuer. There are several mitigations against this kind of attack, such as enabling issuers to provide many tokens at once, so users have an adequate supply of ensuring browsers only ever redeem one token per top-level page view.

Double-spend prevention: malware might attempt to access all of a user's trust tokens. However, tokens will run out over time, since every redemption is sent to the same token issuer, which can verify that each token is used only once. To mitigate risk, issuers could also sign fewer tokens.

Request mechanisms #

It might be possible to allow for sending redemption records outside of fetch(), for example with navigation requests. Sites might also be able to include issuer data in HTTP response headers to enable token redemption in parallel with page loading.

To reiterate: this proposal needs your feedback! If you have comments, please create an issue on the Trust Token explainer repository.

Find out more #

Thanks to all those who helped write and review this post.

Photo by ZSun Fu on Unsplash.

Digging into the Privacy Sandbox

2020-04-08T00:00:00Z

Summary #

This post outlines APIs and concepts from the Privacy Sandbox proposals.
The proposal authors are inviting feedback from the community, particularly from those in the advertising space (publishers, advertisers, and ad tech companies), to suggest missing use cases and share information about how to support your business use cases.
You can comment on the proposals by filing issues on the repositories linked to below.
There's a glossary for the proposals at the end of this post.

The current state of privacy on the web #

Websites use services from other companies to provide analytics, serve video and do lots of other useful stuff. Composability is one of the web's superpowers. Most notably, ads are included in web pages via third-party JavaScript and iframes. Ad views, clicks and conversions are tracked via third-party cookies and scripts.

However, when you visit a website you may not be aware of the third parties involved and what they're doing with your data. Even publishers and web developers may not understand the entire third-party supply chain.

Ad selection, conversion measurement, and other use cases currently rely on establishing stable cross-site user identity. Historically this has been done with third-party cookies, but browsers have begun to restrict access to these cookies. There has also been an increase in the use of other mechanisms for cross-site user tracking, such as covert browser storage, device fingerprinting, and requests for personal information such as email addresses.

This is a dilemma for the web. How can legitimate third-party use cases be supported without enabling users to be tracked across sites?

In particular, how can websites fund content by enabling third parties to show ads and measure ad performance—but not allow individual users to be profiled? How can advertisers and site owners evaluate a user's authenticity without resorting to dark patterns such as device fingerprinting?

The way things work at the moment can be problematic for the entire web ecosystem, not just users. For publishers and advertisers, tracking identity and using a variety of non-standard third-party solutions can add to technical debt, code complexity, and data risk. Users, developers, publishers, and advertisers should be confident that the web is protecting user privacy choices.

Advertising is a core web business model for the internet, but advertising has to work for everyone. Which brings us to the Privacy Sandbox's mission: to create a thriving web ecosystem that is respectful of users and private by default.

Introducing the Privacy Sandbox #

The Privacy Sandbox introduces a set of privacy-preserving APIs to support business models that fund the open web in the absence of tracking mechanisms like third-party cookies.

The Privacy Sandbox APIs require web browsers to take on a new role. Rather than working with limited tools and protections, the APIs enable the user's browser to act on the user's behalf—locally, on their device—to protect the user's identifying information as they navigate the web. The APIs enable use cases such as ad selection and conversion measurement, without revealing individual private and personal information. In engineering terms a sandbox is a protected environment; a key principle of the Privacy Sandbox is that a user's personal information should be protected and not shared in a way that lets the user be identified across sites.

This is a shift in direction for browsers. The Privacy Sandbox's vision of the future has browsers providing specific tools to satisfy specific use cases, while preserving user privacy. A Potential Privacy Model for the Web sets out core principles behind the APIs:

To establish the range of web activity across which the user's browser can let websites treat a person as having a single identity.
To identify the ways in which information can move across identity boundaries without compromising that separation.

The Privacy Sandbox proposals #

In order to successfully transition away from third-party cookies the Privacy Sandbox initiative needs your support. The proposal explainers need feedback from developers as well as publishers, advertisers, and ad technology companies, to suggest missing use cases and share information about how to accomplish their goals in a privacy-safe way.

You can comment on the proposal explainers by filing issues against each repository:

Privacy Model for the Web
Establish the range of web activity across which the user's browser can let websites treat a person as having a single identity. Identify the ways in which information can move across identity boundaries without compromising that separation.
Privacy Budget
Limit the total amount of potentially identifiable data that sites can access. Update APIs to reduce the amount of potentially identifiable data revealed. Make access to potentially identifiable data measurable.
Gnatcatcher
Limit the ability to identify individual users by accessing their IP address.
Trust Token API
Enable an origin that trusts a user to issue them with cryptographic tokens which are stored by the user's browser so they can be used in other contexts to evaluate the user's authenticity.
First-Party Sets
Allow related domain names owned by the same entity to declare themselves as belonging to the same first party.
Aggregated Reporting
Provide privacy preserving mechanisms to support a variety of use cases such as view-through-conversion, brand, lift, and reach measurement.
Attribution Reporting
Provide privacy preserving measurement of clicks and views with event-level and aggregate reports.
Topics API
Enable interest-based advertising, without having to resort to tracking the sites a user visits. The design of the API was informed by community feedback from our earlier FLoC trials, and supersedes the FLoC proposal.
FLEDGE
Provides a solution for remarketing use cases, designed so it cannot be used by third parties to track user browsing behaviour across sites.

You can dive into the API proposal explainers right away, and over the coming months we'll be publishing posts about each proposal individually.

We'll also be adding to our playlist of five-minute videos that give a simple explanation of each API.

Use cases and goals #

Measure conversion #

Goal: Enable advertisers to measure ad performance.

The Attribution Reporting API enables the measurement of two events that are linked together:

An event on a publisher's website, such as a user viewing or clicking an ad.
A subsequent conversion on an advertiser site.

This API supports click-through and view-through measurement.

Other features in this API include cross-device attribution reporting and app-to-web attribution reporting.

The API also offers two types of attribution reports:

Event-level reports associate a particular ad click or view (on the ad side) with data on the conversion side. To preserve user privacy, by preventing the joining of user identity across sites, conversion-side data is very limited, and the data is 'noised' (meaning that for a small percentage of cases, random data is sent). As an extra privacy protection, reports are not sent immediately.
Aggregate reports are not tied to a specific event on the ad side. These reports provide richer, higher-fidelity conversion data than event-level reports. A combination of privacy techniques across cryptography, distribution of trust, and differential privacy help reduce the risk of identity joining across sites.

Both report types can be used simultaneously: they're complementary.

Introduction to Attribution Reporting explains more about the status of these features and how to try this API.

Select ads #

Goal: Enable advertisers to display ads relevant to users.

Relevant ads are more favorable to users and more profitable for publishers (the people running ad-supported websites). Third party ad selection tools make ad space more valuable to advertisers (the people who purchase ad space on websites) which in turn increases revenue for ad-supported websites and enables content to get created and published.

There are many ways to make ads relevant to the user, including the following:

First-party-data: Show ads relevant to topics a person has told a website they have an interest in, or content a person has looked at previously on the current website.
Contextual: Choose where to display ads based on site content. For example, 'Put this ad next to articles about knitting.'
Remarketing: Advertise to people who've already visited your site, while they are not on your site. For example, 'Show this ad for discount wool to people who visited your store and left knitting items in their shopping cart—while they're visiting craft sites.'
Interest-based: Select ads based on a user's browsing history. For example, 'Show this ad to users whose browsing behaviour indicates they might be interested in knitting'.

First-party-data and contextual ad selection can be achieved without knowing anything about the user other than their activity within a site. These techniques don't require cross-site tracking.

Remarketing is usually done by using cookies or some other way to recognize people across websites: adding users to lists and then selecting specific ads to show them.

Interest-based ad selection currently uses cookies to track user behaviour across as many sites as possible. Many people are concerned about the privacy implications of ad selection. The Privacy Sandbox proposes two alternatives, for remarketing and for interest-based selection:

FLEDGE: for remarketing use cases.
The API is designed so it cannot be used by third parties to track user browsing behaviour: the user's browser, not the advertiser or ad tech platform, stores advertiser-defined interest groups that the user's browser is associated with. The user's browser combines interest group data with ad buyer/seller data and business logic to conduct an "auction" locally on the user's device to select an ad, rather than sharing data with a third party.
Topics API: for interest-based audiences.
Enable interest-based advertising, without having to resort to tracking the sites a user visits. The API proposes using machine learning to infer topics from hostnames, and a JavaScript API that returns coarse-grained topics a user might currently be interested in, based on the hostnames of sites recently visited.

Combat fingerprinting #

Goal: Reduce the amount of potentially identifiable data revealed by APIs and make access to potentially identifiable data controllable by users, and measurable.

Browsers have taken steps to deprecate third-party cookies, but techniques to identify and track the behaviour of individual users, known as fingerprinting, have continued to evolve. Fingerprinting uses mechanisms that users aren't aware of and can't control.

The Privacy Budget proposal aims to limit the potential for fingerprinting by identifying how much fingerprint data is exposed by JavaScript APIs or other 'surfaces' (such as HTTP request headers) and setting a limit on how much of this data can be accessed.
Fingerprinting surfaces such as the User-Agent header will be reduced in scope, and the data made available by alternative mechanisms such as Client Hints will be subject to Privacy Budget limits. Other surfaces, such as the device orientation and battery-level APIs, will be updated to keep the information exposed to a minimum.

IP address security #

Goal: Control access to IP addresses to reduce covert fingerprinting, and allow sites to opt out of seeing IP addresses in order to not consume privacy budget.

A user's IP address is the public 'address' of their computer on the internet, which in most cases is dynamically assigned by the network through which they connect to the internet. However, even dynamic IP addresses may remain stable over a significant period of time. Not surprisingly, this means that IP addresses are a significant source of fingerprint data.

The Gnatcatcher proposal is an attempt to provide a privacy-preserving approach that avoids consuming privacy budget, while ensuring that sites requiring access to IP addresses for legitimate purposes such as abuse prevention can do so, subject to certification and auditing.

There are two parts to the proposal:

Willful IP Blindness provides a way for websites to let browsers know they are not connecting IP addresses with users.
Near-path NAT allows groups of users to send their traffic through the same privatizing server, effectively hiding their IP addresses from a site host.

Combat spam, fraud and denial-of-service attacks #

Goal: Verify user authenticity without fingerprinting.

Anti-fraud protection is crucial for keeping users safe, and to ensure that advertisers and site owners can get accurate ad performance measurements. Advertisers and site owners must be able to distinguish between malicious bots and authentic users. If advertisers can't reliably tell which ad clicks are from real humans, they spend less, so site publishers get less revenue. Many third party services currently use techniques such as device fingerprinting to combat fraud.

Unfortunately, the techniques used to identify legitimate users and block spammers, fraudsters, and bots work in ways similar to fingerprinting techniques that damage privacy.

The Trust Tokens API proposes an alternative approach, allowing authenticity established for a user in one context, such as a social media site, to be conveyed to another context, such as an ad running on a news site—without identifying the user or linking the two identities.

Enable domains to belong to the same first party #

Goal: Enable entities to declare that related domain names are owned by the same first party.

Many organizations own sites across multiple domains. This can become a problem if restrictions are imposed on tracking user identity across sites that are seen as 'third-party' but actually belong to the same organization.

First Party Sets aims to make the web's concept of first and third parties more closely aligned with the real world's by enabling multiple domains to declare themselves as belonging to the same first party.

Find out more #

Privacy Sandbox proposal explainers #

The Privacy Sandbox initiative needs your support. The API proposal explainers need feedback, in particular to suggest missing use cases and more-private ways to accomplish their goals.

A Potential Privacy Model for the Web sets out the core principles underlying the APIs.

The Privacy Sandbox #

Technical overview: The Privacy Sandbox
Non-technical overview: privacysandbox.com
Project principles: Building a more private web
The future of third-party cookies

Discussion and participation #

How to participate in the Privacy Sandbox initiative
Progress update on the Privacy Sandbox initiative
Privacy Sandbox Developer Support: join discussions or ask questions.

Use cases, policies, and requirements #

Appendix: Glossary of terms used in the proposal explainers #

Click-through rate (CTR) #

The ratio of users who click on an ad, having seen it. (See also impression.)

Click-through-conversion (CTC) #

A conversion attributed to an ad that was 'clicked'.

Conversion #

The completion of an action on an advertiser's website by a user who has previously interacted with an ad from that advertiser. For example, purchase of a product or sign-up for a newsletter after clicking an ad that links to the advertiser's site.

Differential privacy #

Share information about a dataset to reveal patterns of behaviour without revealing private information about individuals or whether they belong to the dataset.

Domain #

See Top-Level Domain and eTLD.

eTLD, eTLD+1 #

'Effective' top level domains are defined by the Public Suffix List. For example:

co.uk
appspot.com
glitch.me

Effective TLDs are what enable foo.appspot.com to be a different site from bar.appspot.com. The effective top-level domain (eTLD) in this case is appspot.com, and the whole site name (foo.appspot.com, bar.appspot.com) is known as the eTLD+1.

Entropy #

A measure of how much an item of data reveals individual identity.

Data entropy is measured in bits. The more that data reveals identity, the higher its entropy value.

Data can be combined to identify an individual, but it can be difficult to work out whether new data adds to entropy. For example, knowing a person is from Australia doesn't reduce entropy if you already know the person is from Kangaroo Island.

Fingerprinting #

Techniques to identify and track the behaviour of individual users. Fingerprinting uses mechanisms that users aren't aware of and can't control. Sites such as Panopticlick and amiunique.org show how fingerprint data can be combined to identify you as an individual.

Fingerprinting surface #

Something that can be used (probably in combination with other surfaces) to identify a particular user or device. For example, the navigator.userAgent() JavaScript method and the User-Agent HTTP request header provide access to a fingerprinting surface (the user agent string).

First-party #

Resources from the site you're visiting. For example, the page you're reading is on the site web.dev and includes resources from that site. See also Third-party.

Impression #

View of an ad. (See also click-through rate.)

k-anonymity #

A measure of anonymity within a data set. If you have k anonymity, you can't be distinguished from k-1 other individuals in the data set. In other words, k individuals have the same information (including you).

Nonce #

Arbitrary number used once only in cryptographic communication.

Origin #

The origin of a request, including the server name but no path information. For example: https://web.dev.

Passive surface #

Some fingerprinting surfaces, such as user agent strings, IP addresses and accept-language headers, are available to every website whether the site asks for them or not. That means passive surfaces can easily consume a site's privacy budget.

The Privacy Sandbox initiative proposes replacing passive surfaces with active ways to get specific information, for example using Client Hints a single time to get the user's language rather than having an accept-language header for every response to every server.

Publisher #

The Privacy Sandbox proposal explainers are mostly about ads, so the kinds of publishers referred to are ones that put ads on their websites.

Reach #

The total number of people who see an ad.

Remarketing #

Advertising to people who've already visited your site. For example, an online store could show ads for a toy sale to people who previously viewed toys on their site.

Site #

See Top-Level Domain and eTLD.

Surface #

See Fingerprinting surface and Passive surface.

Third-party #

Resources served from a domain that's different from the website you're visiting. For example, a website foo.com might use analytics code from google-analytics.com (via JavaScript), fonts from use.typekit.net (via a link element) and a video from vimeo.com (in an iframe). See also First-party.

Top-level domain (TLD) #

Top-level domains such as .com and .org are listed in the Root Zone Database.

Note that some 'sites' are actually just subdomains. For example, translate.google.com and maps.google.com are just subdomains of google.com (which is the eTLD + 1).

.well-known #

It can be useful to access policy or other information about a host before making a request. For example, robots.txt tells web crawlers which pages to visit and which pages to ignore. IETF RFC8615 outlines a standardized way to make site-wide metadata accessible in standard locations in a /.well-known/ subdirectory. You can see a list of these at iana.org/assignments/well-known-uris/well-known-uris.xhtml.

Thanks to all those who helped with writing and reviewing this post.

Photo by Pierre Bamin on Unsplash.

Feedback from the summer 2019 image optimization survey

2019-11-22T00:00:00Z

This post lists the freeform feedback that Google Web DevRel received in its Summer 2019 image optimization techniques survey. Responses were solicited through Web Fundamentals and @ChromiumDev. The motivation for the survey was to find out why most sites don't follow image optimization best practices even though they seem like a relatively easy way to improve performance. The response data isn't listed here because there were flaws in the survey methodology.

Audience #

If you're a web developer, you might find this post useful for discovering new image optimization techniques, or details on how other web developers have solved a problem that you're facing, as well as the costs, benefits, and limitations of each technique.
If you're an image service or image CDN provider, this post might help you find new market opportunities.
If you're a framework, build tool, or CMS developer, this post might give you ideas on new features to implement.

Comments #

WebP #

"I do like WebP but it isn't yet fully ready. Moreover, our WordPress doesn't support WebP. One of the most popular photo editing apps, Photoshop, also doesn't support WebP out of the box. So we can't rely on 3rd party apps or services for image compression."
"Make WebP usable on Safari."
"I would love to use WebP if I could export them from Photoshop/Figma/Sketch and all browsers supported it." [Note: Sketch does support WebP]
"Next gen formatting solution would be great."
"Stop pushing WebP so hard when browser support is poor, and consider the need for PNG instead of JPEG for screenshots."
"Google Docs doesn't support WebP."
"We would use WebP exclusively, but are concerned about browser compatibility."
"First fix browser compatibility and update legacy browsers or add legacy fixes, then people will be more inclined to adopt to new image types like WebP…"
"Encourage plugin/theme developers to consider providing support to WebP and other next-gen image types, so that non-developers don't need to fiddle with it."

SVG and vector images #

"If possible I'm using (animated) SVG. gatsby-image fixed a lot of this. But when you dig into what they've done, it's completely unrealistic that a normal website should have to build out something like that to get images to work right. The browser should take on more of this responsibility."
"Would it be possible to document how to create SVG animations with lottie.js?"
"We try to use big resolution JPEG pictures with low sizes in our website most of the time to avoid loading times. We also ensure to use SVGs when necessary to provide quality for responsive design."
"We try to use optimized vector graphics for all but pics if possible."

Other image formats #

"We [need to] better educate people to stop using GIF."

Lazy loading #

"Please keep the user in mind when considering features such as lazy load, because for many it's annoying."
"Make the lazy load attribute work with background-image please."
"Frameworks should do better asset processing out of the box."
"We have converted from lazy loading a long time ago. User reports of millions of images and sites "NOT LOADING". That was understanding our team summarized it as. It's hard for a non-technical users to describe issues."
"I'm keen to get a better understanding of using Intersection Observer API for lazy loading rather than using traditional techniques."
"This well works for me: pwafire.org/developer/codelabs/progressive-loading."

Background images #

"I usually load images as backgrounds in CSS."
"The <img> tag is problematic and difficult to control fine-grained details about, especially with user-submitted content. We use <div> and background-image styling much more often as it allows us to use background-size, background-position, and prevent right-click saving of the image."

Transparency #

"It's 2019. How are JPGs still without alpha transparency?"
"I only really use PNGs for photographs when I need a transparent background."

Low Quality Image Placeholders (LQIPs) #

"We use LQIPS and it's a great technique to keep visitors engaged without loading high quality images really early."

Performance #

"We actually had a recent performance issue with images. As a user scrolls down on our site, we show the next 60 cards which include a thumbnail. Due to the 6 connection limit on the same domain, the thumbnails were being blocked as well as the next AJAX request to get the next 60 cards if a user continues to scroll down."
"We would love to use HTTP/2 but most of our customers use IE11! We are therefore exploring domain sharding / loading AJAX JSON data requests off a different domain."

Sizing #

"Sorry for intrinsicsize; leveraging height/width seems better to me."
"Looking for a way to generate less sizes, right now it's ~12."
"Dynamic resizing of images is really hard and impossible without JS."
"A tool like responsivebreakpoints.com is good for web.dev :)."

High quality and high-resolution images #

"How to download compress images without losing DPI quality?"
"We're a document management company. Our apps handle MILLIONS of hi-res scanned images, usually TIFF or PDF."
"It's a hassle. Hi-res img files are necessary for print format; must be optimized for web. It's a hassle to downsize images for web but it's a show-stopper if authors only supply lightweight files for images destined for print publication. We wind up giving mixed messages about requirements for submission of manuscripts with artwork. We then wind up with complex workflows for processing those materials."

Browser capability #

"Auto responsive src crop from browser as [built-in] feature would be very useful as it is time consuming to crop all images to 4 sizes and writing all the markup. If we can upload one large photo and writing a simple picture tag that browsers will automatically create the multiple src attributes that would be a winning feature."
"Personally I'm having a hard time avoiding page reflows when image with is set by CSS for responsive images (max-width: 100%; height auto or height: width: 100%; height auto), especially in combination with art direction from adaptive images/picture tag. Best way to avoid seems to use the "negative padding hack" for a fixed image ratio and then position the image inside this ratio box. Better browser support/responsive image handling would be a really great help!"
"Please disable GIF "autoplay" by fetching just the first frame."

CDNs and image services #

"Google should provide a free CDN like Cloudflare."
"Maybe more tooling to set up dynamic scaling and CDNs with different providers would be nice."
"A single oversized overcompressed image is a very decent solution with no extra production cost. You need around 1000 pixels wide images for mobile (500px render width) and that is also the size you need for large/desktop non-retina displays. I think images resize CDNs are a very bad solution, although I have used it in the past. The CMS should handle the resizing and of that is too complex to set up, a single solution is a good compromise (for now)."
"CloudFlare auto-scales our images to best match the user's display. So we can save on loading time because images are loaded in relative to the user's display. For example, if a user is on a phone, it won't load in a desktop-sized background."
"Cloudflare does this in the background without us having to do anything except check a box in our settings panel."
"Just to reiterate, the only reason I can successfully use srcset, etc. is due to the ease of Cloudinary. But Cloudinary gets expensive, really fast. this feels like a major hole in the development experience."
"We need a way to easily auto crop images in a smart way so they can work with different aspect ratios in different contexts."
"I also use images from Other providers like Unsplash where there is very less control of resolution, quality and compression."

CMS, platform, and framework #

"I still struggle to find out what is the best way to use images, when I am building a site using a CMS. Authors tend to configure images with different dimensions and expect images not to shrink or scale. I am not sure if it is ok to set max-width or max-height on images"
"Been using gatsby-image for the last few projects and have never looked back."
"Images are often the hard part as they are put into CMS by end user, they may use any size, format, sometimes original image in ideal image format and dimensions are not available."
"Images are difficult to maintain since our system is self-serve adding controls is difficult unless things happen automatically without affecting resolution. Also for us images don't look correct in mobile vs desktop"
"I help people to optimize their sites (WordPress). The biggest problems I've seen for images are: Need to depend on a CDN or plugins to create WebP. srcset/picture has to be coded properly by theme developers. Most of the lazy loading plugins loads slowly giving bad UX. Background images are hard to lazy load."

Cost/benefit #

"The new practices are effective but increase the development time of the sites."
"The lack of adherence to the new standards such as srcset and WebP has been slow to be adopted by many Fortune 500 companies. Seeing this, many companies have resisted the change as an unnecessary development cost for current websites. The performance gains are not widely discussed or reported by the end user (UX). If anything, it makes it somewhat harder to save images from the web."
"Costly to create and manage multiple sizes, versions."
"They take up a lot of space on our server."

SEO #

"It's difficult to balance between acceptable image quality and file size. On one hand, I want fast loading for the SEO benefit, but on the other hand, poor quality images will detract from the UI/UX."

The role of images on the web #

"There are too many on the web. Stop using useless images that don't enhance the written content."
"Do you still remember the time when the web didn't have images and we shared selfies as ASCII-art?"

Tooling, guidance, standards and best practice: frustrations and requests #

[One participant wrote a blog post in response to this survey]
"The requirements seems to constantly change. As a web developer it is extremely frustrating because it is time consuming to save out the images in the first place. We optimize the best we can, we check the site and then months later Google has decided that the images could be even more compressed or need to be in a different format. This prevents us from providing the best possible solution to our client that lasts and instead creates a costly endeavor for them and us. Some of our small business clients simply don't have the budget for us to keep fixing images and re-saving them out to adhere to the requirements. We don't have the budget to do this work within their management packages. Writing the code to call different image sizes for different devices is also time consuming. It would be great to come up with a system of saving out images that would be consistent for a longer period of time."
"Yes, I think you got Keep Request Counts Low And File Sizes Small all wrong in Lighthouse. If a site serves over HTTP1.x then sure, but if a site serves over HTTP2 then the number of requests is less important or not even an issue if originating from the same hostname. I have a lite website, but I load 30 small WebP files of approx 35 requests total, over HTTP2 on the same hostname. Lighthouse is flagging this as an "Keep Request Counts Low And File Sizes Small" issue whereas it is superfast and because of the HTTP2 on the same hostname the number of requests are not a problem. And yes, the files are already small (most between 1 KB and 2 KB or less). I could load a sprite but then more CSS computing needs to be done. So please update the "Keep Request Counts Low And File Sizes Small" report in Lighthouse to take HTTP2 over same hostname into account."
"It has been a struggle for people to remember to compress their images."
"Cross browser behavior remains unpredictable so the simplest solutions are often the safest."

Top tips for web performance

2019-06-24T00:00:00Z

According to HTTP Archive, a typical mobile web page weighs over 2.6 MB, and more than two thirds of that weight is images. That's a great opportunity for optimization!

Average mobile page bytes by content type

Summary #

Don't save images larger than their display size.
Save multiple sizes for each image and use the srcset attribute to enable the browser to choose the smallest. The w value tells the browser the width of each version:

<img src="small.jpg"
     srcset="small.jpg 500w,
             medium.jpg 1000w,
             large.jpg 1500w"
     alt="…">

Save images with the right size #

You can make your website faster and less data hungry by using images with dimensions that match the display size. In other words, give images the right width and height when you save them.

Take a look at the images below.

They appear nearly identical, but the file size of one is more than 10 times larger than the other.

Saved width 1000 px, file size 184 KB

Saved width 300 px, file size 16 KB

The first image is much larger in file size because it's saved with dimensions much larger than the display size. Both images are displayed with a fixed width of 300 pixels, so it makes sense to use an image saved at the same size.

For fixed widths, use images saved with the same dimensions as the display size.

But… what if display size varies? #

In a multi-device world, images aren't always displayed at a single fixed size.

Image elements might have a percentage width, or be part of responsive layouts where image display sizes change to fit the screen size.

…and what about pixel-hungry devices like Retina displays?

Help the browser choose the right image size #

Wouldn't it be great if you could make each image available at different sizes, then let the browser choose the best size for the device and display size? Unfortunately there's a catch-22 when it comes to working out which image is best. The browser should use the smallest possible image, but it can't know the width of an image without downloading it to check.

This is where srcset comes in handy. You save images at different sizes, then tell the browser the width of each version:

<img src="small.jpg"
     srcset="small.jpg 500w, medium.jpg 1000w, large.jpg 1500w"
     alt="…">

The w values show the width of each image in pixels. For example, small.jpg 500w tells the browser that small.jpg is 500 pixels wide. This enables the browser to choose the smallest possible image, depending on the screen type and the viewport size—without having to download images to check their size.

You can see srcset in action for the image below. If you're on a laptop or desktop computer, change your browser window size and reopen this page. Then use the Network panel of your browser tools to check which image was used. (You'll need to do that in an Incognito or Private window, otherwise the original image file will be cached.)

How can I create multiple image sizes? #

You'll need to make multiple sizes available for every image you want to use with srcset.

For one-off images such as hero images you can manually save different sizes. If you have lots of images, such as product photos, you'll need to automate. For that there are two approaches.

Incorporate image processing in your build process #

As part of your build process, you can add steps to create different sized versions of your images. See "Use Imagemin to compress images" to learn more.

Use an image service #

Image creation and delivery can be automated using a commercial service like Cloudinary, or an open source equivalent such as Thumbor that you install and run yourself.

You upload your high resolution images, and the image service automatically creates and delivers different image formats and sizes depending on the URL parameters. For an example, open this sample image on Cloudinary and try changing the w value or the file extension in the URL bar.

Image services also have more advanced features such as the ability to automate "smart cropping" for different image sizes and automatically deliver WebP images to browsers that support the format, instead of JPEGs—without changing the file extension.

What if the image doesn't look right at different sizes? #

In that case, you'll need to use the <picture> element for "art direction": providing a different image or image crop at different sizes. To learn more take a look at the "Art direction" codelab.

What about pixel density? #

High-end devices have smaller (more dense) physical pixels. For example, a high-end phone might have two or three times as many pixels in each row of pixels as a cheaper device.

That can affect the size you need to save your images. We won't go into the gory details here, but you can find out more from the "Use density descriptors" codelab.

What about the display size of the image? #

You can use sizes to make srcset work even better.

Without it, the browser uses the full width of the viewport when choosing an image from a srcset. The sizes attribute tells the browser the width that an image element will be displayed, so the browser can choose the smallest possible image file—before it makes any layout calculations.

In the example below, sizes="50vw" tells the browser that this image will be displayed at 50% of the viewport width.

<img src="small.jpg"
     srcset="small.jpg 500w, medium.jpg 1000w, large.jpg 1500w"
     sizes="50vw"
     alt="…">

You can see this in action at simpl.info/sizes and the "Specifying multiple slot widths" codelab.

What about browser support? #

srcset and sizes are supported by over 90% of browsers globally.

If a browser does not support srcset or sizes it will fall back to just using the src attribute.

This makes srcset and sizes great progressive enhancements!

Learn more #

Take a look at the "Optimize your images" section of web.dev for a deeper dive into image optimization. For a more guided experience, consider trying the free "Responsive Images" course offered by Udacity.

Prework

2018-08-16T00:00:00Z

Before gathering performance metrics for a site audit, there are several checks you can do to identify easy fixes and areas for focus.

Validity check: architecture and code #

Pay down technical debt!

Wherever possible fix simple bugs and remove unneeded assets and code before measuring performance — but make sure to keep a before-and-after record of problems and fixes. These improvements can still be a part of your audit work.

Site architecture and assets
Can anything easily be removed from the code repo and from the site, such as unused legacy pages, content or other assets? Check for orphaned pages, redundant templates, unused images and unused code and libraries.

Runtime errors
Check for errors reported in the browser console. There shouldn't be any :).

Linting
Are there errors in your HTML, CSS or JavaScript code? Building linting into your workflow can help maintain code quality and avoid regressions. We recommend HTMLHint, StyleLint and ESLint, which can be used as code editor plugins, or run from the command line within workflow processes and continuous integration tools such as Travis.

Broken links and images
There are many tools to test for broken links at build time and runtime, including Chrome Extensions (this one is good) and Node tools such as Broken Link Checker.

Plugins
Plugins such as Flash and Silverlight can be a security risk, support for them has been deprecated, and they don't work on mobile. Use Lighthouse to check for plugins.

Test with a variety of devices and contexts #

Nothing beats getting real people to test your site with real devices, multiple browsers and different connectivity contexts.

Some of these checks are relatively subjective, but they can identify problems that affect perceived performance. Broken links, for example, waste time and feel 'unresponsive'. Illegible text is slow to read.

Cross-device testing
Try different viewport and window sizes. Use at least one mobile and one desktop device. If possible, try your site on a low-spec mobile device with a small screen. Is the text readable? Are any images broken? Can you zoom? Are touch targets large enough? Is it slow? Are any features unresponsive? Screenshot or video the results.

Cross-platform testing
What platforms do you target? You need to test on the browsers and operating systems your users use now and in the future.

Connectivity
Test on multiple target network types: connected, wifi and cellular. You can use browser tools to emulate a variety of network conditions.

Devices
Make sure to try out your site on the same devices as your users. The following photo shows the same page on two different phones.

On the larger screen, text is small but readable. On the smaller screen the browser renders the layout correctly, but the text is unreadable, even when zoomed in. The display is blurry and has a 'color cast' — white doesn't look white — making content less legible.

Simple findings such as this can be far more important than obscure performance data!

Try out UI and UX #

Accessibility, usability and readability
To ensure that your site's content and functionality are accessible to everyone, you need to understand the diversity of your users. Lighthouse and other tools test for specific accessibility problems, but nothing beats real-world testing. Try reading, navigating and entering data in a variety of scenarios: for example, outdoors in sunlight or on a train. Ask a range of friends, family and colleagues to try out your site. Try consuming content via a screen reader such as VoiceOver on Mac or NVDA on Windows.

You can find out more about implementing and reviewing accessibility in the Udacity course on Accessibility and the Web Fundamentals article How To Do an Accessibility Review.

Keep a record of your accessibility audit. Chances are that you'll be able to make simple improvements that are good for all your users.

Fundamental UI and UX problems
Interactions that don't work how they should, overflowing elements on smaller windows and viewports, too-small tap targets, unreadable content, janky scrolling… Open multiple pages on the site, try out navigation and all core functionality. Keep a record.

Images, audio and video
Test for overflowing content, incorrect aspect ratio, poor cropping, and quality problems.

Subjective UI tests
These may not all be relevant, but simple changes can make refactoring easier:

Is 'What can I do here? immediately clear when you open the site?
Are you drawn to consume content and follow links?
Are there visual hierarchies or pathways — or does everything have the same visual weight?
Is the layout cluttered?
Are there too many fonts?
Are there images or other content that could be removed?
Content design is as important as interface design. Is the text and image content on your site appropriate for mobile and desktop contexts? Can anything be eliminated? Write for mobile.

Auditing Performance

2018-08-16T00:00:00Z

Why and what? #

You've probably heard about all the good things that Progressive Web App techniques can do for your site. You might feel tempted to jump straight in and add PWA features. That's possible, but you'll be much better off if you get 'PWA-ready' first.

No amount of PWA magic will fix problems such as blocking JavaScript or bloated images. PWAs need a solid foundation.

So how do you check the health of your website? The first step is to do a site audit: an objective review of what works well and where (and how) there could be improvement.

Auditing your site or app will help you build a resilient, performant experience — and highlight quick wins that can be implemented with minimal sign-off. An audit also gives you a baseline for data-driven development. Did a change make things better? How does your site compare with competitors? You get metrics to prioritize effort, and concrete evidence to brag about once you've made improvements.

If you only have 5 minutes… #

Run Lighthouse on your homepage and save the report data. You get a quantified baseline and a todo list for improvements to performance, accessibility, security and SEO.

If you only have 30 minutes… #

Lighthouse is probably still the best place to start, but with a little more time you can also record results from other tools:

Chrome DevTools Security panel: HTTPS usage.
Chrome DevTools Network Panel: load timings; resource sizes and number of requests for HTML, CSS, JavaScript, images, fonts and other files.
Chrome Task Manager: if your site constantly uses significant CPU or more memory than other apps then you may need to fix memory leaks, task running or resource loading problems. Make sure to test your site on devices representative of your users.
WebPagetest: performance for different locations and connection types, caching, time to first byte, CDN usage.
Pagespeed Insights: load performance, data cost and resource usage, including Chrome User Experience report data highlighting real-world performance statistics.
Speed Scorecard and Impact Calculator: compare site speed against peers and estimate the potential revenue opportunity of improving site speed.

Make sure to test your website as a first-time user sees it. Open the site an Incognito (Private) Window, or use browser tools to disable caching and clear storage. This ensures that every asset is retrieved from the network and not from a local cache, so you get an accurate picture of first-load performance.

Nothing beats real world testing — try out your site with the same devices and connectivity as your users and keep a record of your subjective experience.

If you find the range of tools bewildering… #

Take a look at our guide: How To Think About Speed Tools.

If nothing else, simply use Lighthouse to check for:

HTTPS: every site should deliver all assets over HTTPS.
Server settings: your web server or CDN should use compression correctly, use HTTP/2, and include appropriate headers to enable your browser to cache resources.
Script elements that can be moved to the bottom of the page and/or given an async or defer attribute.
JavaScript and libraries that can be removed.
Unused CSS and unused JavaScript.
Images that can be saved with higher compression or smaller pixel dimensions.
Image files that would be smaller saved using a different format, for example photos saved as PNGs.

Audience, stakeholders, context #

Priorities for refactoring depend on your audience, content and functionality. Who visits your site? Why and how do they use it? What's your performance budget? If you're not sure of the answer to these questions, try the requirements gathering exercises from our PWA training resources: Your audience, your content and Design for all your users.

Who are your stakeholders, and what are their priorities? This will affect the way you structure, present and share your audit data.

If you can't audit your whole site, check page analytics to get an idea of where to focus. High bounce rates, low time-on-page and unexpected exit pages can be a good indicator of where to begin. Likewise business metrics such as hosting costs, ad clicks and conversions. Get an overview from stakeholders of what data matters to them.

Test, record, fix, repeat #

Record the state of your site before making any changes, to uncover problems and set a starting point for improvements or regressions. That gives you data to justify and reward development effort.

Make sure to test multiple page types within your site — not just the home page. For single page apps, test different components, routes and UX flows, and not just the first load experience.

Feedback #

Use tools to measure performance

2018-08-16T00:00:00Z

There are several core objectives for building a performant, resilient site with low data cost.

For each objective, you need an audit.

Objective	Why?	What to test for?
Ensure privacy, security and data integrity, and enable powerful API usage	Why HTTPS Matters	HTTPS implemented for all site pages/routes and assets.
Improve load performance	53% of users abandon sites that take longer than three seconds to load	JavaScript and CSS that could be loaded asynchronously or deferred. Set goals for time to interactive and payload size: for example TTI on 3G. Set a performance budget.
Reduce page weight	• Reduce data cost for users with capped data plans • Reduce web app storage requirements — particularly important for users on low-spec devices • Reduce hosting and serving costs • Improve serving performance, reliability and resilience	Set a page weight budget: for example, first load under 400 kB. Check for heavy JavaScript. Check file sizes to find bloated images, media, HTML, CSS and JavaScript. Find images that could be lazy loaded, and check for unused code with coverage tools.
Reduce resource requests	• Reduce latency issues • Reduce serving costs • Improve serving performance, reliability and resilience	Look for excessive or unnecessary requests for any type of resource. For example: files that are loaded repeatedly, JavaScript that is loaded in multiple versions, CSS that is never used, images that are never viewed (or could be lazy loaded).
Optimize memory usage	Memory can become the new bottleneck, especially on mobile devices	Use the Chrome Task Manager to compare your site against others for memory usage when loading the home page and using other site features.
Reduce CPU load	Mobile devices have limited CPU, especially low-spec devices	Check for heavy JavaScript. Find unused JavaScript and CSS with coverage tools. Check for excessive DOM size and scripts that run unnecessarily on first load. Look for JavaScript loaded in multiple versions, or libraries that could be avoided with minor refactoring.

There is a wide range of tools and techniques for auditing websites:

System tools
Built-in browser tools
Browser extensions
Online test applications
Emulation tools
Analytics
Metrics provided by servers and business systems
Screen and video recording
Manual tests

Below you'll learn which approach is relevant for each type of audit.

Images constitute by far the most weight and most requests for most web pages. Latency gets worse as connectivity gets worse so excessive image file requests are an increasing problem as the web goes mobile. Images also consume power: more image requests, more radio usage, more flat batteries. Even just to render images takes power – and this is proportional to the size and quantity of images. Likewise for memory: small increases in pixel dimensions result in big increases in memory usage. With images on mobile — especially on low-spec devices — memory can become the new bottleneck. Bloated images are also problematic for users on capped data plans. Remove redundant images! If you can't get rid of them, optimize: increase compression as much as possible, reduce pixel dimensions, and use the format that gives you the smallest file sizes. Optimizing 'hero images' such as banners and backgrounds is an easy, one-off win.

Record resource requests: number, size, type and timing #

A good place to start when auditing a site is to check pages with your browser's network tools. If you're not sure how to do this, work through the Chrome DevTools network panel Get Started Guide. Similar tools are available for Firefox, Safari, Internet Explorer and Edge.

Remember to keep a record of results before you make changes. For network requests, that can be as simple as a screenshot — you can also save profile data as a JSON file. There's more information below about how to save and share test results.

Before you begin auditing network usage, make sure to disable the browser cache to ensure you get accurate statistics for first-load performance. If you already do caching via a service worker, clear Cache API storage. You may want to use an Incognito (Private) window, so that you don't have to worry about disabling the browser cache or removing previously cached entries.

Here are some core features and metrics you should check with browser tools:

Load performance: Lighthouse provides a summary of load metrics. Addy Osmani has written a great summary of key user moments for page load.
Timeline events for loading and parsing resources, and memory usage. If you want to go deeper, run memory and JavaScript profiling.
Total page weight and number of files.
Number and weight of JavaScript files.
Any particularly large individual JavaScript files (over, say, 100KB).
Unused JavaScript. You can check using the Chrome coverage tool.
Total number and weight of image files.
Any particularly large individual image files.
Image formats: are there PNGs that could be JPEGs or SVGs? Is WebP used with fallbacks?
Whether responsive image techniques (such as srcset) are used.
HTML file size.
Total number and weight of CSS files.
Unused CSS. (In Chrome, use the coverage panel.)
Check for problematic usage of other assets such as Web Fonts (including icon fonts).
Check the DevTools timeline for anything that blocks page load.

If you're working from fast wifi or a fast cellular connection, test with low bandwidth and high latency emulation. Remember to test on mobile as well as desktop — some sites use UA sniffing to deliver different assets and layouts for different devices. You may need to test on actual hardware using remote debugging, not just with device simulation.

Check memory and CPU load #

Before you make changes, keep a record of memory and CPU usage.

In Chrome you can access the Task Manager from the Window menu. This is a simple way to check a web page's requirements.

Chrome's Task Manager — watch out for memory and CPU hogs!

Test first and subsequent load performance #

Lighthouse, WebPagetest and Pagespeed Insights are useful for analyzing speed, data cost and resource usage. WebPagetest will also check static-content caching, time to first byte, and if your site makes effective use of CDNs.

Save the results #

WebPagetest: test results each have their own URL.
Pagespeed Insights: the online Pagespeed Insights tool now includes Chrome User Experience report data highlighting real-world performance stats.
Lighthouse: save reports from the Chrome DevTools Audit panel by clicking on the download button:

Test for core Progressive Web App requirements #

Lighthouse helps you test security, functionality, accessibility, performance and search engine performance. In particular, Lighthouse checks if your site successfully implements PWA features such as service workers and a Web App manifest.

Lighthouse also tests whether your site can provide an acceptable offline experience.

You can download a Lighthouse report as JSON or, if you're using the Lighthouse Chrome Extension, share the report as a GitHub Gist: click on the share button, select Open in Viewer, then click on the share button again in the new window and Save as Gist.

Export a report to a gist from the Lighthouse Chrome Extension — click the share button

Use analytics, event tracking and business metrics to track real-world performance #

If you can, keep a record of analytics data before you implement changes: bounce rates, time on page, exit pages: whatever's relevant to your business requirements.

If possible, record business and technical metrics that might be affected, so you can compare results after making changes. For example: an e-commerce site might track orders-per-minute or record stats for stress and endurance testing. Back-end storage costs, CPU requirements, serving costs and resilience are likely to improve if you cut page weight and resource requests.

If analytics aren't implemented, now is the time! Business metrics and analytics are the final arbiter of whether or not your site is working. If appropriate, incorporate event tracking for user actions such as button clicks and video plays. You may also want to implement goal flow analysis: the paths by which your users navigate towards 'conversions'.

You can keep an eye on Google Analytics Site Speed to check how performance metrics correlate with business metrics. For example: 'how fast did the homepage load?' compared to 'did entry via the home page result in a sale?'

Google Analytics uses data from the Navigation Timing API.

You may want to record data using one of the JavaScript performance APIs or your own metrics, for example:

    const subscribeBtn = document.querySelector('#subscribe');

    subscribeBtn.addEventListener('click', (event) => {
     // Event listener logic goes here...

     const lag = performance.now() - event.timeStamp;
     if (lag > 100) {
      ga('send', 'event', {
       eventCategory: 'Performance Metric'
       eventAction: 'input-latency',
       eventLabel: '#subscribe:click',
       eventValue: Math.round(lag),
       nonInteraction: true,
      });
     }
    });

You can also use ReportingObserver to check for browser deprecation and intervention warnings. This is one of many APIs for getting real-world, live measurements from actual users.

Real-world experience: screen and video recording #

Make a video recording of page load on mobile and desktop. This works even better at high frame rates and if you add a timer display.

You may also want to save screencasts. There are many screencast recording apps for Android, iOS, and desktop platforms (and scripts to do the same).

Video-recording page load works much like the filmstrip view in WebPagetest or Capture Screenshots in Chrome DevTools. You get a real-world record of page component load speed: what's fast and what's slow. Save video recordings and screencasts to compare against later improvements.

A side-by-side before-and-after comparison can be a great way to demonstrate improvements!

What else? #

If relevant, get a Web Bloat Score. This is a fun test, but it can also be a compelling way to demonstrate code bloat — or to show you've made improvements.

What Does My Site Cost?, shown below, gives a rough guide to the financial cost of loading your site in different regions.

Many other standalone and online tools are available: take a look at perf.rocks/tools.

Next steps

2018-08-16T00:00:00Z

Having completed a site audit, you will have accurate review data in a form that makes it easy for developers and other stakeholders to prioritize and justify changes.

Next, you may want to revisit the sections on this site that provide in-depth advice on how to improve load performance and rendering performance.

Find out more #

perf.rocks: resources to help you build fast sites
Web Fundamentals Performance guide: why performance matters and how to improve it
How To Think About Speed Tools: overview of guidance and tools for developers and marketers from Google
High Performance Web Sites by Steve Souders: more than 10 years old, but still valuable
High Performance Browser Networking by Ilya Grigorik: comprehensive reference including introductory guides to TCP, UDP, TLS and other topics
Can You Afford It?: Real-world Web Performance Budgets

Check site security

2018-08-16T00:00:00Z

You won't be able to build a PWA without HTTPS.

Serving your site over HTTPS is fundamental for security, and many APIs won't work without it. If you need to justify implementation costs, find out why HTTPS matters.

If a site uses HTTP for any assets, users will be warned in the URL bar. Chrome displays a warning like the following.

From Chrome 68, the address bar warns if not all assets use HTTPS

HTTPS should be implemented everywhere — not just, for example, on login or checkout pages. Any insecure page or asset can be a vector for attack, making your site a liability for your users and your business.

Site security is easy to check with Chrome DevTools Security panel. Keep a record of any problems.

The site in the following example is not secure, since some assets are served over HTTP.

Chrome DevTools Security panel

Share the results

2018-08-16T00:00:00Z

Once you've audited a site, make sure to package the results in a digestible form.

Consider producing different reports for different stakeholders.
Focus on business needs and show how technical metrics support these.
Start with a summary.
Structure data by topic (such as load performance and page weight) rather than simply listing tool output data.
Order results by priority.
Leave out results if they're not relevant or interesting.
Where possible present numerical data as charts or graphs.
Avoid a wall of data — site reviews should not be boring.

Be sensitive to the people on the receiving end of your audit.

Those working on a site may be well aware of problems. There may be complex, non-technical reasons why problems haven't been fixed.

It's much more helpful to describe poor performance in terms of opportunities and solutions, rather than simply listing a catalog of failures.

Wherever possible, talk to site developers and other stakeholders before presenting your findings more widely.

Provide context #

When sharing review results you may want to include contextual data to justify effort and motivate developers or other stakeholders to implement improvements you suggest — such as the following from DoubleClick:

53% of users abandon sites that take longer than three seconds to load.
Mobile sites load in 5 seconds earn up to 2x more mobile ad revenue.
The average load time for mobile sites is 19 seconds.

There is a comprehensive list of business reasons for improving performance on neotys.com. More information about how to improve site performance is available from perf.rocks and Web Fundamentals, along with case studies and success stories.

If you don't have a performance budget, now is the time! Calculate a budget and show how your site weighs up.

Demonstrate potential #

Chrome DevTools Local Overrides allow you to override website assets with local versions. This can be a great way to show how simple changes can make a big difference.

For example:

Create a version of the CSS used by a site's homepage, with redundant rules removed.
Change HTML to defer JavaScript loading.
Replace image files with optimized versions.

You can even share changed files with developers working on a site, to enable them to demonstrate potential improvements directly to their colleagues. Local Overrides also makes it simple to create side-by-side screencasts showing performance differences between optimized and unoptimized versions. This approach can be much more compelling than a lengthy todo list! Find out how to use Local Overrides here.

Multi-device content

2016-05-10T00:00:00Z

How people read on the web #

The US government writing guide summarizes what people want from writing on the web:

Research shows that people don't read web pages, they scan. On average, people only read 20–28% of web page content. Reading from screens is much slower than reading from paper. People will give up and leave your site unless information is easy to access and understand.

How to write for mobile #

Focus on the subject at hand and tell the story upfront. For writing to work across a range of devices and viewports, make sure to get your main points across at the start: as a rule, ideally in the first four paragraphs, in around 70 words.

Ask yourself what people want from your site. Are they trying to find something out? If people visit your site for information, make sure that all your text is oriented to helping them achieve their goal. Write in the active voice, offer actions and solutions.

Publish only what your visitors want, and nothing more.

UK government research also shows that:

In other words: use plain language, shorter words and simple sentence structures — even for a literate, technical audience. Unless there's a good reason not to, keep your tone of voice conversational. An old rule of journalism is to write as if you are speaking to an intelligent 11 year old.

The next billion users #

The pared-down approach to writing is particularly important for readers on mobile devices, and is crucial when creating content for low-cost phones with small viewports that require more scrolling and may have lower quality displays and less responsive screens.

Most of the next billion users coming online will have cheap devices. They will not want to spend their data budget on navigating long-winded content, and may not be reading in their first language. Trim your text: use short sentences, minimal punctuation, paragraphs five lines or less, and single line headings. Consider responsive text (for example, using shorter headlines for smaller viewports) but beware of the downsides.

A minimalist attitude to text will also make your content easier to localize and internationalize — and make it more likely that your content gets quoted in social media.

The bottom line:

Keep it simple
Reduce clutter
Get to the point

Eliminate unnecessary content #

In terms of byte size, web pages are big and getting bigger.

Responsive design techniques make it possible to serve different content for smaller viewports, but it's always sensible to start by streamlining text, images and other content.

Web users are often action oriented, "leaning forward" in the hunt for answers to their current question, rather than leaning back to absorb a good book.

Jackob Nielsen

Ask yourself: what are people are trying to achieve when they visit my site?

Does every page component help users achieve their goal?

Remove redundant page elements #

HTML files constitute nearly 70k and more than nine requests for the average web page, according to HTTP Archive.

Many popular sites use several thousand HTML elements per page, and several thousand lines of code, even on mobile. Excessive HTML file size may not make pages load more slowly, but a heavy HTML payload can be a sign of content bloat: larger .html files mean more elements, more text content, or both.

Reducing HTML complexity will also reduce page weight, help enable localization and internationalization and make responsive design easier to plan and debug. For information about writing more efficient HTML, see High performance HTML.

Every step you make a user perform before they get value out of your app will cost you 20% of users

Gabor Cselle, Twitter

The same applies to content: help users get to what they want as quickly as possible.

Don't just hide content from mobile users. Aim for content parity, since guessing what features your mobile users won't miss is bound to fail for someone. If you have the resources, create alternative versions of the same content for different viewport sizes — even if only for high priority page elements.

Consider content management and workflow: are legacy systems resulting in legacy content?

Simplify text #

As the web goes mobile, you need to change the way you write. Keep it simple, reduce clutter and get to the point.

Remove redundant images #

According to HTTP Archive data, the average web page makes 54 requests for images.

Images can be beautiful, fun and informative — but they also use page real estate, add to page weight, and increase the number of file requests. Latency gets worse as connectivity gets worse, meaning that an excess of image file requests is an increasing problem as the web goes mobile.

Images constitute over 60% of page weight.

Images also consume power. After the screen, radio is the second biggest drain on your battery. More image requests, more radio usage, more flat batteries. Even just to render images takes power – and this is proportional to size and number. Check out the Stanford report Who Killed My Battery?

If you can, get rid of images!

Here are some suggestions:

Consider designs that avoid images altogether, or use images sparingly. Text-only can be beautiful! Ask yourself, "What are visitors to my site trying to achieve? Do images help that process?"
In the old days, it was commonplace to save headings and other text as graphics. That approach does not respond well to viewport size changes, and adds to page weight and latency. Using text as a graphic also means the text can't be found by search engines, and isn't accessible by screenreaders and other assistive technologies. Use "real" text where possible — Web Fonts and CSS can enable beautiful typography.
Use CSS rather than images for gradients, shadows, rounded corners, and background textures, features supported by all modern browsers. Bear in mind, however, that CSS may be better than images, but there can still be a processing and rendering penalty, especially significant on mobile.
Background images rarely work well on mobile. You can use media queries to avoid background images on small viewports.
Avoid splash screen images.
Use CSS for UI animations.
Get to know your glyphs; use Unicode symbols and icons instead of images, with Web Fonts if necessary.
Consider icon fonts; they are vector graphics that can be infinitely scaled, and an entire set of images can be downloaded in one font. (Be aware of these concerns, however.)
The <canvas> element can be used to build images in JavaScript from lines, curves, text, and other images.
Inline SVG or Data URI images will not reduce page weight, but they can reduce latency by reducing the number of resource requests. Inline SVG has great support on mobile and desktop browsers, and optimization tools can significantly reduce SVG size. Likewise, Data URIs are well supported. Both can be inlined in CSS.
Consider using <video> instead of animated GIFs. The video element is supported by all browsers on mobile (apart from Opera Mini).

For more information see Image Optimization and Eliminating and replacing images.

Design content to work well across different viewport sizes #

"Create a product, don't re-imagine one for small screens. Great mobile products are created, never ported."

Mobile Design and Development, Brian Fling

Great designers don't "optimize for mobile" — they think responsively to build sites that work across a range of devices. The structure of text and other page content is critical to cross-device success.

Many of the next billion users coming online use low-cost devices with small viewports. Reading on a low resolution 3.5" or 4" screen can be hard work.

Here is a photograph of the two together:

On the larger screen, text is small but readable.

On the smaller screen the browser renders the layout correctly, but the text is unreadable, even when zoomed in. The display is blurry and has a 'color cast' — white doesn't look white — making content less legible.

Design content for mobile #

When building for a range of viewports, consider content as well as layout and graphic design, design with real text and images, not placeholder content.

"Content precedes design. Design in the absence of content is not design, it's decoration."

Jeffrey Zeldman

Put your most important content at the top, since users tend to read web pages in an F-shaped pattern.
Users visit your site to achieve a goal. Ask yourself what they need to achieve that goal and get rid of everything else. Get tough on visual and textual embellishments, legacy content, excessive links, and other clutter.
Be careful with social sharing icons; they can clutter layouts, and the code for them can slow down page loading.
Design responsive layouts for content, not fixed device sizes.

Test content #

Check readability on smaller viewports using Chrome DevTools and other emulation tools.
Test your content under conditions of low bandwidth and high latency; try out content in a variety of connectivity scenarios.
Try reading and interacting with your content on a low-cost phone.
Ask friends and colleagues to try out your app or site.
Build a simple device test lab. The GitHub repo for Google's Mini Mobile Device Lab has instructions on how to build your own. OpenSTF is a simple web application for testing websites on multiple Android devices.

Here is OpenSTF in action:

Mobile devices are increasingly used to consume content and obtain information — not just as devices for communication, games and media.

This makes it increasingly import to plan content to work well on a range of viewports, and to prioritize content when considering cross-device layout, interface and interaction design.

Understand data cost #

Web pages are getting bigger.

According to HTTP Archive, the average page weight for the top one million sites is now over 2MB.

Users avoid sites or apps perceived to be slow or expensive, so it's crucial to understand the cost of loading page and app components.

Reducing page weight can also be profitable. Chris Zacharias from YouTube found that when they reduced the watch-page size from 1.2MB to 250KB:

In other words, reducing page weight can open up whole new markets.

Calculate page weight #

There are a number of tools for calculating page weight. The Chrome DevTools Network panel shows the total byte size for all resources, and can be used to ascertain weights for individual asset types. You can also check which items have been retrieved from the browser cache.

Firefox and other browsers offer similar tools.

WebPagetest provides the ability to test first and subsequent page loads. You can automate testing with scripts (for example, to log in to a site) or by using their RESTful APIs. The following example (loading developers.google.com/web) shows that caching was successful and that subsequent page loads required no additional resources.

WebPagetest also gives a size and request breakdown by MIME type.

Calculate page cost #

For many users, data doesn't just cost bytes and performance — it costs money.

The site What Does My Site Cost? enables you to estimate the actual financial cost of loading your site. The histogram below shows how much it costs (using a prepaid data plan) to load amazon.com.

Bear in mind that this doesn't take into account affordability relative to income. Data from blog.jana.com shows the cost of data.

	500MB data plan cost (USD)	Hourly minimum wage (USD)	Hours of work to pay for 500MB data plan
India	$3.38	$0.20	17 hours
Indonesia	$2.39	$0.43	6 hours
Brazil	$13.77	$1.04	13 hours

Page weight isn't just a problem for emerging markets. In many countries, people use mobile plans with limited data, and will avoid your site or app if they perceive it to be heavy and expensive. Even "unlimited" cell and wifi data plans generally have a data limit beyond which they are blocked or throttled. For these reasons, it's best to be as transparent as possible about how much data your page consumes. The following blog post provides specific best practices: Nurture trust through cost transparency

The bottom line: page weight affects performance and costs money. Optimizing content efficiency shows how to reduce that cost.

Understanding Low Bandwidth and High Latency

2016-05-09T00:00:00Z

It's important to understand what using your app or site feels like when connectivity is poor or unreliable, and build accordingly. A range of tools can help you.

Test with low bandwidth and high latency #

An increasing proportion of people experience the web on mobile devices. Even at home, many people are abandoning fixed broadband for mobile.

In this context, it's important to understand what using your app or site feels like when connectivity is poor or unreliable. A range of software tools can help you emulate and simulate low bandwidth and high latency.

Emulate network throttling #

When building or updating a site, you must ensure adequate performance in a variety of connectivity conditions. Several tools can help.

Browser tools #

Chrome DevTools lets you test your site with a variety of upload/download speeds and round-trip times, using presets or custom settings from the Network panel. See Get Started with Analyze Network Performance to learn the basics.

System tools #

Network Link Conditioner is a preference panel available on Mac if you install Hardware IO Tools for Xcode:

Device emulation #

Android Emulator allows you to simulate various network conditions while running apps (including web browsers and hybrid web apps) on Android:

For iPhone, Network Link Conditioner can be used to simulate impaired network conditions (see above).

Test from different locations and networks #

Connectivity performance depends on server location as well as network type.

WebPagetest is an online service that enables a set of performance tests to be run for your site using a variety of networks and host locations. For example, you can try out your site from a server in India on a 2G network, or over cable from a city in the US.

Select a location and, from advanced settings, select a connection type. You can even automate testing using scripts (for example, to log in to a site) or using their RESTful APIs. This helps you to include connectivity testing into build processes or performance logging.

Fiddler supports Global proxying via GeoEdge, and its custom rules can be used to simulate modem speeds:

Test on an impaired network #

Software and hardware proxies enable you to emulate problematic mobile network conditions, such as bandwidth throttling, packet delay, and random packet loss. A shared proxy or impaired network can enable a team of developers to incorporate real-world network testing in their workflow.

Facebook's Augmented Traffic Control (ATC) is a BSD-licensed set of applications that can be used to shape traffic and emulate impaired network conditions:

Facebook even instituted 2G Tuesdays to help understand how people on 2G use their product. On Tuesdays, employees get a pop-up that gives them the option to simulate a 2G connection.

The Charles HTTP/HTTPS proxy can be used to adjust bandwidth and latency. Charles is commercial software, but a free trial is available.

More information about Charles is available from codewithchris.com.

Handle unreliable connectivity and "lie-fi" #

What is lie-fi? #

The term lie-fi dates back to at least 2008 (when phones looked like this), and refers to connectivity that isn't what it seems. Your browser behaves as if it has connectivity when, for whatever reason, it doesn't.

Misinterpreted connectivity can result in a poor experience as the browser (or JavaScript) persists in trying to retrieve resources rather than giving up and choosing a sensible fallback. Lie-fi can actually be worse than offline; at least if a device is definitely offline, your JavaScript can take appropriate evasive action.

Lie-fi is likely to become a bigger problem as more people move to mobile and away from fixed broadband. Recent US Census data shows a move away from fixed broadband. The following chart shows the use of mobile internet at home in 2015 compared with 2013:

Use timeouts to handle intermittent connectivity #

In the past, hacky methods using XHR have been used to test for intermittent connectivity, but service worker enables more reliable methods to set network timeouts. This can be achieved using Workbox with only a few lines of code:

workboxSW.router.registerRoute(
  '/path/to/image',
  workboxSW.strategies.networkFirst({networkTimeoutSeconds: 3}),
);

You can learn more about Workbox in Jeff Posnick's Chrome Dev Summit talk, Workbox: Flexible PWA Libraries.

Timeout functionality is also being developed for the Fetch API, and the Streams API should help by optimizing content delivery and avoiding monolithic requests. Jake Archibald gives more details about tackling lie-fi in Supercharging page load.

Feedback #

The

2014-02-15T00:00:00Z

You've properly prepared a video file for the web. You've given it correct dimensions and the correct resolution. You've even created separate WebM and MP4 files for different browsers.

For anyone to see your video, you still need to add it to a web page. Doing so properly requires adding two HTML elements: the <video> element and the <source> element. In addition to basics about these tags, this article explains attributes you should add to those tags to craft a good user experience.

Specify a single file #

Although it's not recommended, you can use the video element by itself. Always use the type attribute as shown below. The browser uses this to determine if it can play the provided video file. If it can't, the enclosed text displays.

<video src="chrome.webm" type="video/webm">
    <p>Your browser cannot play the provided video file.</p>
</video>

Specify multiple file formats #

Recall from Media file basics that not all browsers support the same video formats. The <source> element lets you specify multiple formats as a fallback in case the user's browser doesn't support one of them.

The example below produces the embedded video that is used as an example later in this article.

<video controls>
  <source src="https://storage.googleapis.com/web-dev-assets/video-and-source-tags/chrome.webm" type="video/webm">
  <source src="https://storage.googleapis.com/web-dev-assets/video-and-source-tags/chrome.mp4" type="video/mp4">
  <p>Your browser cannot play the provided video file.</p>
</video>

Try it on Glitch (source)

You should always add a type attribute to the <source> tags event though it is optional. This ensures that the browser only downloads the file that it is capable of playing.

This approach has several advantages over serving different HTML or server-side scripting, especially on mobile:

You can list formats in order of preference.
Client-side switching reduces latency; only one request is made to get content.
Letting the browser choose a format is simpler, quicker, and potentially more reliable than using a server-side support database with user-agent detection.
Specifying each file source's type improves network performance; the browser can select a video source without having to download part of the video to "sniff" the format.

These issues are especially important in mobile contexts, where bandwidth and latency are at a premium, and the user's patience is likely limited. Omitting the type attribute can affect performance when there are multiple sources with unsupported types.

There are a few ways you can dig into the details. Check out A Digital Media Primer for Geeks to find out more about how video and audio work on the web. You can also use remote debugging in DevTools to compare network activity with type attributes and without type attributes.

Specify start and end times #

Save bandwidth and make your site feel more responsive: use media fragments to add start and end times to the video element.

This browser does not support the video element.

To use a media fragment, add #t=[start_time][,end_time] to the media URL. For example, to play the video from seconds 5 to 10, specify:

<source src="video/chrome.webm#t=5,10" type="video/webm">

You can also specify the times in <hours>:<minutes>:<seconds>. For example, #t=00:01:05 starts the video at one minute, five seconds. To play only the first minute of video, specify #t=,00:01:00.

You can use this feature to deliver multiple views on the same video—like cue points in a DVD–without having to encode and serve multiple files.

For this feature to work, your server must support range requests and that capability must be enabled. Most servers enable range requests by default. Because some hosting services turn them off, you should confirm that range requests are available for using fragments on your site.

Fortunately, you can do this in your browser developer tools. In Chrome, for instance, it's in the Network panel. Look for the Accept-Ranges header and verify that it says bytes. In the image, I've drawn a red box around this header. If you do not see bytes as the value, you'll need to contact your hosting provider.

Chrome DevTools screenshot: Accept-Ranges: bytes.

Include a poster image #

Add a poster attribute to the video element so that viewers have an idea of the content as soon as the element loads, without needing to download the video or start playback.

<video poster="poster.jpg" ...>
  …
</video>

A poster can also be a fallback if the video src is broken or if none of the supplied video formats are supported. The only downside to a poster images is an additional file request, which consumes some bandwidth and requires rendering. For more information see Efficiently encode images.

Don't

Without a fallback poster, the video just looks broken.

A fallback poster makes it seem as if the first frame has been captured.

Ensure videos don't overflow containers #

When video elements are too big for the viewport, they may overflow their container, making it impossible for the user to see the content or use the controls.

Android Chrome screenshot, portrait: unstyled video element overflows viewport.

Android Chrome screenshot, landscape: unstyled video element overflows viewport.

You can control video dimensions using CSS. If CSS does not meet all of your needs, JavaScript libraries and plugins such as FitVids (outside the scope of this article) can help, even for videos from YouTube and other sources. Unfortunately, these resources can increase your network payload sizes with negative consequences for your revenues and your users' wallets.

For simple uses like the ones I'm describing here, use CSS media queries to specify the size of elements depending on the viewport dimensions; max-width: 100% is your friend.

For media content in iframes (such as YouTube videos), try a responsive approach (like the one proposed by John Surdakowski).

CSS #

.video-container {
    position: relative;
    padding-bottom: 56.25%;
    padding-top: 0;
    height: 0;
    overflow: hidden;
}

.video-container iframe,
.video-container object,
.video-container embed {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
}

HTML #

<div class="video-container">
  <iframe
    src="//www.youtube.com/embed/l-BA9Ee2XuM"
    frameborder="0"
    width="560"
    height="315"
  ></iframe>
</div>

Try it

Compare the responsive sample to the unresponsive version. As you can see, the unresponsive version isn't a great user experience.

Device orientation #

Device orientation isn't an issue for desktop monitors or laptops, but it's hugely important when considering web page design for mobile devices and tablets.

Safari on iPhone does a good job of switching between portrait and landscape orientation:

Screenshot of video playing in Safari on iPhone, portrait.

Screenshot of video playing in Safari on iPhone, landscape.

Device orientation on an iPad and Chrome on Android can be problematic. For example, without any customization a video playing on an iPad in landscape orientation looks like this:

Screenshot of video playing in Safari on iPad, landscape.

Setting the video width: 100% or max-width: 100% with CSS can resolve many device orientation layout problems.

Autoplay #

The autoplay attribute controls whether the browser downloads and plays a video immediately. The precise way it works depends on the platform and browser.

Chrome: Depends on multiple factors including but not limited to whether the viewing is on desktop and whether the mobile user has added your site or app to their homescreen. For details, see Autoplay best practices.
Firefox: Blocks all video and sound, but gives users the ability to relax these restrictions for either all sites or particular sites. For details, see Allow or block media autoplay in Firefox
Safari: Has historically required a user gesture, but has been relaxing that requirement in recent versions. For details, see New <video> Policies for iOS.

Even on platforms where autoplay is possible, you need to consider whether it's a good idea to enable it:

Data usage can be expensive.
Playing media before the user wants it can hog bandwidth and CPU, and thereby delay page rendering.
Users may be in a context where playing video or audio is intrusive.

Preload #

The preload attribute provides a hint to the browser as to how much information or content to preload.

Value	Description
`none`	The user might chose not to watch the video, so don't preload anything.
`metadata`	Metadata (duration, dimensions, text tracks) should be preloaded, but with minimal video.
`auto`	Downloading the entire video right away is considered desirable. An empty string produces the same result.

The preload attribute has different effects on different platforms. For example, Chrome buffers 25 seconds of video on desktop but none on iOS or Android. This means that on mobile, there may be playback startup delays that don't happen on desktop. See Fast playback with audio and video preload or Steve Souders' blog for more details.

Now that you know how to add media to your web page it's time to learn about Media accessibility where you will add captions to your video for hearing impaired, or when playing the audio is not a viable option.

Send data between browsers with WebRTC data channels

2014-02-04T00:00:00Z

Sending data between two browsers for communication, gaming, or file transfer can be a rather involved process. It requires setting up and paying for a server to relay data, and perhaps scaling this to multiple data centers. In this scenario, there is potential for high latency and it's difficult to keep data private.

These problems can be alleviated by using WebRTC's RTCDataChannel API to transfer data directly from one peer to another. This article covers the basics of how to set up and use data channels, as well as the common use cases on the web today.

Why another data channel? #

We have WebSocket, AJAX and Server Sent Events. Why do we need another communication channel? WebSocket is bidirectional, but all these technologies are designed for communication to or from a server.

RTCDataChannel takes a different approach:

It works with the RTCPeerConnection API, which enables peer-to-peer connectivity. This can result in lower latency - no intermediary server and fewer 'hops'.
RTCDataChannel uses Stream Control Transmission Protocol (SCTP), allowing configurable delivery semantics-out-of-order delivery and retransmit configuration.

RTCDataChannel is available now with SCTP support on desktop and Android in Google Chrome, Opera, and Firefox.

A caveat: Signaling, STUN, and TURN #

WebRTC enables peer-to-peer communication, but it still needs servers for signaling to exchange media and network metadata to bootstrap a peer connection.

WebRTC copes with NATs and firewalls with:

The ICE framework to establish the best possible network path between peers.
STUN servers to ascertain a publicly accessible IP and port for each peer.
TURN servers if direct connection fails and data relaying is required.

For more information about how WebRTC works with servers for signaling and networking, see WebRTC inthe real world: STUN, TURN, and signaling.

The capabilities #

The RTCDataChannel API supports a flexible set of data types. The API is designed to mimic WebSocket exactly, and RTCDataChannel supports strings as well as some of the binary types in JavaScript, such as Blob, ArrayBuffer, and ArrayBufferView. These types can be helpful when working with file transfer and multiplayer gaming.

RTCDataChannel can work in unreliable and unordered mode (analogous to User Datagram Protocol or UDP), reliable and ordered mode (analogous to Transmission Control Protocol or TCP), and partial reliable modes:

Reliable and ordered mode guarantees the transmission of messages and also the order in which they are delivered. This takes extra overhead, thus potentially making this mode slower.
Unreliable and unordered mode does not guarantee every message gets to the other side nor what order they get there in. This removes the overhead, allowing this mode to work much faster.
Partial reliable mode guarantees the transmission of message under a specific condition, such as a retransmit timeout or a maximum amount of retransmissions. The ordering of messages is also configurable.

Performance for the first two modes is about the same when there are no packet losses. However, in reliable and ordered mode, a lost packet causes other packets to get blocked behind it, and the lost packet might be stale by the time that it is retransmitted and arrives. It is, of course, possible to use multiple data channels within the same app, each with their own reliable or unreliable semantics.

Here's a helpful table from High Performance Browser Networking by Ilya Grigorik:


TCP	UDP	SCTP
Reliability	Reliable	Unreliable	Configurable
Delivery	Ordered	Unordered	Configurable
Transmission	Byte-oriented	Message-oriented	Message-oriented
Flow control	Yes	No	Yes
Congestion control	Yes	No	Yes

Next, you learn how to configure RTCDataChannel to use reliable and ordered or unreliable and unordered mode.

Configuring data channels #

There are several simple demos of RTCDataChannel online:

In these examples, the browser makes a peer connection to itself, then creates a data channel and sends a message through the peer connection. It is then creating a data channel and sending the message along the peer connection. Finally, your message appears in the box on the other side of the page!

The code to get started with this is short:

const peerConnection = new RTCPeerConnection();

// Establish your peer connection using your signaling channel here
const dataChannel =
  peerConnection.createDataChannel("myLabel", dataChannelOptions);

dataChannel.onerror = (error) => {
  console.log("Data Channel Error:", error);
};

dataChannel.onmessage = (event) => {
  console.log("Got Data Channel Message:", event.data);
};

dataChannel.onopen = () => {
  dataChannel.send("Hello World!");
};

dataChannel.onclose = () => {
  console.log("The Data Channel is Closed");
};

The dataChannel object is created from an already-established peer connection. It can be created before or after signaling happens. You then pass in a label to distinguish this channel from others and a set of optional configuration settings:

const dataChannelOptions = {
  ordered: false, // do not guarantee order
  maxPacketLifeTime: 3000, // in milliseconds
};

It is also possible to add a maxRetransmits option (the number of times to try before failing), but you can only specify maxRetransmits or maxPacketLifeTime, not both. For UDP semantics, set maxRetransmits to 0 and ordered to false. For more information, see these IETF RFCs: Stream Control Transmission Protocol and Stream Control Transmission Protocol Partial Reliability Extension.

ordered: if the data channel should guarantee order or not
maxPacketLifeTime: the maximum time to try and retransmit a failed message
maxRetransmits: the maximum number of times to try and retransmit a failed message
protocol: allows a subprotocol to be used, which provides meta information toward the app
negotiated: if set to true, removes the automatic setting up of a data channel on the other peer, providing your own way to create a data channel with the same ID on the other side
id: allows you to provide your own ID for the channel that can only be used in combination with negotiated set to true)

The only options that most people need to use are the first three: ordered, maxPacketLifeTime, and maxRetransmits. With SCTP (now used by all browsers that support WebRTC) reliable and ordered is true by default. It makes sense to use unreliable and unordered if you want full control from the app layer, but in most cases, partial reliability is helpful.

Note that, as with WebSocket, RTCDataChannel fires events when a connection is established, closed, or errors, and when it receives a message from the other peer.

Is it safe? #

Encryption is mandatory for all WebRTC components. With RTCDataChannel, all data is secured with Datagram Transport Layer Security (DTLS). DTLS is a derivative of SSL, meaning your data will be as secure as using any standard SSL-based connection. DTLS is standardized and built into all browsers that support WebRTC. For more information, see Wireshark wiki.

Change how you think about data #

Handling large amounts of data can be a pain point in JavaScript. As the developers of Sharefest pointed out, this required thinking about data in a new way. If you are transferring a file that is larger than the amount of memory you have available, you have to think about new ways to save this information. This is where technologies, such as the FileSystem API, come into play, as you see next.

Creating a web app that can share files in the browser is now possible with RTCDataChannel. Building on top of RTCDataChannel means that the transferred file data is encrypted and does not touch an app provider's servers. This functionality, combined with the possibility of connecting to multiple clients for faster sharing, makes WebRTC file sharing a strong candidate for the web.

Several steps are required to make a successful transfer:

Read a file in JavaScript using the File API.
Make a peer connection between clients with RTCPeerConnection.
Create a data channel between clients with RTCDataChannel.

There are several points to consider when trying to send files over RTCDataChannel:

File size: if file size is reasonably small and can be stored and loaded as one Blob, you can load into memory using the File API and then send the file over a reliable channel as is (though bear in mind that browsers impose limits on maximum transfer size). As file size gets larger, things get trickier. When a chunking mechanism is required, file chunks are loaded and sent to another peer, accompanied with chunkID metadata so the peer can recognize them. Note that, in this case, you also need to save the chunks first to offline storage (for example, using the FileSystem API) and save it to the user's disk only when you have the file in its entirety.
Chunk size: these are the smallest "atoms" of data for your app. Chunking is required because there is currently a send size limit (though this will be fixed in a future version of data channels). The current recommendation for maximum chunk size is 64KiB.

Once the file is fully transferred to the other side, it can be downloaded using an anchor tag:

function saveFile(blob) {
  const link = document.createElement('a');
  link.href = window.URL.createObjectURL(blob);
  link.download = 'File Name';
  link.click();
};

These file-sharing apps on PubShare and GitHub use this technique. They are both open source and provide a good foundation for a file-sharing app based on RTCDataChannel.

So what can you do? #

RTCDataChannel opens the doors to new ways to build apps for file sharing, multiplayer gaming, and content delivery.

Peer-to-peer file sharing as previously described
Multiplayer gaming, paired with other technologies, such as WebGL, as seen in Mozilla's BananaBread
Content delivery as being reinvented by PeerCDN, a framework that delivers web assets through peer-to-peer data communication

Change the way you build apps #

You can now provide more-engaging apps by using high-performance, low-latency connections through RTCDataChannel. Frameworks, such as PeerJS and the PubNub WebRTC SDK, make RTCDataChannel easier to implement and the API now has wide support across platforms.

The advent of RTCDataChannel can change the way that you think about data transfer in the browser.

Find out more #

What is EME?

2014-01-16T00:00:00Z

Encrypted Media Extensions provides an API that enables web applications to interact with content protection systems, to allow playback of encrypted audio and video.

EME is designed to enable the same app and encrypted files to be used in any browser, regardless of the underlying protection system. The former is made possible by the standardized APIs and flow while the latter is made possible by the concept of Common Encryption.

EME is an extension to the HTMLMediaElement specification — hence the name. Being an 'extension' means that browser support for EME is optional: if a browser does not support encrypted media, it will not be able to play encrypted media, but EME is not required for HTML spec compliance. From the EME spec:

This proposal extends HTMLMediaElement providing APIs to control playback of protected content.

The API supports use cases ranging from simple clear key decryption to high value video (given an appropriate user agent implementation). License/key exchange is controlled by the application, facilitating the development of robust playback applications supporting a range of content decryption and protection technologies.

This specification does not define a content protection or Digital Rights Management system. Rather, it defines a common API that may be used to discover, select and interact with such systems as well as with simpler content encryption systems. Implementation of Digital Rights Management is not required for compliance with this specification: only the Clear Key system is required to be implemented as a common baseline.

The common API supports a simple set of content encryption capabilities, leaving application functions such as authentication and authorization to page authors. This is achieved by requiring content protection system-specific messaging to be mediated by the page rather than assuming out-of-band communication between the encryption system and a license or other server.

EME implementations use the following external components:

Key System: A content protection (DRM) mechanism. EME doesn't define Key Systems themselves, apart from Clear Key (more about that below).
Content Decryption Module (CDM): A client-side software or hardware mechanism that enables playback of encrypted media. As with Key Systems, EME doesn't define any CDMs, but provides an interface for applications to interact with CDMs that are available.
License (Key) server: Interacts with a CDM to provide keys to decrypt media. Negotiation with the license server is the responsibility of the application.
Packaging service: Encodes and encrypts media for distribution/consumption.

Note that an application using EME interacts with a license server to get keys to enable decryption, but user identity and authentication are not part of EME. Retrieval of keys to enable media playback happens after (optionally) authenticating a user. Services such as Netflix must authenticate users within their web application: when a user signs into the application, the application determines the user's identity and privileges.

How does EME work? #

Here's how the components of EME interact, corresponding to the code example below:

If multiple formats or codecs are available, MediaSource.isTypeSupported() or HTMLMediaElement.canPlayType() can both be used to select the right one. However, the CDM may only support a subset of what the browser supports for unencrypted content. It's best to negotiate a MediaKeys configuration before selecting a format and codec. If the application waits for the encrypted event but then MediaKeys shows it can't handle the chosen format/codec, it may be too late to switch without interrupting playback.

The recommended flow is to negotiate MediaKeys first, using MediaKeysSystemAccess.getConfiguration() to find out the negotiated configuration.

If there is only one format/codec to choose from, then there's no need for getConfiguration(). However, it's still preferable to set up MediaKeys first. The only reason to wait for the encrypted event is if there is no way of knowing whether the content is encrypted or not, but in practice that's unlikely.

A web application attempts to play audio or video that has one or more encrypted streams.
The browser recognizes that the media is encrypted (see box below for how that happens) and fires an encrypted event with metadata (initData) obtained from the media about the encryption.
The application handles the encrypted event:
1. If no MediaKeys object has been associated with the media element, first select an available Key System by using navigator.requestMediaKeySystemAccess() to check what Key Systems are available, then create a MediaKeys object for an available Key System via a MediaKeySystemAccess object. Note that initialization of the MediaKeys object should happen before the first encrypted event. Getting a license server URL is done by the app independently of selecting an available key system. A MediaKeys object represents all the keys available to decrypt the media for an audio or video element. It represents a CDM instance and provides access to the CDM, specifically for creating key sessions, which are used to obtain keys from a license server.
2. Once the MediaKeys object has been created, assign it to the media element: setMediaKeys() associates the MediaKeys object with an HTMLMediaElement, so that its keys can be used during playback, i.e. during decoding.
The app creates a MediaKeySession by calling createSession() on the MediaKeys. This creates a MediaKeySession, which represents the lifetime of a license and its key(s).
The app generates a license request by passing the media data obtained in the encrypted handler to the CDM, by calling generateRequest() on the MediaKeySession.
The CDM fires a message event: a request to acquire a key from a license server.
The MediaKeySession object receives the message event and the application sends a message to the license server (via XHR, for example).
The application receives a response from the license server and passes the data to the CDM using the update() method of the MediaKeySession.
The CDM decrypts the media using the keys in the license. A valid key may be used, from any session within the MediaKeys associated with the media element. The CDM will access the key and policy, indexed by Key ID.

Media playback resumes.

How does the browser know that media is encrypted?

This information is in the metadata of the media container file, which will be in a format such as ISO BMFF or WebM. For ISO BMFF this means header metadata, called the protection scheme information box. WebM uses the Matroska ContentEncryption element, with some WebM-specific additions. Guidelines are provided for each container in an EME-specific registry.

Note that there may be multiple messages between the CDM and the license server, and all communication in this process is opaque to the browser and application: messages are only understood by the CDM and license server, although the app layer can see what type of message the CDM is sending. The license request contains proof of the CDM's validity (and trust relationship) as well as a key to use when encrypting the content key(s) in the resulting license.

But what do CDMs actually do? #

An EME implementation does not in itself provide a way to decrypt media: it simply provides an API for a web application to interact with Content Decryption Modules.

What CDMs actually do is not defined by the EME spec, and a CDM may handle decoding (decompression) of media as well as decryption. From least to most robust, there are several potential options for CDM functionality:

Decryption only, enabling playback using the normal media pipeline, for example via a <video> element.
Decryption and decoding, passing video frames to the browser for rendering.
Decryption and decoding, rendering directly in the hardware (for example, the GPU).

There are multiple ways to make a CDM available to a web app:

Bundle a CDM with the browser.
Distribute a CDM separately.
Build a CDM into the operating system.
Include a CDM in firmware.
Embed a CDM in hardware.

How a CDM is made available is not defined by the EME spec, but in all cases the browser is responsible for vetting and exposing the CDM.

EME doesn't mandate a particular Key System; among current desktop and mobile browsers, Chrome supports Widevine and IE11 supports PlayReady.

Getting a key from a license server #

In typical commercial use, content will be encrypted and encoded using a packaging service or tool. Once the encrypted media is made available online, a web client can obtain a key (contained within a license) from a license server and use the key to enable decryption and playback of the content.

The following code (adapted from the spec examples) shows how an application can select an appropriate key system and obtain a key from a license server.

    var video = document.querySelector('video');

    var config = [{initDataTypes: ['webm'],
      videoCapabilities: [{contentType: 'video/webm; codecs="vp09.00.10.08"'}]}];

    if (!video.mediaKeys) {
      navigator.requestMediaKeySystemAccess('org.w3.clearkey',
          config).then(
        function(keySystemAccess) {
          var promise = keySystemAccess.createMediaKeys();
          promise.catch(
            console.error.bind(console, 'Unable to create MediaKeys')
          );
          promise.then(
            function(createdMediaKeys) {
              return video.setMediaKeys(createdMediaKeys);
            }
          ).catch(
            console.error.bind(console, 'Unable to set MediaKeys')
          );
          promise.then(
            function(createdMediaKeys) {
              var initData = new Uint8Array([...]);
              var keySession = createdMediaKeys.createSession();
              keySession.addEventListener('message', handleMessage,
                  false);
              return keySession.generateRequest('webm', initData);
            }
          ).catch(
            console.error.bind(console,
              'Unable to create or initialize key session')
          );
        }
      );
    }

    function handleMessage(event) {
      var keySession = event.target;
      var license = new Uint8Array([...]);
      keySession.update(license).catch(
        console.error.bind(console, 'update() failed')
      );
    }

Common encryption #

Common Encryption solutions allow content providers to encrypt and package their content once per container/codec and use it with a variety of Key Systems, CDMs and clients: that is, any CDM that supports Common Encryption. For example, a video packaged using Playready could be played back in a browser using a Widevine CDM obtaining a key from a Widevine license server.

This is in contrast to legacy solutions that would only work with a complete vertical stack, including a single client that often also included an application runtime.

Common Encryption (CENC) is an ISO standard defining a protection scheme for ISO BMFF; a similar concept applies to WebM.

Clear Key #

Although EME does not define DRM functionality, the spec currently mandates that all browsers supporting EME must implement Clear Key. Using this system, media can be encrypted with a key and then played back simply by providing that key. Clear Key can be built into the browser: it does not require the use of a separate decryption module.

While not likely to be used for many types of commercial content, Clear Key is fully interoperable across all browsers that support EME. It is also handy for testing EME implementations, and applications using EME, without the need to request a content key from a license server. There is a simple Clear Key example at simpl.info/ck. Below is a walkthrough of the code, which parallels the steps described above, though without license server interaction.

// Define a key: hardcoded in this example
// – this corresponds to the key used for encryption
var KEY = new Uint8Array([
  0xeb, 0xdd, 0x62, 0xf1, 0x68, 0x14, 0xd2, 0x7b, 0x68, 0xef, 0x12, 0x2a, 0xfc,
  0xe4, 0xae, 0x3c,
]);

var config = [
  {
    initDataTypes: ['webm'],
    videoCapabilities: [
      {
        contentType: 'video/webm; codecs="vp8"',
      },
    ],
  },
];

var video = document.querySelector('video');
video.addEventListener('encrypted', handleEncrypted, false);

navigator
  .requestMediaKeySystemAccess('org.w3.clearkey', config)
  .then(function (keySystemAccess) {
    return keySystemAccess.createMediaKeys();
  })
  .then(function (createdMediaKeys) {
    return video.setMediaKeys(createdMediaKeys);
  })
  .catch(function (error) {
    console.error('Failed to set up MediaKeys', error);
  });

function handleEncrypted(event) {
  var session = video.mediaKeys.createSession();
  session.addEventListener('message', handleMessage, false);
  session
    .generateRequest(event.initDataType, event.initData)
    .catch(function (error) {
      console.error('Failed to generate a license request', error);
    });
}

function handleMessage(event) {
  // If you had a license server, you would make an asynchronous XMLHttpRequest
  // with event.message as the body.  The response from the server, as a
  // Uint8Array, would then be passed to session.update().
  // Instead, we will generate the license synchronously on the client, using
  // the hard-coded KEY at the top.
  var license = generateLicense(event.message);

  var session = event.target;
  session.update(license).catch(function (error) {
    console.error('Failed to update the session', error);
  });
}

// Convert Uint8Array into base64 using base64url alphabet, without padding.
function toBase64(u8arr) {
  return btoa(String.fromCharCode.apply(null, u8arr))
    .replace(/\+/g, '-')
    .replace(/\//g, '_')
    .replace(/=*$/, '');
}

// This takes the place of a license server.
// kids is an array of base64-encoded key IDs
// keys is an array of base64-encoded keys
function generateLicense(message) {
  // Parse the clearkey license request.
  var request = JSON.parse(new TextDecoder().decode(message));
  // We only know one key, so there should only be one key ID.
  // A real license server could easily serve multiple keys.
  console.assert(request.kids.length === 1);

  var keyObj = {
    kty: 'oct',
    alg: 'A128KW',
    kid: request.kids[0],
    k: toBase64(KEY),
  };
  return new TextEncoder().encode(
    JSON.stringify({
      keys: [keyObj],
    }),
  );
}

To test this code, you need an encrypted video to play. Encrypting a video for use with Clear Key can be done for WebM as per the webm_crypt instructions. Commercial services are also available (for ISO BMFF/MP4 at least) and other solutions are being developed.

The HTMLMediaElement is a creature of simple beauty.

We can load, decode and play media simply by providing a src URL:

<video src="foo.webm"></video>

The Media Source API is an extension to HTMLMediaElement enabling more fine- grained control over the source of media, by allowing JavaScript to build streams for playback from 'chunks' of video. This in turn enables techniques such as adaptive streaming and time shifting.

Why is MSE important to EME? Because in addition to distributing protected content, commercial content providers must be able to adapt content delivery to network conditions and other requirements. Netflix, for example, dynamically changes stream bitrate as network conditions change. EME works with playback of media streams provided by an MSE implementation, just as it would with media provided via a src attribute.

How to chunk and play back media encoded at different bitrates? See the DASH section below.

You can see MSE in action at simpl.info/mse; for the purposes of this example, a WebM video is split into five chunks using the File APIs. In a production application, chunks of video would be retrieved via AJAX.

First a SourceBuffer is created:

var sourceBuffer = mediaSource.addSourceBuffer(
  'video/webm; codecs="vorbis,vp8"',
);

The entire movie is then 'streamed' to a video element by appending each chunk using the appendBuffer() method:

reader.onload = function (e) {
  sourceBuffer.appendBuffer(new Uint8Array(e.target.result));
  if (i === NUM_CHUNKS - 1) {
    mediaSource.endOfStream();
  } else {
    if (video.paused) {
      // start playing after first chunk is appended
      video.play();
    }
    readChunk_(++i);
  }
};

Find out more about MSE in the MSE primer.

Multi-device, multi-platform, mobile — whatever you call it, the web is often experienced under conditions of changeable connectivity. Dynamic, adaptive delivery is crucial for coping with bandwidth constraints and variability in the multi-device world.

DASH (aka MPEG-DASH) is designed to enable the best possible media delivery in a flaky world, for both streaming and download. Several other technologies do something similar — such as Apple's HTTP Live Streaming (HLS) and Microsoft's Smooth Streaming — but DASH is the only method of adaptive bitrate streaming via HTTP that is based on an open standard. DASH is already in use by sites such as YouTube.

What does this have to do with EME and MSE? MSE-based DASH implementations can parse a manifest, download segments of video at an appropriate bitrate, and feed them to a video element when it gets hungry — using existing HTTP infrastructure.

In other words, DASH enables commercial content providers to do adaptive streaming of protected content.

DASH does what it says on the tin:

Dynamic: responds to changing conditions.
Adaptive: adapts to provide an appropriate audio or video bitrate.
Streaming: allows for streaming as well as download.
HTTP: enables content delivery with the advantage of HTTP, without the disadvantages of a traditional streaming server.

The BBC has begun providing test streams using DASH:

The media is encoded a number of times at different bitrates. Each encoding is called a Representation. These are split into a number of Media Segments. The client plays a programme by requesting segments, in order, from a representation over HTTP. Representations can be grouped into Adaptation Sets of representations containing equivalent content. If the client wishes to change bitrate it can pick an alternative from the current adaption set and start requesting segments from that representation. Content is encoded in such a way to make this switching easy for the client to do. In addition to a number of media segments, a representation generally also has an Initialization Segment. This can be thought of as a header, containing information about the encoding, frame sizes, etc. A client needs to obtain this for a given representation before consuming media segments from that representation.

To summarize:

Media is encoded at different bitrates.
The different bitrate files are made available from an HTTP server.
A client web app chooses which bitrate to retrieve and play back with DASH.

As part of the video segmentation process, an XML manifest known as a Media Presentation Description (MPD) is built programmatically. This describes Adaptation Sets and Representations, with durations and URLs. An MPD looks like this:

    <MPD xmlns="urn:mpeg:DASH:schema:MPD:2011" mediaPresentationDuration="PT0H3M1.63S" minBufferTime="PT1.5S" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011"
    type="static">
      <Period duration="PT0H3M1.63S" start="PT0S">
        <AdaptationSet>
          <ContentComponent contentType="video" id="1" />
          <Representation bandwidth="4190760" codecs="avc1.640028" height="1080" id="1" mimeType="video/mp4" width="1920">
            <BaseURL>car-20120827-89.mp4</BaseURL>
            <SegmentBase indexRange="674-1149">
              <Initialization range="0-673" />
            </SegmentBase>
          </Representation>
          <Representation bandwidth="2073921" codecs="avc1.4d401f" height="720" id="2" mimeType="video/mp4" width="1280">
            <BaseURL>car-20120827-88.mp4</BaseURL>
            <SegmentBase indexRange="708-1183">
              <Initialization range="0-707" />
            </SegmentBase>
          </Representation>

          …

        </AdaptationSet>
      </Period>
    </MPD>

(This XML is taken from the .mpd file used for the YouTube DASH demo player.)

According to the DASH spec, an MPD file could in theory be used as the src for a video. However, to give more flexibility to web developers, browser vendors have chosen instead to leave DASH support up to JavaScript libraries using MSE such as dash.js. Implementing DASH in JavaScript allows the adaptation algorithm to evolve without requiring browser updates. Using MSE also allows experimentation with alternative manifest formats and delivery mechanisms without requiring browser changes. Google's Shaka Player implements a DASH client with EME support.

Mozilla Developer Network has instructions on how to use WebM tools and FFmpeg to segment video and build an MPD.

Conclusion #

Use of the web to deliver paid-for video and audio is growing at a huge rate. It seems that every new device, whether it's a tablet, game console, connected TV, or set-top box, is able to stream media from the major content providers over HTTP. Over 85% of mobile and desktop browsers now support <video> and <audio>, and Cisco estimates that video will be 80 to 90 percent of global consumer internet traffic by 2017. In this context, browser support for protected content distribution is likely to be become increasingly significant, as browser vendors curtail support for APIs that most media plugins rely on.

Feedback #

EME WTF?

2014-01-16T00:00:00Z

Encrypted Media Extensions provides an API that enables web applications to interact with content protection systems, to allow playback of encrypted audio and video.

EME is an extension to the HTMLMediaElement specification - hence the name. Being an 'extension' means that browser support for EME is optional: if a browser does not support encrypted media, it will not be able to play encrypted media, but EME is not required for HTML spec compliance. From the EME spec:

This proposal extends HTMLMediaElement providing APIs to control playback of protected content. The API supports use cases ranging from simple clear key decryption to high value video (given an appropriate user agent implementation). License/key exchange is controlled by the application, facilitating the development of robust playback applications supporting a range of content decryption and protection technologies. This specification does not define a content protection or Digital Rights Management system. Rather, it defines a common API that may be used to discover, select and interact with such systems as well as with simpler content encryption systems. Implementation of Digital Rights Management is not required for compliance with this specification: only the Clear Key system is required to be implemented as a common baseline. The common API supports a simple set of content encryption capabilities, leaving application functions such as authentication and authorization to page authors. This is achieved by requiring content protection system-specific messaging to be mediated by the page rather than assuming out-of-band communication between the encryption system and a license or other server.

EME implementations use the following external components:

Key System: A content protection (DRM) mechanism. EME doesn't define Key Systems themselves, apart from Clear Key (more about that below).
Content Decryption Module (CDM): A client-side software or hardware mechanism that enables playback of encrypted media. As with Key Systems, EME doesn't define any CDMs, but provides an interface for applications to interact with CDMs that are available.
License (Key) server: Interacts with a CDM to provide keys to decrypt media. Negotiation with the license server is the responsibility of the application.
Packaging service: Encodes and encrypts media for distribution/consumption.

How does EME work? #

Here's how the components of EME interact, corresponding to the code example below:

If multiple formats or codecs are available, MediaSource.isTypeSupported() or HTMLMediaElement.canPlayType() can both be used to select the right one. However, the CDM may only support a subset of what the browser supports for unencrypted content. It's best to negotiate a MediaKeys configuration before selecting a format and codec. If the application waits for the encrypted event but then MediaKeys shows it can't handle the chosen format/codec, it may be too late to switch without interrupting playback. The recommended flow is to negotiate MediaKeys first, using MediaKeysSystemAccess.getConfiguration() to find out the negotiated configuration. If there is only one format/codec to choose from, then there's no need for getConfiguration(). However, it's still preferable to set up MediaKeys first. The only reason to wait for the encrypted event is if there is no way of knowing whether the content is encrypted or not, but in practice that's unlikely.

A web application attempts to play audio or video that has one or more encrypted streams.
The browser recognizes that the media is encrypted (see box below for how that happens) and fires an encrypted event with metadata (initData) obtained from the media about the encryption.
The application handles the encrypted event:
1. If no MediaKeys object has been associated with the media element, first select an available Key System by using navigator.requestMediaKeySystemAccess() to check what Key Systems are available, then create a MediaKeys object for an available Key System via a MediaKeySystemAccess object. Note that initialization of the MediaKeys object should happen before the first encrypted event. Getting a license server URL is done by the app independently of selecting an available key system. A MediaKeys object represents all the keys available to decrypt the media for an audio or video element. It represents a CDM instance and provides access to the CDM, specifically for creating key sessions, which are used to obtain keys from a license server.
2. Once the MediaKeys object has been created, assign it to the media element: setMediaKeys() associates the MediaKeys object with an HTMLMediaElement, so that its keys can be used during playback, i.e. during decoding.
The app creates a MediaKeySession by calling createSession() on the MediaKeys. This creates a MediaKeySession, which represents the lifetime of a license and its key(s).
The app generates a license request by passing the media data obtained in the encrypted handler to the CDM, by calling generateRequest() on the MediaKeySession.
The CDM fires a message event: a request to acquire a key from a license server.
The MediaKeySession object receives the message event and the application sends a message to the license server (via XHR, for example).
The application receives a response from the license server and passes the data to the CDM using the update() method of the MediaKeySession.
The CDM decrypts the media using the keys in the license. A valid key may be used, from any session within the MediaKeys associated with the media element. The CDM will access the key and policy, indexed by Key ID.
Media playback resumes.

Phew…

…but what do CDMs actually do? #

An EME implementation does not in itself provide a way to decrypt media: it simply provides an API for a web application to interact with Content Decryption Modules.

Decryption only, enabling playback using the normal media pipeline, for example via a <video> element.
Decryption and decoding, passing video frames to the browser for rendering.
Decryption and decoding, rendering directly in the hardware (for example, the GPU).

There are multiple ways to make a CDM available to a web app:

Bundle a CDM with the browser.
Distribute a CDM separately.
Build a CDM into the operating system.
Include a CDM in firmware.
Embed a CDM in hardware.

How a CDM is made available is not defined by the EME spec, but in all cases the browser is responsible for vetting and exposing the CDM.

EME doesn't mandate a particular Key System; among current desktop and mobile browsers, Chrome supports Widevine and IE11 supports PlayReady.

Getting a key from a license server #

The following code (adapted from the spec examples) shows how an application can select an appropriate key system and obtain a key from a license server.

var video = document.querySelector('video');

var config = [{initDataTypes: ['webm'],
  videoCapabilities: [{contentType: 'video/webm; codecs="vp9"'}]}];

if (!video.mediaKeys) {
  navigator.requestMediaKeySystemAccess('org.w3.clearkey',
      config).then(
    function(keySystemAccess) {
      var promise = keySystemAccess.createMediaKeys();
      promise.catch(
        console.error.bind(console, 'Unable to create MediaKeys')
      );
      promise.then(
        function(createdMediaKeys) {
          return video.setMediaKeys(createdMediaKeys);
        }
      ).catch(
        console.error.bind(console, 'Unable to set MediaKeys')
      );
      promise.then(
        function(createdMediaKeys) {
          var initData = new Uint8Array([...]);
          var keySession = createdMediaKeys.createSession();
          keySession.addEventListener('message', handleMessage,
              false);
          return keySession.generateRequest('webm', initData);
        }
      ).catch(
        console.error.bind(console,
          'Unable to create or initialize key session')
      );
    }
  );
}

function handleMessage(event) {
  var keySession = event.target;
  var license = new Uint8Array([...]);
  keySession.update(license).catch(
    console.error.bind(console, 'update() failed')
  );
}

Common Encryption #

This is in contrast to legacy solutions that would only work with a complete vertical stack, including a single client that often also included an application runtime.

Common Encryption (CENC) is an ISO standard defining a protection scheme for ISO BMFF; a similar concept applies to WebM.

Clear Key #

// Define a key: hardcoded in this example
// – this corresponds to the key used for encryption
var KEY = new Uint8Array([
  0xeb, 0xdd, 0x62, 0xf1, 0x68, 0x14, 0xd2, 0x7b,
  0x68, 0xef, 0x12, 0x2a, 0xfc, 0xe4, 0xae, 0x3c
]);

var config = [{
  initDataTypes: ['webm'],
  videoCapabilities: [{
    contentType: 'video/webm; codecs="vp8"'
  }]
}];

var video = document.querySelector('video');
video.addEventListener('encrypted', handleEncrypted, false);

navigator.requestMediaKeySystemAccess('org.w3.clearkey', config).then(
  function(keySystemAccess) {
    return keySystemAccess.createMediaKeys();
  }
).then(
  function(createdMediaKeys) {
    return video.setMediaKeys(createdMediaKeys);
  }
).catch(
  function(error) {
    console.error('Failed to set up MediaKeys', error);
  }
);

function handleEncrypted(event) {
  var session = video.mediaKeys.createSession();
  session.addEventListener('message', handleMessage, false);
  session.generateRequest(event.initDataType, event.initData).catch(
    function(error) {
      console.error('Failed to generate a license request', error);
    }
  );
}

function handleMessage(event) {
  // If you had a license server, you would make an asynchronous XMLHttpRequest
  // with event.message as the body.  The response from the server, as a
  // Uint8Array, would then be passed to session.update().
  // Instead, we will generate the license synchronously on the client, using
  // the hard-coded KEY at the top.
  var license = generateLicense(event.message);

  var session = event.target;
  session.update(license).catch(
    function(error) {
      console.error('Failed to update the session', error);
    }
  );
}

// Convert Uint8Array into base64 using base64url alphabet, without padding.
function toBase64(u8arr) {
  return btoa(String.fromCharCode.apply(null, u8arr)).
      replace(/\+/g, '-').replace(/\//g, '_').replace(/=*$/, '');
}

// This takes the place of a license server.
// kids is an array of base64-encoded key IDs
// keys is an array of base64-encoded keys
function generateLicense(message) {
  // Parse the clearkey license request.
  var request = JSON.parse(new TextDecoder().decode(message));
  // We only know one key, so there should only be one key ID.
  // A real license server could easily serve multiple keys.
  console.assert(request.kids.length === 1);

  var keyObj = {
    kty: 'oct',
    alg: 'A128KW',
    kid: request.kids[0],
    k: toBase64(KEY)
  };
  return new TextEncoder().encode(JSON.stringify({
    keys: [keyObj]
  }));
}

Media Source Extensions (MSE) #

The HTMLMediaElement is a creature of simple beauty.

We can load, decode and play media simply by providing a src URL:

<video src='foo.webm'></video>

The Media Source API is an extension to HTMLMediaElement enabling more fine-grained control over the source of media, by allowing JavaScript to build streams for playback from 'chunks' of video. This in turn enables techniques such as adaptive streaming and time shifting.

How to chunk and play back media encoded at different bitrates? See the DASH section below.

First a SourceBuffer is created:

var sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="vorbis,vp8"');

The entire movie is then 'streamed' to a video element by appending each chunk using the appendBuffer() method:

reader.onload = function (e) {
  sourceBuffer.appendBuffer(new Uint8Array(e.target.result));
  if (i === NUM_CHUNKS - 1) {
    mediaSource.endOfStream();
  } else {
    if (video.paused) {
      // start playing after first chunk is appended
      video.play();
    }
    readChunk_(++i);
  }
};

Find out more about MSE in the HTML5 Rocks article.

Dynamic Adaptive Streaming over HTTP (DASH) #

Multi-device, multi-platform, mobile - whatever you call it, the web is often experienced under conditions of changeable connectivity. Dynamic, adaptive delivery is crucial for coping with bandwidth constraints and variability in the multi-device world.

DASH (aka MPEG-DASH) is designed to enable the best possible media delivery in a flaky world, for both streaming and download. Several other technologies do something similar - such as Apple's HTTP Live Streaming (HLS) and Microsoft's Smooth Streaming - but DASH is the only method of adaptive bitrate streaming via HTTP that is based on an open standard. DASH is already in use by sites such as YouTube.

In other words, DASH enables commercial content providers to do adaptive streaming of protected content.

DASH does what it says on the tin:

Dynamic: responds to changing conditions.
Adaptive: adapts to provide an appropriate audio or video bitrate.
Streaming: allows for streaming as well as download.
HTTP: enables content delivery with the advantage of HTTP, without the disadvantages of a traditional streaming server.

The BBC has begun providing test streams using DASH:

The media is encoded a number of times at different bitrates. Each encoding is called a Representation. These are split into a number of Media Segments. The client plays a programme by requesting segments, in order, from a representation over HTTP. Representations can be grouped into Adaptation Sets of representations containing equivalent content. If the client wishes to change bitrate it can pick an alternative from the current adaption set and start requesting segments from that representation. Content is encoded in such a way to make this switching easy for the client to do. In addition to a number of media segments, a representation generally also has an Initialisation Segment. This can be thought of as a header, containing information about the encoding, frame sizes, etc. A client needs to obtain this for a given representation before consuming media segments from that representation.

To summarize:

Media is encoded at different bitrates.
The different bitrate files are made available from an HTTP server.
A client web app chooses which bitrate to retrieve and play back with DASH.

<MPD xmlns="urn:mpeg:DASH:schema:MPD:2011" mediaPresentationDuration="PT0H3M1.63S" minBufferTime="PT1.5S" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011"
type="static">
  <Period duration="PT0H3M1.63S" start="PT0S">
    <AdaptationSet>
      <ContentComponent contentType="video" id="1" />
      <Representation bandwidth="4190760" codecs="avc1.640028" height="1080" id="1" mimeType="video/mp4" width="1920">
        <BaseURL>car-20120827-89.mp4</BaseURL>
        <SegmentBase indexRange="674-1149">
          <Initialization range="0-673" />
        </SegmentBase>
      </Representation>
      <Representation bandwidth="2073921" codecs="avc1.4d401f" height="720" id="2" mimeType="video/mp4" width="1280">
        <BaseURL>car-20120827-88.mp4</BaseURL>
        <SegmentBase indexRange="708-1183">
          <Initialization range="0-707" />
        </SegmentBase>
      </Representation>

      …

    </AdaptationSet>
  </Period>
</MPD>

(This XML is taken from the .mpd file used for the YouTube DASH demo player)

According to the DASH spec, an MPD file could in theory be used as the src for a video. However, to give more flexibility to web developers, browser vendors have chosen instead to leave DASH support up to JavaScript libraries using MSE such as dash.js. Implementing DASH in JavaScript allows the adaptation algorithm to evolve without requiring browser updates. Using MSE also allows experimentation with alternative manifest formats and delivery mechanisms without requiring browser changes. Google's Shaka Player implements a DASH client with EME support.

Mozilla Developer Network has instructions on how to use WebM tools and FFmpeg to segment video and build an MPD.

Conclusion #

Use of the web to deliver paid-for video and audio is growing at a huge rate. It seems that every new device, whether it's a tablet, game console, connected TV, or set-top box, is able to stream media from the major content providers over HTTP. Over 85% of mobile and desktop browsers now support <video> and <audio>, and Cisco estimates that video will be 80 to 90 percent of global consumer internet traffic by 2017. In this context, browser support for protected content distribution is likely to be become increasingly significant, as browser vendors curtail support for APIs that most media plugins rely on.

Build the backend services needed for a WebRTC app

2013-11-04T00:00:00Z

What is signaling? #

Signaling is the process of coordinating communication. In order for a WebRTC app to set up a call, its clients need to exchange the following information:

Session-control messages used to open or close communication
Error messages
Media metadata, such as codecs, codec settings, bandwidth, and media types
Key data used to establish secure connections
Network data, such as a host's IP address and port as seen by the outside world

This signaling process needs a way for clients to pass messages back and forth. That mechanism is not implemented by the WebRTC APIs. You need to build it yourself. Later in this article, you learn ways to build a signaling service. First, however, you need a little context.

Why is signaling not defined by WebRTC? #

To avoid redundancy and to maximize compatibility with established technologies, signaling methods and protocols are not specified by WebRTC standards. This approach is outlined by the JavaScript Session Establishment Protocol (JSEP):

JSEP's architecture also avoids a browser having to save state, that is, to function as a signaling state machine. This would be problematic if, for example, signaling data was lost each time a page was reloaded. Instead, signaling state can be saved on a server.

JSEP architecture

JSEP requires the exchange between peers of offer and answer, the media metadata mentioned above. Offers and answers are communicated in Session Description Protocol (SDP) format, which look like this:

v=0
o=- 7614219274584779017 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE audio video
a=msid-semantic: WMS
m=audio 1 RTP/SAVPF 111 103 104 0 8 107 106 105 13 126
c=IN IP4 0.0.0.0
a=rtcp:1 IN IP4 0.0.0.0
a=ice-ufrag:W2TGCZw2NZHuwlnf
a=ice-pwd:xdQEccP40E+P0L5qTyzDgfmW
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=mid:audio
a=rtcp-mux
a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:9c1AHz27dZ9xPI91YNfSlI67/EMkjHHIHORiClQe
a=rtpmap:111 opus/48000/2
…

Want to know what all this SDP gobbledygook actually means? Take a look at the Internet Engineering Task Force (IETF) examples.

Bear in mind that WebRTC is designed so that the offer or answer can be tweaked before being set as the local or remote description by editing the values in the SDP text. For example, the preferAudioCodec() function in appr.tc can be used to set the default codec and bitrate. SDP is somewhat painful to manipulate with JavaScript and there is discussion about whether future versions of WebRTC should use JSON instead, but there are some advantages to sticking with SDP.

`RTCPeerConnection` API and signaling: Offer, answer, and candidate #

RTCPeerConnection is the API used by WebRTC apps to create a connection between peers, and communicate audio and video.

To initialize this process, RTCPeerConnection has two tasks:

Ascertain local media conditions, such as resolution and codec capabilities. This is the metadata used for the offer-and-answer mechanism.
Get potential network addresses for the app's host, known as candidates.

Once this local data has been ascertained, it must be exchanged through a signaling mechanism with the remote peer.

Imagine Alice is trying to call Eve. Here's the full offer/answer mechanism in all its gory detail:

Alice creates an RTCPeerConnection object.
Alice creates an offer (an SDP session description) with the RTCPeerConnection createOffer() method.
Alice calls setLocalDescription() with her offer.
Alice stringifies the offer and uses a signaling mechanism to send it to Eve.
Eve calls setRemoteDescription() with Alice's offer, so that her RTCPeerConnection knows about Alice's setup.
Eve calls createAnswer() and the success callback for this is passed a local session description - Eve's answer.
Eve sets her answer as the local description by calling setLocalDescription().
Eve then uses the signaling mechanism to send her stringified answer to Alice.
Alice sets Eve's answer as the remote session description using setRemoteDescription().

Alice and Eve also need to exchange network information. The expression "finding candidates" refers to the process of finding network interfaces and ports using the ICE framework.

Alice creates an RTCPeerConnection object with an onicecandidate handler.
The handler is called when network candidates become available.
In the handler, Alice sends stringified candidate data to Eve through their signaling channel.
When Eve gets a candidate message from Alice, she calls addIceCandidate() to add the candidate to the remote peer description.

JSEP supports ICE Candidate Trickling, which allows the caller to incrementally provide candidates to the callee after the initial offer, and for the callee to begin acting on the call and set up a connection without waiting for all candidates to arrive.

Code WebRTC for signaling #

The following code snippet is a W3C code example that summarizes the complete signaling process. The code assumes the existence of some signaling mechanism, SignalingChannel. Signaling is discussed in greater detail later.

// handles JSON.stringify/parse
const signaling = new SignalingChannel();
const constraints = {audio: true, video: true};
const configuration = {iceServers: [{urls: 'stun:stun.example.org'}]};
const pc = new RTCPeerConnection(configuration);

// Send any ice candidates to the other peer.
pc.onicecandidate = ({candidate}) => signaling.send({candidate});

// Let the "negotiationneeded" event trigger offer generation.
pc.onnegotiationneeded = async () => {
  try {
    await pc.setLocalDescription(await pc.createOffer());
    // send the offer to the other peer
    signaling.send({desc: pc.localDescription});
  } catch (err) {
    console.error(err);
  }
};

// After remote track media arrives, show it in remote video element.
pc.ontrack = (event) => {
  // Don't set srcObject again if it is already set.
  if (remoteView.srcObject) return;
  remoteView.srcObject = event.streams[0];
};

// Call start() to initiate.
async function start() {
  try {
    // Get local stream, show it in self-view, and add it to be sent.
    const stream =
      await navigator.mediaDevices.getUserMedia(constraints);
    stream.getTracks().forEach((track) =>
      pc.addTrack(track, stream));
    selfView.srcObject = stream;
  } catch (err) {
    console.error(err);
  }
}

signaling.onmessage = async ({desc, candidate}) => {
  try {
    if (desc) {
      // If you get an offer, you need to reply with an answer.
      if (desc.type === 'offer') {
        await pc.setRemoteDescription(desc);
        const stream =
          await navigator.mediaDevices.getUserMedia(constraints);
        stream.getTracks().forEach((track) =>
          pc.addTrack(track, stream));
        await pc.setLocalDescription(await pc.createAnswer());
        signaling.send({desc: pc.localDescription});
      } else if (desc.type === 'answer') {
        await pc.setRemoteDescription(desc);
      } else {
        console.log('Unsupported SDP type.');
      }
    } else if (candidate) {
      await pc.addIceCandidate(candidate);
    }
  } catch (err) {
    console.error(err);
  }
};

To see the offer/answer and candidate-exchange processes in action, see simpl.info RTCPeerConnection and look at the console log for a single-page video chat example. If you want more, download a complete dump of WebRTC signaling and stats from the about://webrtc-internals page in Google Chrome or the opera://webrtc-internals page in Opera.

Peer discovery #

This is a fancy way of asking, "How do I find someone to talk to?"

For telephone calls, you have telephone numbers and directories. For online video chat and messaging, you need identity and presence management systems, and a means for users to initiate sessions. WebRTC apps need a way for clients to signal to each other that they want to start or join a call.

Peer discovery mechanisms are not defined by WebRTC and you don't go into the options here. The process can be as simple as emailing or messaging a URL. For video chat apps, such as Talky, tawk.to and Browser Meeting, you invite people to a call by sharing a custom link. Developer Chris Ball built an intriguing serverless-webrtc experiment that enables WebRTC call participants to exchange metadata by any messaging service they like, such as IM, email, or homing pigeon.

How can you build a signaling service? #

To reiterate, signaling protocols and mechanisms are not defined by WebRTC standards. Whatever you choose, you need an intermediary server to exchange signaling messages and app data between clients. Sadly, a web app cannot simply shout into the internet, "Connect me to my friend!"

Thankfully signaling messages are small and mostly exchanged at the start of a call. In testing with appr.tc for a video chat session, a total of around 30-45 messages were handled by the signaling service with a total size for all messages of around 10KB.

As well as being relatively undemanding in terms of bandwidth, WebRTC signaling services don't consume much processing or memory because they only need to relay messages and retain a small amount of session state data, such as which clients are connected.

Push messages from the server to the client #

A message service for signaling needs to be bidirectional: client to server and server to client. Bidirectional communication goes against the HTTP client/server request/response model, but various hacks such as long polling have been developed over many years in order to push data from a service running on a web server to a web app running in a browser.

More recently, the EventSource API has been widely implemented. This enables server-sent events - data sent from a web server to a browser client through HTTP. EventSource is designed for one-way messaging, but it can be used in combination with XHR to build a service for exchanging signaling messages. A signaling service passes a message from a caller, delivered by XHR request, by pushing it through EventSource to the callee.

WebSocket is a more-natural solution, designed for full duplex client–server communication - messages that can flow in both directions at the same time. One advantage of a signaling service built with pure WebSocket or server-sent events (EventSource) is that the backend for these APIs can be implemented on a variety of web frameworks common to most web-hosting packages for languages such as PHP, Python, and Ruby.

All modern browsers except Opera Mini support WebSocket and, more importantly, all browsers that support WebRTC also support WebSocket, both on desktop and mobile. TLS should be used for all connections to ensure messages cannot be intercepted unencrypted and also to reduce problems with proxy traversal. (For more information about WebSocket and proxy traversal see the WebRTC chapter in Ilya Grigorik's High Performance Browser Networking.)

It is also possible to handle signaling by getting WebRTC clients to poll a messaging server repeatedly through Ajax, but that leads to a lot of redundant network requests, which is especially problematic for mobile devices. Even after a session has been established, peers need to poll for signaling messages in case of changes or session termination by other peers. The WebRTC Book app example takes this option with some optimizations for polling frequency.

Scale signaling #

Although a signaling service consumes relatively little bandwidth and CPU per client, signaling servers for a popular app may have to handle a lot of messages from different locations with high levels of concurrency. WebRTC apps that get a lot of traffic need signaling servers able to handle considerable load. You don't go into detail here, but there are a number of options for high-volume, high-performance messaging, including the following:

eXtensible Messaging and Presence Protocol (XMPP), originally known as Jabber-a protocol developed for instant messaging that can be used for signaling (Server implementations include ejabberd and Openfire. JavaScript clients, such as Strophe.js, use BOSH to emulate bidirectional streaming, but for various reasons, BOSH may not be as efficient as WebSocket and, for the same reasons, may not scale well.) (On a tangent, Jingle is an XMPP extension to enable voice and video. The WebRTC project uses network and transport components from the libjingle library - a C++ implementation of Jingle.)
Open source libraries, such as ZeroMQ (as used by TokBox for their Rumour service) and OpenMQ (NullMQ applies ZeroMQ concepts to web platforms using the STOMP protocol over WebSocket.)
Commercial cloud-messaging platforms that use WebSocket (though they may fall back to long polling), such as Pusher, Kaazing, and PubNub (PubNub also has an API for WebRTC.)
Commercial WebRTC platforms, such as vLine

(Developer Phil Leggetter's Real-Time Web Technologies Guide provides a comprehensive list of messaging services and libraries.)

Build a signaling service with Socket.io on Node #

The following is code for a simple web app that uses a signaling service built with Socket.io on Node. The design of Socket.io makes it simple to build a service to exchange messages and Socket.io is particularly suited to WebRTC signaling because of its built-in concept of rooms. This example is not designed to scale as a production-grade signaling service, but is simple to understand for a relatively small number of users.

Socket.io uses WebSocket with fallbacks: AJAX long polling, AJAX multipart streaming, Forever Iframe, and JSONP polling. It has been ported to various backends, but is perhaps best known for its Node version used in this example.

There's no WebRTC in this example. It's designed only to show how to build signaling into a web app. View the console log to see what's happening as clients join a room and exchange messages. This WebRTC codelab gives step-by-step instructions for how to integrate this into a complete WebRTC video chat app.

Here is the client index.html:

<!DOCTYPE html>
<html>
  <head>
    <title>WebRTC client</title>
  </head>
  <body>
    <script src='/socket.io/socket.io.js'></script>
    <script src='js/main.js'></script>
  </body>
</html>

Here's the JavaScript file main.js referenced in the client:

const isInitiator;

room = prompt('Enter room name:');

const socket = io.connect();

if (room !== '') {
  console.log('Joining room ' + room);
  socket.emit('create or join', room);
}

socket.on('full', (room) => {
  console.log('Room ' + room + ' is full');
});

socket.on('empty', (room) => {
  isInitiator = true;
  console.log('Room ' + room + ' is empty');
});

socket.on('join', (room) => {
  console.log('Making request to join room ' + room);
  console.log('You are the initiator!');
});

socket.on('log', (array) => {
  console.log.apply(console, array);
});

Here's the complete server app:

const static = require('node-static');
const http = require('http');
const file = new(static.Server)();
const app = http.createServer(function (req, res) {
  file.serve(req, res);
}).listen(2013);

const io = require('socket.io').listen(app);

io.sockets.on('connection', (socket) => {

  // Convenience function to log server messages to the client
  function log(){
    const array = ['>>> Message from server: '];
    for (const i = 0; i < arguments.length; i++) {
      array.push(arguments[i]);
    }
      socket.emit('log', array);
  }

  socket.on('message', (message) => {
    log('Got message:', message);
    // For a real app, would be room only (not broadcast)
    socket.broadcast.emit('message', message);
  });

  socket.on('create or join', (room) => {
    const numClients = io.sockets.clients(room).length;

    log('Room ' + room + ' has ' + numClients + ' client(s)');
    log('Request to create or join room ' + room);

    if (numClients === 0){
      socket.join(room);
      socket.emit('created', room);
    } else if (numClients === 1) {
      io.sockets.in(room).emit('join', room);
      socket.join(room);
      socket.emit('joined', room);
    } else { // max two clients
      socket.emit('full', room);
    }
    socket.emit('emit(): client ' + socket.id +
      ' joined room ' + room);
    socket.broadcast.emit('broadcast(): client ' + socket.id +
      ' joined room ' + room);

  });

});

(You don't need to learn about node-static for this. It just happens to be used in this example.)

To run this app on localhost, you need to have Node, Socket.IO, and node-static installed. Node can be downloaded from Node.js (installation is straightforward and quick). To install Socket.IO and node-static, run Node Package Manager from a terminal in your app directory:

npm install socket.io
npm install node-static

To start the server, run the following command from a terminal in your app directory:

node server.js

From your browser, open localhost:2013. Open a new tab or window in any browser and open localhost:2013 again. To see what's happening, check the console. In Chrome and Opera, you can access the console through Google Chrome Developer Tools with Ctrl+Shift+J (or Command+Option+J on Mac).

Whatever approach you choose for signaling, your backend and client app - at the very least - need to provide services similar to this example.

Signaling gotchas #

RTCPeerConnection won't start gathering candidates until setLocalDescription() is called. This is mandated in the JSEP IETF draft.
Take advantage of Trickle ICE. Call addIceCandidate() as soon as candidates arrive.

Readymade signaling servers #

If you don't want to roll your own, there are several WebRTC signaling servers available, which use Socket.IO like the previous example and are integrated with WebRTC client JavaScript libraries:

webRTC.io is one of the first abstraction libraries for WebRTC.
Signalmaster is a signaling server created for use with the SimpleWebRTC JavaScript client library.

If you don't want to write any code at all, complete commercial WebRTC platforms are available from companies, such as vLine, OpenTok, and Asterisk.

For the record, Ericsson built a signaling server using PHP on Apache in the early days of WebRTC. This is now somewhat obsolete, but it's worth looking at the code if you're considering something similar.

Signaling security #

"Security is the art of making nothing happen."

Salman Rushdie

Encryption is mandatory for all WebRTC components.

However, signaling mechanisms aren't defined by WebRTC standards, so it's up to you to make signaling secure. If an attacker manages to hijack signaling, they can stop sessions, redirect connections, and record, alter, or inject content.

The most important factor in securing signaling is to use secure protocols - HTTPS and WSS (for example, TLS) - which ensure that messages cannot be intercepted unencrypted. Also, be careful not to broadcast signaling messages in a way that they can be accessed by other callers using the same signaling server.

After signaling: Use ICE to cope with NATs and firewalls #

For metadata signaling, WebRTC apps use an intermediary server, but for actual media and data streaming once a session is established, RTCPeerConnection attempts to connect clients directly or peer-to-peer.

In a simpler world, every WebRTC endpoint would have a unique address that it could exchange with other peers in order to communicate directly.

A world without NATs and firewalls

In reality, most devices live behind one or more layers of NAT, some have antivirus software that blocks certain ports and protocols, and many are behind proxies and corporate firewalls. A firewall and NAT may in fact be implemented by the same device, such as a home WIFI router.

The real world

WebRTC apps can use the ICE framework to overcome the complexities of real-world networking. To enable this to happen, your app must pass ICE server URLs to RTCPeerConnection, as described in this article.

ICE tries to find the best path to connect peers. It tries all possibilities in parallel and chooses the most efficient option that works. ICE first tries to make a connection using the host address obtained from a device's operating system and network card. If that fails (which it will for devices behind NATs), ICE obtains an external address using a STUN server and, if that fails, traffic is routed through a TURN relay server.

In other words, a STUN server is used to get an external network address and TURN servers are used to relay traffic if direct (peer-to-peer) connection fails.

Every TURN server supports STUN. A TURN server is a STUN server with additional built-in relaying functionality. ICE also copes with the complexities of NAT setups. In reality, NAT hole-punching may require more than just a public IP:port address.

URLs for STUN and/or TURN servers are (optionally) specified by a WebRTC app in the iceServers configuration object that is the first argument to the RTCPeerConnection constructor. For appr.tc, that value looks like this:

{
  'iceServers': [
    {
      'urls': 'stun:stun.l.google.com:19302'
    },
    {
      'urls': 'turn:192.158.29.39:3478?transport=udp',
      'credential': 'JZEOEt2V3Qb0y27GRntt2u2PAYA=',
      'username': '28224511:1379330808'
    },
    {
      'urls': 'turn:192.158.29.39:3478?transport=tcp',
      'credential': 'JZEOEt2V3Qb0y27GRntt2u2PAYA=',
      'username': '28224511:1379330808'
    }
  ]
}

Once RTCPeerConnection has that information, the ICE magic happens automatically. RTCPeerConnection uses the ICE framework to work out the best path between peers, working with STUN and TURN servers as necessary.

STUN #

NATs provide a device with an IP address for use within a private local network, but this address can't be used externally. Without a public address, there's no way for WebRTC peers to communicate. To get around this problem, WebRTC uses STUN.

STUN servers live on the public internet and have one simple task - check the IP:port address of an incoming request (from an app running behind a NAT) and send that address back as a response. In other words, the app uses a STUN server to discover its IP:port from a public perspective. This process enables a WebRTC peer to get a publicly accessible address for itself and then pass it to another peer through a signaling mechanism in order to set up a direct link. (In practice, different NATs work in different ways and there may be multiple NAT layers, but the principle is still the same.)

STUN servers don't have to do much or remember much, so relatively low-spec STUN servers can handle a large number of requests.

Most WebRTC calls successfully make a connection using STUN - 86% according to Webrtcstats.com, though this can be less for calls between peers behind firewalls and complex NAT configurations.

Using STUN servers to get public IP:port addresses

TURN #

RTCPeerConnection tries to set up direct communication between peers over UDP. If that fails, RTCPeerConnection resorts to TCP. If that fails, TURN servers can be used as a fallback, relaying data between endpoints.

Just to reiterate, TURN is used to relay audio, video, and data streaming between peers, not signaling data!

TURN servers have public addresses, so they can be contacted by peers even if the peers are behind firewalls or proxies. TURN servers have a conceptually simple task - to relay a stream. However, unlike STUN servers, they inherently consume a lot of bandwidth. In other words, TURN servers need to be beefier.

The full Monty: STUN, TURN, and signaling

This diagram shows TURN in action. Pure STUN didn't succeed, so each peer resorts to using a TURN server.

Deploying STUN and TURN servers #

For testing, Google runs a public STUN server, stun.l.google.com:19302, as used by appr.tc. For a production STUN/TURN service, use the rfc5766-turn-server. Source code for STUN and TURN servers is available on GitHub, where you can also find links to several sources of information about server installation. A VM image for Amazon Web Services is also available.

An alternative TURN server is restund, available as source code and also for AWS. Here are instructions for how to set up restund on Compute Engine.

Open firewall as necessary for tcp=443, udp/tcp=3478.
Create four instances, one for each public IP, Standard Ubuntu 12.06 image.
Set up local firewall config (allow ANY from ANY).

Install tools:

sudo apt-get install make
sudo apt-get install gcc

Install libre from creytiv.com/re.html.
Fetch restund from creytiv.com/restund.html and unpack./
wget hancke.name/restund-auth.patch and apply with patch -p1 < restund-auth.patch.
Run make, sudo make install for libre and restund.
Adapt restund.conf to your needs (replace IP addresses and make sure it contains the same shared secret) and copy to /etc.
Copy restund/etc/restund to /etc/init.d/.
Configure restund:
1. Set LD_LIBRARY_PATH.
2. Copy restund.conf to /etc/restund.conf.
3. Set restund.conf to use the right 10. IP address.
Run restund
Test using stund client from remote machine: ./client IP:port

Beyond one-to-one: Multiparty WebRTC #

You may also want to take a look at Justin Uberti's proposed IETF standard for a REST API for access to TURN Services.

It's easy to imagine use cases for media streaming that go beyond a simple one-to-one call. For example, video conferencing between a group of colleagues or a public event with one speaker and hundreds or millions of viewers.

A WebRTC app can use multiple RTCPeerConnections so that every endpoint connects to every other endpoint in a mesh configuration. This is the approach taken by apps, such as talky.io, and works remarkably well for a small handful of peers. Beyond that, processing and bandwidth consumption becomes excessive, especially for mobile clients.

Full mesh topology: Everyone connected to everyone

Alternatively, a WebRTC app could choose one endpoint to distribute streams to all others in a star configuration. It would also be possible to run a WebRTC endpoint on a server and construct your own redistribution mechanism (a sample client app is provided by webrtc.org).

Since Chrome 31 and Opera 18, a MediaStream from one RTCPeerConnection can be used as the input for another. This can enable more flexible architectures because it enables a web app to handle call-routing by choosing which other peer to connect to. To see this in action, see WebRTC samples Peer connection relay and WebRTC samples Multiple peer connections.

Multipoint Control Unit #

A better option for a large number of endpoints is to use a Multipoint Control Unit (MCU). This is a server that works as a bridge to distribute media between a large number of participants. MCUs can cope with different resolutions, codecs, and frame rates in a video conference; handle transcoding; do selective stream forwarding; and mix or record audio and video. For multiparty calls, there are a number of issues to consider, particularly how to display multiple video inputs and mix audio from multiple sources. Cloud platforms, such as vLine, also attempt to optimize traffic routing.

It's possible to buy a complete MCU hardware package or build your own.

The back of a Cisco MCU

Several open source MCU software options are available. For example, Licode (previously known as Lynckia) produces an open source MCU for WebRTC. OpenTok has Mantis.

Beyond browsers: VoIP, telephones, and messaging #

The standardized nature of WebRTC makes it possible to establish communication between a WebRTC app running in a browser and a device or platform running on another communication platform, such as a telephone or a video-conferencing system.

SIP is a signaling protocol used by VoIP and video-conferencing systems. To enable communication between a WebRTC web app and a SIP client, such as a video-conferencing system, WebRTC needs a proxy server to mediate signaling. Signaling must flow through the gateway but, once communication has been established, SRTP traffic (video and audio) can flow directly peer to peer.

The Public Switched Telephone Network (PSTN) is the circuit-switched network of all "plain old" analog telephones. For calls between WebRTC web apps and telephones, traffic must go through a PSTN gateway. Likewise, WebRTC web apps need an intermediary XMPP server to communicate with Jingle endpoints such as IM clients. Jingle was developed by Google as an extension to XMPP to enable voice and video for messaging services. Current WebRTC implementations are based on the C++ libjingle library, an implementation of Jingle initially developed for Talk.

A number of apps, libraries, and platforms make use of WebRTC's ability to communicate with the outside world:

sipML5: an open source JavaScript SIP client
jsSIP: JavaScript SIP library
Phono: open source JavaScript phone API built as a plugin
Zingaya: an embeddable phone widget
Twilio: voice and messaging
Uberconference: conferencing

The sipML5 developers have also built the webrtc2sip gateway. Tethr and Tropo have demonstrated a framework for disaster communications "in a briefcase" using an OpenBTS cell to enable communications between feature phones and computers through WebRTC. That's telephone communication without a carrier!

Find out more #

The WebRTC codelab provides step-by-step instructions for how to build a video and text chat app using a Socket.io signaling service running on Node.

Google I/O WebRTC presentation from 2013 with WebRTC tech lead, Justin Uberti

Chris Wilson's SFHTML5 presentation - Introduction to WebRTC Apps

The 350-page book WebRTC: APIs and RTCWEB Protocols of the HTML5 Real-Time Web provides a lot of detail about data and signaling pathways, and includes a number of detailed network topology diagrams.

WebRTC and Signaling: What Two Years Has Taught Us - TokBox blog post about why leaving signaling out of the spec was a good idea

Ben Strong's A Practical Guide to Building WebRTC Apps provides a lot of information about WebRTC topologies and infrastructure.

The WebRTC chapter in Ilya Grigorik's High Performance Browser Networking goes deep into WebRTC architecture, use cases, and performance.

Get started with WebRTC

2012-07-23T00:00:00Z

WebRTC is a new front in the long war for an open and unencumbered web.

Brendan Eich, inventor of JavaScript

Real-time communication without plugins #

Imagine a world where your phone, TV, and computer could communicate on a common platform. Imagine it was easy to add video chat and peer-to-peer data sharing to your web app. That's the vision of WebRTC.

Want to try it out? WebRTC is available on desktop and mobile in Google Chrome, Safari, Firefox, and Opera. A good place to start is the simple video chat app at appr.tc:

Open appr.tc in your browser.
Click Join to join a chat room and let the app use your webcam.
Open the URL displayed at the end of the page in a new tab or, better still, on a different computer.

Quick start #

Haven't got time to read this article or only want code?

To get an overview of WebRTC, watch the following Google I/O video or view these slides:
If you haven't used the getUserMedia API, see Capture audio and video in HTML5 and simpl.info getUserMedia.
To learn about the RTCPeerConnection API, see the following example and 'simpl.info RTCPeerConnection'.
To learn how WebRTC uses servers for signaling, and firewall and NAT traversal, see the code and console logs from appr.tc.
Can’t wait and just want to try WebRTC right now? Try some of the more-than 20 demos that exercise the WebRTC JavaScript APIs.
Having trouble with your machine and WebRTC? Visit the WebRTC Troubleshooter.

Alternatively, jump straight into the WebRTC codelab, a step-by-step guide that explains how to build a complete video chat app, including a simple signaling server.

A very short history of WebRTC #

One of the last major challenges for the web is to enable human communication through voice and video: real-time communication or RTC for short. RTC should be as natural in a web app as entering text in a text input. Without it, you're limited in your ability to innovate and develop new ways for people to interact.

Historically, RTC has been corporate and complex, requiring expensive audio and video technologies to be licensed or developed in house. Integrating RTC technology with existing content, data, and services has been difficult and time-consuming, particularly on the web.

Gmail video chat became popular in 2008 and, in 2011, Google introduced Hangouts, which uses Talk (as did Gmail). Google bought GIPS, a company that developed many components required for RTC, such as codecs and echo cancellation techniques. Google open sourced the technologies developed by GIPS and engaged with relevant standards bodies at the Internet Engineering Task Force (IETF) and World Wide Web Consortium (W3C) to ensure industry consensus. In May 2011, Ericsson built the first implementation of WebRTC.

WebRTC implemented open standards for real-time, plugin-free video, audio, and data communication. The need was real:

Many web services used RTC, but needed downloads, native apps, or plugins. These included Skype, Facebook, and Hangouts.
Downloading, installing, and updating plugins is complex, error prone, and annoying.
Plugins are difficult to deploy, debug, troubleshoot, test, and maintain - and may require licensing and integration with complex, expensive technology. It's often difficult to persuade people to install plugins in the first place!

The guiding principles of the WebRTC project are that its APIs should be open source, free, standardized, built into web browsers, and more efficient than existing technologies.

Where are we now? #

WebRTC is used in various apps, such as Google Meet. WebRTC has also been integrated with WebKitGTK+ and Qt native apps.

WebRTC implements these three APIs:

MediaStream (also known as getUserMedia)
RTCPeerConnection
RTCDataChannel

The APIs are defined in these two specs:

All three APIs are supported on mobile and desktop by Chrome, Safari, Firefox, Edge, and Opera.

getUserMedia: For demos and code, see WebRTC samples or try Chris Wilson's amazing examples that use getUserMedia as input for web audio.

RTCPeerConnection: For a simple demo and a fully functional video-chat app, see WebRTC samples Peer connection and appr.tc, respectively. This app uses adapter.js, a JavaScript shim maintained by Google with help from the WebRTC community, to abstract away browser differences and spec changes.

RTCDataChannel: To see this in action, see WebRTC samples to check out one of the data-channel demos.

The WebRTC codelab shows how to use all three APIs to build a simple app for video chat and file sharing.

Your first WebRTC #

WebRTC apps need to do several things:

Get streaming audio, video, or other data.
Get network information, such as IP addresses and ports, and exchange it with other WebRTC clients (known as peers) to enable connection, even through NATs and firewalls.
Coordinate signaling communication to report errors and initiate or close sessions.
Exchange information about media and client capability, such as resolution and codecs.
Communicate streaming audio, video, or data.

To acquire and communicate streaming data, WebRTC implements the following APIs:

MediaStream gets access to data streams, such as from the user's camera and microphone.
RTCPeerConnection enables audio or video calling with facilities for encryption and bandwidth management.
RTCDataChannel enables peer-to-peer communication of generic data.

(There is detailed discussion of the network and signaling aspects of WebRTC later.)

`MediaStream` API (also known as `getUserMedia` API) #

The MediaStream API represents synchronized streams of media. For example, a stream taken from camera and microphone input has synchronized video and audio tracks. (Don't confuse MediaStreamTrack with the <track> element, which is something entirely different.)

Probably the easiest way to understand the MediaStream API is to look at it in the wild:

In your browser, navigate to WebRTC samples getUserMedia.
Open the console.
Inspect the stream variable, which is in global scope.

Each MediaStream has an input, which might be a MediaStream generated by getUserMedia(), and an output, which might be passed to a video element or an RTCPeerConnection.

The getUserMedia() method takes a MediaStreamConstraints object parameter and returns a Promise that resolves to a MediaStream object.

Each MediaStream has a label, such as 'Xk7EuLhsuHKbnjLWkW4yYGNJJ8ONsgwHBvLQ'. An array of MediaStreamTracks is returned by the getAudioTracks() and getVideoTracks() methods.

For the getUserMedia example, stream.getAudioTracks() returns an empty array (because there's no audio) and, assuming a working webcam is connected, stream.getVideoTracks() returns an array of one MediaStreamTrack representing the stream from the webcam. Each MediaStreamTrack has a kind ('video' or 'audio'), a label (something like 'FaceTime HD Camera (Built-in)'), and represents one or more channels of either audio or video. In this case, there is only one video track and no audio, but it is easy to imagine use cases where there are more, such as a chat app that gets streams from the front camera, rear camera, microphone, and an app sharing its screen.

A MediaStream can be attached to a video element by setting the srcObject attribute. Previously, this was done by setting the src attribute to an object URL created with URL.createObjectURL(), but this has been deprecated.

getUserMedia can also be used as an input node for the Web Audio API:

// Cope with browser differences.
let audioContext;
if (typeof AudioContext === 'function') {
  audioContext = new AudioContext();
} else if (typeof webkitAudioContext === 'function') {
  audioContext = new webkitAudioContext(); // eslint-disable-line new-cap
} else {
  console.log('Sorry! Web Audio not supported.');
}

// Create a filter node.
var filterNode = audioContext.createBiquadFilter();
// See https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html#BiquadFilterNode-section
filterNode.type = 'highpass';
// Cutoff frequency. For highpass, audio is attenuated below this frequency.
filterNode.frequency.value = 10000;

// Create a gain node to change audio volume.
var gainNode = audioContext.createGain();
// Default is 1 (no change). Less than 1 means audio is attenuated
// and vice versa.
gainNode.gain.value = 0.5;

navigator.mediaDevices.getUserMedia({audio: true}, (stream) => {
  // Create an AudioNode from the stream.
  const mediaStreamSource =
    audioContext.createMediaStreamSource(stream);
  mediaStreamSource.connect(filterNode);
  filterNode.connect(gainNode);
  // Connect the gain node to the destination. For example, play the sound.
  gainNode.connect(audioContext.destination);
});

Chromium-based apps and extensions can also incorporate getUserMedia. Adding audioCapture and/or videoCapture permissions to the manifest enables permission to be requested and granted only once upon installation. Thereafter, the user is not asked for permission for camera or microphone access.

Permission only has to be granted once for getUserMedia(). First time around, an Allow button is displayed in the browser's infobar. HTTP access for getUserMedia() was deprecated by Chrome at the end of 2015 due to it being classified as a Powerful feature.

The intention is potentially to enable a MediaStream for any streaming data source, not only a camera or microphone. This would enable streaming from stored data or arbitrary data sources, such as sensors or other inputs.

getUserMedia() really comes to life in combination with other JavaScript APIs and libraries:

Webcam Toy is a photobooth app that uses WebGL to add weird and wonderful effects to photos that can be shared or saved locally.
FaceKat is a face-tracking game built with headtrackr.js.
ASCII Camera uses the Canvas API to generate ASCII images.

gUM ASCII art!

Constraints #

Constraints can be used to set values for video resolution for getUserMedia(). This also allows support for other constraints, such as aspect ratio; facing mode (front or back camera); frame rate, height and width; and an applyConstraints() method.

For an example, see WebRTC samples getUserMedia: select resolution.

Setting a disallowed constraint value gives a DOMException or an OverconstrainedError if, for example, a resolution requested is not available. To see this in action, see WebRTC samples getUserMedia: select resolution for a demo.

Screen and tab capture #

Chrome apps also make it possible to share a live video of a single browser tab or the entire desktop through chrome.tabCapture and chrome.desktopCapture APIs. (For a demo and more information, see Screensharing with WebRTC. The article is a few years old, but it's still interesting.)

It's also possible to use screen capture as a MediaStream source in Chrome using the experimental chromeMediaSource constraint. Note that screen capture requires HTTPS and should only be used for development due to it being enabled through a command-line flag as explained in this post.

Signaling: Session control, network, and media information #

WebRTC uses RTCPeerConnection to communicate streaming data between browsers (also known as peers), but also needs a mechanism to coordinate communication and to send control messages, a process known as signaling. Signaling methods and protocols are not specified by WebRTC. Signaling is not part of the RTCPeerConnection API.

Instead, WebRTC app developers can choose whatever messaging protocol they prefer, such as SIP or XMPP, and any appropriate duplex (two-way) communication channel. The appr.tc example uses XHR and the Channel API as the signaling mechanism. The codelab uses Socket.io running on a Node server.

Signaling is used to exchange three types of information:

Session control messages: to initialize or close communication and report errors.
Network configuration: to the outside world, what's your computer's IP address and port?
Media capabilities: what codecs and resolutions can be handled by your browser and the browser it wants to communicate with?

The exchange of information through signaling must have completed successfully before peer-to-peer streaming can begin.

For example, imagine Alice wants to communicate with Bob. Here's a code sample from the W3C WebRTC spec, which shows the signaling process in action. The code assumes the existence of some signaling mechanism created in the createSignalingChannel() method. Also note that on Chrome and Opera, RTCPeerConnection is currently prefixed.

// handles JSON.stringify/parse
const signaling = new SignalingChannel();
const constraints = {audio: true, video: true};
const configuration = {iceServers: [{urls: 'stun:stun.example.org'}]};
const pc = new RTCPeerConnection(configuration);

// Send any ice candidates to the other peer.
pc.onicecandidate = ({candidate}) => signaling.send({candidate});

// Let the "negotiationneeded" event trigger offer generation.
pc.onnegotiationneeded = async () => {
  try {
    await pc.setLocalDescription(await pc.createOffer());
    // Send the offer to the other peer.
    signaling.send({desc: pc.localDescription});
  } catch (err) {
    console.error(err);
  }
};

// Once remote track media arrives, show it in remote video element.
pc.ontrack = (event) => {
  // Don't set srcObject again if it is already set.
  if (remoteView.srcObject) return;
  remoteView.srcObject = event.streams[0];
};

// Call start() to initiate.
async function start() {
  try {
    // Get local stream, show it in self-view, and add it to be sent.
    const stream =
      await navigator.mediaDevices.getUserMedia(constraints);
    stream.getTracks().forEach((track) =>
      pc.addTrack(track, stream));
    selfView.srcObject = stream;
  } catch (err) {
    console.error(err);
  }
}

signaling.onmessage = async ({desc, candidate}) => {
  try {
    if (desc) {
      // If you get an offer, you need to reply with an answer.
      if (desc.type === 'offer') {
        await pc.setRemoteDescription(desc);
        const stream =
          await navigator.mediaDevices.getUserMedia(constraints);
        stream.getTracks().forEach((track) =>
          pc.addTrack(track, stream));
        await pc.setLocalDescription(await pc.createAnswer());
        signaling.send({desc: pc.localDescription});
      } else if (desc.type === 'answer') {
        await pc.setRemoteDescription(desc);
      } else {
        console.log('Unsupported SDP type.');
      }
    } else if (candidate) {
      await pc.addIceCandidate(candidate);
    }
  } catch (err) {
    console.error(err);
  }
};

First, Alice and Bob exchange network information. (The expression finding candidates refers to the process of finding network interfaces and ports using the ICE framework.)

Alice creates an RTCPeerConnection object with an onicecandidate handler, which runs when network candidates become available.
Alice sends serialized candidate data to Bob through whatever signaling channel they are using, such as WebSocket or some other mechanism.
When Bob gets a candidate message from Alice, he calls addIceCandidate to add the candidate to the remote peer description.

WebRTC clients (also known as peers, or Alice and Bob in this example) also need to ascertain and exchange local and remote audio and video media information, such as resolution and codec capabilities. Signaling to exchange media configuration information proceeds by exchanging an offer and an answer using the Session Description Protocol (SDP):

Alice runs the RTCPeerConnection createOffer() method. The return from this is passed an RTCSessionDescription - Alice's local session description.
In the callback, Alice sets the local description using setLocalDescription() and then sends this session description to Bob through their signaling channel. Note that RTCPeerConnection won't start gathering candidates until setLocalDescription() is called. This is codified in the JSEP IETF draft.
Bob sets the description Alice sent him as the remote description using setRemoteDescription().
Bob runs the RTCPeerConnection createAnswer() method, passing it the remote description he got from Alice so a local session can be generated that is compatible with hers. The createAnswer() callback is passed an RTCSessionDescription. Bob sets that as the local description and sends it to Alice.
When Alice gets Bob's session description, she sets that as the remote description with setRemoteDescription.
Ping!

RTCSessionDescription objects are blobs that conform to the Session Description Protocol, SDP. Serialized, an SDP object looks like this:

v=0
o=- 3883943731 1 IN IP4 127.0.0.1
s=
t=0 0
a=group:BUNDLE audio video
m=audio 1 RTP/SAVPF 103 104 0 8 106 105 13 126

// ...

a=ssrc:2223794119 label:H4fjnMzxy3dPIgQ7HxuCTLb4wLLLeRHnFxh810

The acquisition and exchange of network and media information can be done simultaneously, but both processes must have completed before audio and video streaming between peers can begin.

The offer/answer architecture previously described is called JavaScript Session Establishment Protocol, or JSEP. (There's an excellent animation explaining the process of signaling and streaming in Ericsson's demo video for its first WebRTC implementation.)

JSEP architecture

Once the signaling process has completed successfully, data can be streamed directly peer to peer, between the caller and callee - or, if that fails, through an intermediary relay server (more about that later). Streaming is the job of RTCPeerConnection.

RTCPeerConnection #

RTCPeerConnection is the WebRTC component that handles stable and efficient communication of streaming data between peers.

The following is a WebRTC architecture diagram showing the role of RTCPeerConnection. As you will notice, the green parts are complex!

WebRTC architecture (from webrtc.org)

From a JavaScript perspective, the main thing to understand from this diagram is that RTCPeerConnection shields web developers from the myriad complexities that lurk beneath. The codecs and protocols used by WebRTC do a huge amount of work to make real-time communication possible, even over unreliable networks:

Packet-loss concealment
Echo cancellation
Bandwidth adaptivity
Dynamic jitter buffering
Automatic gain control
Noise reduction and suppression
Image-cleaning

The previous W3C code shows a simplified example of WebRTC from a signaling perspective. The following are walkthroughs of two working WebRTC apps. The first is a simple example to demonstrate RTCPeerConnection and the second is a fully operational video chat client.

RTCPeerConnection without servers #

The following code is taken from WebRTC samples Peer connection, which has local and remote RTCPeerConnection (and local and remote video) on one web page. This doesn't constitute anything very useful - caller and callee are on the same page - but it does make the workings of the RTCPeerConnection API a little clearer because the RTCPeerConnection objects on the page can exchange data and messages directly without having to use intermediary signaling mechanisms.

In this example, pc1 represents the local peer (caller) and pc2 represents the remote peer (callee).

Caller #

Create a new RTCPeerConnection and add the stream from getUserMedia():

// Servers is an optional configuration file. (See TURN and STUN discussion later.)
pc1 = new RTCPeerConnection(servers);
// ...
localStream.getTracks().forEach((track) => {
pc1.addTrack(track, localStream);
});

Create an offer and set it as the local description for pc1 and as the remote description for pc2. This can be done directly in the code without using signaling because both caller and callee are on the same page:

pc1.setLocalDescription(desc).then(() => {
    onSetLocalSuccess(pc1);
    },
    onSetSessionDescriptionError
);
trace('pc2 setRemoteDescription start');
pc2.setRemoteDescription(desc).then(() => {
    onSetRemoteSuccess(pc2);
    },
    onSetSessionDescriptionError
);

Callee #

Create pc2 and, when the stream from pc1 is added, display it in a video element:

pc2 = new RTCPeerConnection(servers);
pc2.ontrack = gotRemoteStream;
//...
function gotRemoteStream(e){
vid2.srcObject = e.stream;
}

`RTCPeerConnection` API plus servers #

In the real world, WebRTC needs servers, however simple, so the following can happen:

Users discover each other and exchange real-world details, such as names.
WebRTC client apps (peers) exchange network information.
Peers exchange data about media, such as video format and resolution.
WebRTC client apps traverse NAT gateways and firewalls.

In other words, WebRTC needs four types of server-side functionality:

User discovery and communication
Signaling
NAT/firewall traversal
Relay servers in case peer-to-peer communication fails

NAT traversal, peer-to-peer networking, and the requirements for building a server app for user discovery and signaling are beyond the scope of this article. Suffice to say that the STUN protocol and its extension, TURN, are used by the ICE framework to enable RTCPeerConnection to cope with NAT traversal and other network vagaries.

ICE is a framework for connecting peers, such as two video chat clients. Initially, ICE tries to connect peers directly with the lowest possible latency through UDP. In this process, STUN servers have a single task: to enable a peer behind a NAT to find out its public address and port. (For more information about STUN and TURN, see Build the backend services needed for a WebRTC app.)

Finding connection candidates

If UDP fails, ICE tries TCP. If direct connection fails - in particular because of enterprise NAT traversal and firewalls - ICE uses an intermediary (relay) TURN server. In other words, ICE first uses STUN with UDP to directly connect peers and, if that fails, falls back to a TURN relay server. The expression finding candidates refers to the process of finding network interfaces and ports.

WebRTC data pathways

WebRTC engineer Justin Uberti provides more information about ICE, STUN, and TURN in the 2013 Google I/O WebRTC presentation. (The presentation slides give examples of TURN and STUN server implementations.)

A simple video-chat client #

A good place to try WebRTC, complete with signaling and NAT/firewall traversal using a STUN server, is the video-chat demo at appr.tc. This app uses adapter.js, a shim to insulate apps from spec changes and prefix differences.

The code is deliberately verbose in its logging. Check the console to understand the order of events. The following is a detailed walkthrough of the code.

Network topologies #

WebRTC, as currently implemented, only supports one-to-one communication, but could be used in more complex network scenarios, such as with multiple peers each communicating with each other directly or through a Multipoint Control Unit (MCU), a server that can handle large numbers of participants and do selective stream forwarding, and mixing or recording of audio and video.

Multipoint Control Unit topology example

Many existing WebRTC apps only demonstrate communication between web browsers, but gateway servers can enable a WebRTC app running on a browser to interact with devices, such as telephones (also known as PSTN) and with VOIP systems. In May 2012, Doubango Telecom open sourced the sipml5 SIP client built with WebRTC and WebSocket, which (among other potential uses) enables video calls between browsers and apps running on iOS and Android. At Google I/O, Tethr and Tropo demonstrated a framework for disaster communications in a briefcase using an OpenBTS cell to enable communications between feature phones and computers through WebRTC. Telephone communication without a carrier!

Tethr/Tropo: Disaster communications in a briefcase

`RTCDataChannel` API< #

As well as audio and video, WebRTC supports real-time communication for other types of data.

The RTCDataChannel API enables peer-to-peer exchange of arbitrary data with low latency and high throughput. For single-page demos and to learn how to build a simple file-transfer app, see WebRTC samples and the WebRTC codelab, respectively.

There are many potential use cases for the API, including:

Gaming
Remote desktop apps
Real-time text chat
File transfer
Decentralized networks

The API has several features to make the most of RTCPeerConnection and enable powerful and flexible peer-to-peer communication:

Leveraging of RTCPeerConnection session setup
Multiple simultaneous channels with prioritization
Reliable and unreliable delivery semantics
Built-in security (DTLS) and congestion control
Ability to use with or without audio or video

The syntax is deliberately similar to WebSocket with a send() method and a message event:

const localConnection = new RTCPeerConnection(servers);
const remoteConnection = new RTCPeerConnection(servers);
const sendChannel =
  localConnection.createDataChannel('sendDataChannel');

// ...

remoteConnection.ondatachannel = (event) => {
  receiveChannel = event.channel;
  receiveChannel.onmessage = onReceiveMessage;
  receiveChannel.onopen = onReceiveChannelStateChange;
  receiveChannel.onclose = onReceiveChannelStateChange;
};

function onReceiveMessage(event) {
  document.querySelector("textarea#send").value = event.data;
}

document.querySelector("button#send").onclick = () => {
  var data = document.querySelector("textarea#send").value;
  sendChannel.send(data);
};

Communication occurs directly between browsers, so RTCDataChannel can be much faster than WebSocket even if a relay (TURN) server is required when hole-punching to cope with firewalls and NATs fails.

RTCDataChannel is available in Chrome, Safari, Firefox, Opera, and Samsung Internet. The Cube Slam game uses the API to communicate game state. Play a friend or play the bear! The innovative platform Sharefest enabled file sharing through RTCDataChannel and peerCDN offered a glimpse of how WebRTC could enable peer-to-peer content distribution.

For more information about RTCDataChannel, take a look at the IETF's draft protocol spec.

Security #

There are several ways a real-time communication app or plugin might compromise security. For example:

Unencrypted media or data might be intercepted between browsers, or between a browser and a server.
An app might record and distribute video or audio without the user knowing.
Malware or viruses might be installed alongside an apparently innocuous plugin or app.

WebRTC has several features to avoid these problems:

WebRTC implementations use secure protocols, such as DTLS and SRTP.
Encryption is mandatory for all WebRTC components, including signaling mechanisms.
WebRTC is not a plugin. Its components run in the browser sandbox and not in a separate process. Components do not require separate installation and are updated whenever the browser is updated.
Camera and microphone access must be granted explicitly and, when the camera or microphone are running, this is clearly shown by the user interface.

A full discussion of security for streaming media is out of scope for this article. For more information, see the Proposed WebRTC Security Architecture proposed by the IETF.

In conclusion #

The APIs and standards of WebRTC can democratize and decentralize tools for content creation and communication, including telephony, gaming, video production, music making, and news gathering.

Technology doesn't get much more disruptive than this.

As blogger Phil Edholm put it, "Potentially, WebRTC and HTML5 could enable the same transformation for real-time communication that the original browser did for information."

Developer tools #

WebRTC stats for an ongoing session can be found at:
- about://webrtc-internals in Chrome
- opera://webrtc-internals in Opera
- about:webrtc in Firefox
  
  chrome://webrtc-internals screenshot
Cross browser interop notes
adapter.js is a JavaScript shim for WebRTC maintained by Google with help from the WebRTC community that abstracts vendor prefixes, browser differences, and spec changes.
To learn more about WebRTC signaling processes, check the appr.tc log output to the console.
If it's all too much, you may prefer to use a WebRTC framework or even a complete WebRTC service.
Bug reports and feature requests are always appreciated:

Learn more #

Justin Uberti's WebRTC session at Google I/O 2012
Alan B. Johnston and Daniel C. Burnett maintain a WebRTC book now in its third edition in print and eBook formats at webrtcbook.com.
webrtc.org is home to all things WebRTC, including demos, documentation, and discussion.
discuss-webrtc is a Google Group for technical WebRTC discussion.
@webrtc
Google Developers Talk documentation provides more information about NAT traversal, STUN, relay servers, and candidate gathering.
WebRTC on GitHub
Stack Overflow is a good place to look for answers and ask questions about WebRTC.

Standards and protocols #

The WebRTC W3C Editor's Draft
W3C Editor's Draft: Media Capture and Streams (also known as getUserMedia)
IETF Working Group Charter
IETF WebRTC Data Channel Protocol Draft
IETF JSEP Draft
IETF proposed standard for ICE
IETF RTCWEB Working Group Internet-Draft: Web Real-Time Communication Use-cases and Requirements

WebRTC support summary #

`MediaStream` and `getUserMedia` APIs #

Chrome desktop 18.0.1008 and higher; Chrome for Android 29 and higher
Opera 18 and higher; Opera for Android 20 and higher
Opera 12, Opera Mobile 12 (based on the Presto engine)
Firefox 17 and higher
Microsoft Edge 16 and higher
Safari 11.2 and higher on iOS, and 11.1 and higher on MacOS
UC 11.8 and higher on Android
Samsung Internet 4 and higher

`RTCPeerConnection` API #

Chrome desktop 20 and higher; Chrome for Android 29 and higher (flagless)
Opera 18 and higher (on by default); Opera for Android 20 and higher (on by default)
Firefox 22 and higher (on by default)
Microsoft Edge 16 and higher
Safari 11.2 and higher on iOS, and 11.1 and higher on MacOS
Samsung Internet 4 and higher

`RTCDataChannel` API #

Experimental version in Chrome 25, but more stable (and with Firefox interoperability) in Chrome 26 and higher; Chrome for Android 29 and higher
Stable version (and with Firefox interoperability) in Opera 18 and higher; Opera for Android 20 and higher
Firefox 22 and higher (on by default)

For more detailed information about cross-platform support for APIs, such as getUserMedia and RTCPeerConnection, see caniuse.com and Chrome Platform Status.

Native APIs for RTCPeerConnection are also available at documentation on webrtc.org.

Capture audio and video in HTML5

2012-02-22T00:00:00Z

Introduction #

Audio/Video capture has been the "Holy Grail" of web development for a long time. For many years we've had to rely on browser plugins (Flash or Silverlight) to get the job done. Come on!

HTML5 to the rescue. It might not be apparent, but the rise of HTML5 has brought a surge of access to device hardware. Geolocation (GPS), the Orientation API (accelerometer), WebGL (GPU), and the Web Audio API (audio hardware) are perfect examples. These features are ridiculously powerful, exposing high level JavaScript APIs that sit on top of the system's underlying hardware capabilities.

This tutorial introduces a new API, [navigator.getUserMedia()][getusermedia-spec], which allows web apps to access a user's camera and microphone.

The road to getUserMedia() #

If you're not aware of its history, the way we arrived at the getUserMedia() API is an interesting tale.

Several variants of "Media Capture APIs" have evolved over the past few years. Many folks recognized the need to be able to access native devices on the web, but that led everyone and their mom to put together a new spec. Things got so messy that the W3C finally decided to form a working group. Their sole purpose? Make sense of the madness! The Device APIs Policy (DAP) Working Group has been tasked to consolidate + standardize the plethora of proposals.

I'll try to summarize what happened in 2011…

Round 1: HTML Media Capture #

HTML Media Capture was the DAP's first go at standardizing media capture on the web. It works by overloading the <input type="file"> and adding new values for the accept parameter.

If you wanted to let users take a snapshot of themselves with the webcam, that's possible with capture=camera:

<input type="file" accept="image/*;capture=camera">

Recording a video or audio is similar:

<input type="file" accept="video/*;capture=camcorder">
<input type="file" accept="audio/*;capture=microphone">

Kinda nice right? I particularly like that it reuses a file input. Semantically, it makes a lot of sense. Where this particular "API" falls short is the ability to do realtime effects (e.g. render live webcam data to a <canvas> and apply WebGL filters). HTML Media Capture only allows you to record a media file or take a snapshot in time.

Support:

Android 3.0 browser - one of the first implementations. Check out this video to see it in action.
Chrome for Android (0.16)
Firefox Mobile 10.0
iOS6 Safari and Chrome (partial support)

Round 2: device element #

Many thought HTML Media Capture was too limiting, so a new spec emerged that supported any type of (future) device. Not surprisingly, the design called for a new element, the <device> element, which became the predecessor to getUserMedia().

Opera was among the first browsers to create initial implementations of video capture based on the <device> element. Soon after (the same day to be precise), the WhatWG decided to scrap the <device> tag in favor of another up and comer, this time a JavaScript API called navigator.getUserMedia(). A week later, Opera put out new builds that included support for the updated getUserMedia() spec. Later that year, Microsoft joined the party by releasing a Lab for IE9 supporting the new spec.

Here's what <device> would have looked like:

<device type="media" onchange="update(this.data)"></device>
<video autoplay></video>
<script>
    function update(stream) {
    document.querySelector('video').src = stream.url;
    }
</script>

Support:

Unfortunately, no released browser ever included <device>. One less API to worry about I guess :) <device> did have two great things going for it though: 1.) it was semantic, and 2.) it was easily extendible to support more than just audio/video devices.

Take a breath. This stuff moves fast!

Round 3: WebRTC #

The <device> element eventually went the way of the Dodo.

The pace to find a suitable capture API accelerated thanks to the larger [WebRTC][webrtc-spec] (Web Real Time Communications) effort. That spec is overseen by the W3C WebRTC Working Group. Google, Opera, Mozilla, and a few others have implementations.

getUserMedia() is related to WebRTC because it's the gateway into that set of APIs. It provides the means to access the user's local camera/microphone stream.

Support:

getUserMedia() has been supported since Chrome 21, Opera 18, and Firefox 17.

Getting started #

With navigator.getUserMedia(), we can finally tap into webcam and microphone input without a plugin. Camera access is now a call away, not an install away. It's baked directly into the browser. Excited yet?

Feature detection #

Feature detecting is a simple check for the existence of navigator.getUserMedia:

function hasGetUserMedia() {
    return !!(navigator.getUserMedia || navigator.webkitGetUserMedia ||
            navigator.mozGetUserMedia || navigator.msGetUserMedia);
}

if (hasGetUserMedia()) {
    // Good to go!
} else {
    alert('getUserMedia() is not supported in your browser');
}

You can also use Modernizr to detect getUserMedia to avoid the vendor prefix dance yourself:

if (Modernizr.getusermedia){
    var gUM = Modernizr.prefixed('getUserMedia', navigator);
    gUM({video: true}, function( //...
    //...
}

Gaining access to an input device #

To use the webcam or microphone, we need to request permission. The first parameter to getUserMedia() is an object specifying the details and requirements for each type of media you want to access. For example, if you want to access the webcam, the first parameter should be {video: true}. To use both the microphone and camera, pass {video: true, audio: true}:

<video autoplay></video>

<script>
    var errorCallback = function(e) {
    console.log('Reeeejected!', e);
    };

    // Not showing vendor prefixes.
    navigator.getUserMedia({video: true, audio: true}, function(localMediaStream) {
    var video = document.querySelector('video');
    video.src = window.URL.createObjectURL(localMediaStream);

    // Note: onloadedmetadata doesn't fire in Chrome when using it with getUserMedia.
    // See crbug.com/110938.
    video.onloadedmetadata = function(e) {
        // Ready to go. Do some stuff.
    };
    }, errorCallback);
</script>

OK. So what's going on here? Media capture is a perfect example of new HTML5 APIs working together. It works in conjunction with our other HTML5 buddies, <audio> and <video>. Notice that we're not setting a src attribute or including <source> elements on the <video> element. Instead of feeding the video a URL to a media file, we're feeding it a Blob URL obtained from a LocalMediaStream object representing the webcam.

I'm also telling the <video> to autoplay, otherwise it would be frozen on the first frame. Adding controls also works as you'd expected.

If you want something that works cross-browser, try this:

navigator.getUserMedia  = navigator.getUserMedia ||
                            navigator.webkitGetUserMedia ||
                            navigator.mozGetUserMedia ||
                            navigator.msGetUserMedia;

var video = document.querySelector('video');

if (navigator.getUserMedia) {
    navigator.getUserMedia({audio: true, video: true}, function(stream) {
    video.src = window.URL.createObjectURL(stream);
    }, errorCallback);
} else {
    video.src = 'somevideo.webm'; // fallback.
}

Setting media constraints (resolution, height, width) #

The first parameter to getUserMedia() can also be used to specify more requirements (or constraints) on the returned media stream. For example, instead of just indicating you want basic access to video (e.g. {vide: true}), you can additionally require the stream to be HD:

var hdConstraints = {
    video: {
    mandatory: {
        minWidth: 1280,
        minHeight: 720
    }
    }
};

navigator.getUserMedia(hdConstraints, successCallback, errorCallback);

...

var vgaConstraints = {
    video: {
    mandatory: {
        maxWidth: 640,
        maxHeight: 360
    }
    }
};

navigator.getUserMedia(vgaConstraints, successCallback, errorCallback);

For more configurations, see the constraints API

Selecting a media source #

In Chrome 30 or later, getUserMedia() also supports selecting the video/audio source using the MediaStreamTrack.getSources() API.

In this example, the last microphone and camera that's found is selected as the media stream source:

MediaStreamTrack.getSources(function(sourceInfos) {
    var audioSource = null;
    var videoSource = null;

    for (var i = 0; i != sourceInfos.length; ++i) {
    var sourceInfo = sourceInfos[i];
    if (sourceInfo.kind === 'audio') {
        console.log(sourceInfo.id, sourceInfo.label || 'microphone');

        audioSource = sourceInfo.id;
    } else if (sourceInfo.kind === 'video') {
        console.log(sourceInfo.id, sourceInfo.label || 'camera');

        videoSource = sourceInfo.id;
    } else {
        console.log('Some other kind of source: ', sourceInfo);
    }
    }

    sourceSelected(audioSource, videoSource);
});

function sourceSelected(audioSource, videoSource) {
    var constraints = {
    audio: {
        optional: [{sourceId: audioSource}]
    },
    video: {
        optional: [{sourceId: videoSource}]
    }
    };

    navigator.getUserMedia(constraints, successCallback, errorCallback);
}

Check out Sam Dutton's great demo of how to let users select the media source.

Security #

Some browsers throw up an infobar upon calling getUserMedia(), which gives users the option to grant or deny access to their camera/mic. The spec unfortunately is very quiet when it comes to security. For example, here is Chrome's permission dialog:

Permission dialog in Chrome

If your app is running from SSL (https://), this permission will be persistent. That is, users won't have to grant/deny access every time.

Providing fallback #

For users that don't have support for getUserMedia(), one option is to fallback to an existing video file if the API isn't supported and/or the call fails for some reason:

// Not showing vendor prefixes or code that works cross-browser:

function fallback(e) {
    video.src = 'fallbackvideo.webm';
}

function success(stream) {
    video.src = window.URL.createObjectURL(stream);
}

if (!navigator.getUserMedia) {
    fallback();
} else {
    navigator.getUserMedia({video: true}, success, fallback);
}

Sam Dutton on web.dev

2021 Scroll Survey Report

Noteworthy results #

Overall satisfaction with web scrolling #

Difficulty working with scroll #

Importance of touch interactions #

Difficulty of tab key or gamepad navigation #

Learning touch-action #

Cyclical scrolling #

Are scrollable areas important #

Carousels #

Infinite scroll #

Scroll-linked or scroll-triggered animations #

Compete with built-in scrolling #

Overall satisfaction building scroll interactions on the web #

Survey Takeaways #

Compatibility #

Education #

APIs #

Features #

Resources #

Take the 2021 scroll survey to help improve scrolling on the web

2021 Scroll Survey #

What is FLoC?

Why do we need FLoC? #

What can FLoC be used for? #

How does FLoC work? #

1. FLoC service #

2. Browser #

3. Advertiser: shoestore.example #

4. Publisher: dailynews.example #

5. Adtech platform: adnetwork.example #

Who runs the back-end service that creates the FLoC model? #

How does the FLoC service enable the browser to work out its cohort? #

Can a browser's cohort change? #

How does the browser work out its cohort? #

How does FLoC work out the right size of cohort? #

Can FLoC be used to group people based on sensitive categories? #

Is FLoC just another way of categorizing people online? #

Do websites have to participate and share information? #

How does the FLoC JavaScript API work? #

Can websites opt out of being included in the FLoC computation? #

Can a user stop sites from getting their browser's FLoC cohort? #

How can I make suggestions or provide feedback? #

Find out more #

Payment and address form best practices

Checklist #

Use meaningful HTML #

Use HTML elements as intended #

Put your form in a <form> #

Use <label> to label elements #

Make buttons helpful #

Make the most of HTML attributes #

Make it easy for users to enter data #

Use autocomplete to improve accessibility and help users avoid re-entering data #

Help users enter the right data #

Help users avoid missing required data #

Simplify checkout #

Mind the mobile commerce gap! #

Make guest checkout the default #

Show checkout progress #

Remove distractions #

Make it easy to enter name and address #

Only ask for the data you need #

Use a single name input #

Enable name autofill #

Allow international names #

Allow for a variety of address formats #

Make address forms flexible #

Consider using a single textarea for address #

Internationalize and localize your address forms #

Consider avoiding postal code address lookup #

Simplify payment forms #

Help users avoid re-entering payment data #

Avoid using custom elements for payment card dates #

Use a single input for payment card and phone numbers #

Validate carefully #

Test on a range of devices, platforms, browsers and versions #

Implement analytics and RUM #

Keep learning #

Learning `touch-action` #

Use `<label>` to label elements #

Use `<form>` #

Use `<label>` #

Use `<button>` #

Use `autocomplete="new-password"` and `id="new-password"` for a new password #

Use `autocomplete="current-password"` and `id="current-password"` for an existing password #