Boris Smus on web.dev

Easy high DPI images

2013-03-28T00:00:00Z

Displays with high pixel density are quickly becoming the norm. Content creators need to adapt to this fact. This is a short guide on how to serve high quality images on the web today, without polyfills, JavaScript, CSS hacks, and browser features that aren't quite implemented yet. In a word: without drastic changes to your workflow.

There are many responsive image proposals today, many of which involve significant changes for the web developer. The standards-track srcset <img> attribute is difficult to implement, especially with the complexity of srcset's additional viewport-based selection:

banner-HD.jpeg 2x, banner-phone.jpeg 100w, banner-phone-HD.jpeg 100w 2x

Even though the image-set CSS property only uses devicePixelRatio to decide which image to load, it still forces developers to write a lot of extra markup for every image.

Other proposals, like the <picture> element are even more verbose. Furthermore, they are not standards-track, so their ubiquitous availability is even further out than the srcset attribute. JavaScript and server side solutions are the only other alternative, but these approaches have their own drawbacks as covered in other articles.

This article will go through several uses of images commonly found on the web and propose simple solutions that work on screens with high pixel densities as well as ordinary ones. For the purposes of this discussion, any device that reports window.devicePixelRatio greater than 1 can be considered high DPI, since that means that CSS pixels aren't the same as device pixels, and that images are being scaled up.

Here's a summary:

Use CSS/SVG rather than raster imagery if possible.
Use images optimized for high density displays by default.
Use PNGs for simple drawings and pixel art (eg. logos).
Use compressed JPEGs for images with a variety of colors (eg. photos).
Always set explicit sizes (using CSS or HTML) on all image elements.

Simple drawings and pixel art #

Small images can often be avoided entirely by using CSS features or SVG. There is no need to use images for rounded corners, for example, since the border-radius CSS property is widely supported. Similarly, custom fonts are widely supported, so using "imaged" text is unadvisable.

However, in some cases, like logos, an image may be the only way forward. For example, this Chrome logo has a natural size of 256x256. On a Retina display, you can see the line aliasing at diagonals and curves, which looks chunky and bad, especially when compared to crisply rendered text:

Natural dimensions: 256x256px, asset size: 31 kB, format: PNG

Convinced? Good. Now let's use a high density image. You might be tempted to save space by storing your logo as a JPEG, but this may not be a good idea, since saving logos and other graphics in a lossy format tends to introduce artifacts. In this case, I've exaggerated the problem by using a very high compression, but notice the banding on the gradients, the speckles on white backgrounds and the messy lines:

Natural dimensions: 512x512px, asset size: 13 kB, format: JPEG

The thing to do for relatively small images is to use 2x PNGs. Be aware that the difference in size between a 1x and 2x PNG is generally quite high (52 kB in this case). However, in the case of a logo, it is your website's face and the first thing your visitors will see. By skimping too much on quality in exchange for size, it will also be the last thing your visitors will see!

Here is the Chrome logo in all its glory, sized down to half its natural dimensions for 2x displays:

Natural dimensions: 512x512px, asset size: 83 kB, format: PNG

The markup to make the above render is the following:

<img src="chrome2x.png" style="width: 256px; height: 256px;"/>

Note that I have specified a width and height on the image. This is necessary because the natural size of the image is 512px. It's also good for performance because the rendering engine has a good grasp on the size of the element and won't need to work too hard to compute it.

One possible optimization that might work is to reduce the 24-bit PNG to a paletted 8-bit one. This works for images with a small number of colors, the Chrome logo included. To do this optimization, you can use a tool such as http://pngquant.org/. You can see a bit of banding here, but this file is just 13 kB, which is a whopping 6x size saving compared to the original 512x512 PNG.

Natural dimensions: 512x512px, asset size: 13 kB, format: PNG, 8-bit palette

Images with a variety of colors #

I wrote an HTML5Rocks article surveying a number of different responsive image techniques, and did some research around compressing 1x and 2x JPEG and comparing resulting sizes and visual quality. Here's one such tile from the above article:

I've labeled the images with their compression level (indicated by JPEG quality), their size (in bytes), and my subjective opinion on their comparative visual fidelity (ranked by numbers). The interesting bit here is that the highly compressed 2x image (labeled 3) is smaller in size and looks better than the uncompressed 1x image (labeled 4). In other words, between images 4 and 3, we've managed to improve the quality of the image by doubling each dimension and significantly increasing the compression, while at the same time, reducing the size by 2 kB.

Compression, dimensions and visual quality #

I wanted to get a bit more insight into tradeoffs between compression level, image dimensions, visual quality and image size. I ran a study with the following hypothesis based on the study above:

Hypothesis #

With enough compression, a 2x image will look equivalent to the same image in 1x size at some other (lower) compression. However, in this case, the highly compressed 2x image will be smaller in size than the 1x image.

Process #

Given a 2x image, generate the 1x one.
Compress both images at various levels.
Create a test page that shows both image sets side-by-side.
Find the place in the two sets where the images are equivalent.
Note equivalent image sizes and compression levels.
Try it on both a 1x and a 2x display.

I built a side-by-side image comparison app similar to Lightroom's compare view. The intention is to show a 1x and a 2x images side-by-side, but also allow you to zoom into any section of the image to get more detail. You can also select between JPEG and WebP formats and change compression quality to see file size and image quality comparisons. The idea is to tweak settings over several images, figure out what compression quality, scaling and format vs. image quality tradeoff you are comfortable with, and use that setting for all of your images.

The tool itself is available for you to play with. You can zoom into the image by selecting a sub-area to zoom into.

Analysis #

I'll say up front that image quality is a subjective thing. Also, your particular use case will likely dictate where your priorities lie in the visual fidelity vs. filesize spectrum. Additionally, different kinds of image features react differently to scaling and compression quality, so a one-size-fits-all solution may not necessarily work here. The point of the tool is to help you build up an intuition for image quality compressions, scales and formats.

From playing with the image zoomer, a few things quickly became apparent to me. Firstly, I prefer quality=30 dpr=2x images to quality=90 dpr=1x images for the increase in detail. These images are comparable in file size as well (in the plane case, the compressed 2x image is 76 kB whereas the uncompressed 1x is 80 kB).

Exceptions to this rule are highly compressed (quality<30) images with gradients. These tend to suffer from color banding, which is equally bad regardless of image scale. The bird and car samples found in the tool are examples of this.

WebP images look way cleaner than JPEG, especially at low compression levels. This color banding seems to be much less of an issue. Lastly, WebP images are much more compact.

Caveats and fin #

Making images look good on high density displays is only half of the image related problems caused by a huge variation in screens. In some cases, you might want to serve entirely different images depending on viewport size. For example, Obama's headshot might be appropriate for a phone-sized screen, but the stand in front of him and flag behind him and some might be a better fit for a laptop display.

I deliberately avoided this "art direction" topic to focus on high DPI images only. This problem can be solved by a number of different approaches: using media queries and background images, via JavaScript, via some new features like image-set, or on the server. This topic is covered in High DPI Images for Variable Pixel Densities.

I'll sign off with a few open issues:

Effects of high compression on performance. What are the penalties of decoding highly compressed images?
What are the performance penalties of having to resize the image down when a 2x image is loaded on a 1x display?

To summarize, opt for CSS and SVG instead of using raster images. If raster images are strictly required, use PNGs for images with a limited palettes and many solid colors, and use JPEGs for images with many colors and gradients. The great thing about this approach is that your markup is virtually unchanged. All that is required of the web developer is to generate 2x assets and size your images properly in the DOM.

For further reading, check out Scott Jehl's article on a similar topic. May your images look sharp and your cell data usage be low!

High DPI images for variable pixel densities

2012-08-22T00:00:00Z

One of the features of today's complex device landscape is that there's a very wide range of screen pixel densities available. Some devices feature very high resolution displays, while others straggle behind. Application developers need to support a range of pixel densities, which can be quite challenging. On the mobile web, the challenges are compounded by several factors:

Large variety of devices with different form factors.
Constrained network bandwidth and battery life.

In terms of images, the goal of web app developers is to serve the best quality images as efficiently as possible. This article will cover some useful techniques for doing this today and in the near future.

Avoid images if possible #

Before opening this can of worms, remember that the web has many powerful technologies that are largely resolution- and DPI-independent. Specifically, text, SVG and much of CSS will "just work" because of the automatic pixel scaling feature of the web (via devicePixelRatio).

That said, you can't always avoid raster images. For example, you may be given assets that would be quite hard to replicate in pure SVG/CSS, or you are dealing with a photograph. While you could convert the image into SVG automatically, vectorizing photographs makes little sense because scaled-up versions usually don't look good.

Background #

A very short history of display density #

In the early days, computer displays had a pixel density of 72 or 96dpi (dots per inch).

Displays gradually improved in pixel density, largely driven by the mobile use case, in which users generally hold their phones closer to their faces, making pixels more visible. By 2008, 150dpi phones were the new norm. The trend in increased display density continued, and today's new phones sport 300dpi displays (branded "Retina" by Apple).

The holy grail, of course, is a display in which pixels are completely invisible. For the phone form factor, the current generation of Retina/HiDPI displays may be close to that ideal. But new classes of hardware and wearables like Project Glass will likely continue to drive increased pixel density.

In practice, low density images should look the same on new screens as they did on old ones, but compared to the crisp imagery high density users are used to seeing, the low density images look jarring and pixelated. The following is a rough simulation of how a 1x image will look on a 2x display. In contrast, the 2x image looks quite good.

Baboons! at differing pixel densities.

Pixels on the web #

When the web was designed, 99% of displays were 96dpi (or pretended to be), and few provisions were made for variation on this front. Because of a large variation in screen sizes and densities, we needed a standard way to make images look good across a variety of screen densities and dimensions.

The HTML specification recently tackled this problem by defining a reference pixel that manufacturers use to determine the size of a CSS pixel.

Using the reference pixel, a manufacturer can determine the size of the device’s physical pixel relative to the standard or ideal pixel. This ratio is called the device pixel ratio.

Calculating the device pixel ratio #

Suppose a smart phone has a screen with a physical pixel size of 180 pixels per inch (ppi). Calculating the device pixel ratio takes three steps:

Compare the actual distance at which the device is held to the distance for the reference pixel.

Per the spec, we know that at 28 inches, the ideal is 96 pixels per inch. However, since it's a smart phone, people hold the device closer to their faces than they hold a laptop. Let's estimate that distance to be 18 inches.
Multiply the distance ratio against the standard density (96ppi) to get the ideal pixel density for the given distance.

idealPixelDensity = (28/18) * 96 = 150 pixels per inch (approximately)
Take the ratio of the physical pixel density to the ideal pixel density to get the device pixel ratio.

devicePixelRatio = 180/150 = 1.2

A diagram showing one reference angular pixel, to help illustrate how devicePixelRatio is calculated.

So now when a browser needs to know how to resize an image to fit the screen according to the ideal or standard resolution, the browser refers to the device pixel ratio of 1.2 - which says, for every ideal pixel, this device has 1.2 physical pixels. The formula to go between ideal (as defined by the web spec) and physical (dots on device screen) pixels is the following:

physicalPixels = window.devicePixelRatio * idealPixels

Historically, device vendors have tended to round devicePixelRatios (DPRs). Apple's iPhone and iPad report DPR of 1, and their Retina equivalents report 2. The CSS specification recommends that

the pixel unit refer to the whole number of device pixels that best approximates the reference pixel.

One reason why round ratios can be better is because they may lead to fewer sub-pixel artifacts.

However, the reality of the device landscape is much more varied, and Android phones often have DPRs of 1.5. The Nexus 7 tablet has a DPR of ~1.33, which was arrived at by a calculation similar to the one above. Expect to see more devices with variable DPRs in the future. Because of this, you should never assume that your clients will have integer DPRs.

Overview of HiDPI image techniques #

There are many techniques for solving the problem of showing the best quality images as fast as possible, broadly falling into two categories:

Optimizing single images, and
Optimizing selection between multiple images.

Single image approaches: use one image, but do something clever with it. These approaches have the drawback that you will inevitably sacrifice performance, since you will be downloading HiDPI images even on older devices with lower DPI. Here are some approaches for the single image case:

Heavily compressed HiDPI image
Totally awesome image format
Progressive image format

Multiple image approaches: use multiple images, but do something clever to pick which to load. These approaches have inherent overhead for the developer to create multiple versions of the same asset and then figure out a decision strategy. Here are the options:

JavaScript
Server side delivery
CSS media queries
Built-in browser features (image-set(), <img srcset>)

Heavily compressed HiDPI image #

Images already comprise a whopping 60% of bandwidth spent downloading an average website. By serving HiDPI images to all clients, we will increase this number. How much bigger will it grow?

I ran some tests which generated 1x and 2x image fragments with JPEG quality at 90, 50 and 20. Here is shell script I used (employing ImageMagick) to generate them:

Samples of images at different compressions and pixel densities.

From this small, unscientific sampling, it seems that compressing large images provides a good quality-to-size tradeoff. For my eye, heavily compressed 2x imagery actually looks better than uncompressed 1x pictures.

Of course, serving low quality, highly compressed 2x imagery to 2x devices is worse than serving higher quality ones, and the above approach incurs image quality penalties. If you compare quality: 90 images to quality: 20 images, you will see a drop in crispness and increased graininess. These artifacts may not be acceptable in cases where high quality images are key (for example, a photo viewer application), or for app developers that are not willing to compromise.

The above comparison was made entirely with compressed JPEGs. It's worth noting that there are many tradeoffs between the widely implemented image formats (JPEG, PNG, GIF), which brings us to…

Totally awesome image format #

WebP is a pretty compelling image format that compresses very well while keeping high image fidelity. Of course, it's not implemented everywhere yet!

One way is to check for WebP support is via JavaScript. You load a 1px image via data-uri, wait for either loaded or error events fired, and then verify that the size is correct. Modernizr ships with such a feature detection script, which is available via Modernizr.webp.

A better way of doing this, however, is directly in CSS using the image() function. So if you have a WebP image and JPEG fallback, you can write the following:

#pic {
  background: image("foo.webp", "foo.jpg");
}

There are a few problems with this approach. Firstly, image() is not at all widely implemented. Secondly, while WebP compression blows JPEG out of the water, it's still a relatively incremental improvement – about 30% smaller based on this WebP gallery. Thus, WebP alone isn't enough to address the high DPI problem.

Progressive image formats #

Progressive image formats like JPEG 2000, Progressive JPEG, Progressive PNG and GIF have the (somewhat debated) benefit of seeing the image come into place before it's fully loaded. They may incur some size overhead, though there is conflicting evidence about this. Jeff Atwood claimed that progressive mode "adds about 20% to the size of PNG images, and about 10% to the size of JPEG and GIF images". However, Stoyan Stefanov claimed that for large files, progressive mode is more efficient (in most cases).

At first glance, progressive images look very promising in the context of serving the best quality images as fast as possible. The idea is that the browser can stop downloading and decoding an image once it knows that additional data won't increase the image quality (ie. all of the fidelity improvements are sub-pixel).

While connections are easy to terminate, they are often expensive to restart. For a site with many images, the most efficient approach is to keep a single HTTP connection alive, reusing it for as long as possible. If the connection is terminated prematurely because one image has been downloaded enough, the browser then needs to create a new connection, which can be really slow in low latency environments.

One workaround to this is to use the HTTP Range request, which lets browsers specify a range of bytes to fetch. A smart browser could make a HEAD request to get at the header, process it, decide how much of the image is actually needed, and then fetch. Unfortunately HTTP Range is poorly supported in web servers, making this approach impractical.

Finally, an obvious limitation of this approach is that you don't get to choose which image to load, only varying fidelities of the same image. As a result, this doesn't address the "art direction" use case.

Use JavaScript to decide which image to load #

The first, and most obvious approach to deciding which image to load is to use JavaScript in the client. This approach lets you find out everything about your user agent and do the right thing. You can determine device pixel ratio via window.devicePixelRatio, get screen width and height, and even potentially do some network connection sniffing via navigator.connection or issuing a fake request, like the foresight.js library does. Once you've collected all of this information, you can decide which image to load.

There are approximately one million JavaScript libraries that do something like the above, and unfortunately none of them are particularly outstanding.

One big drawback to this approach is that using JavaScript means that you will delay image loading until after the look-ahead parser has finished. This essentially means that images won't even start downloading until after the pageload event fires. More on this in Jason Grigsby's article.

Decide what image to load on the server #

You can defer the decision to the server-side by writing custom request handlers for each image you serve. Such a handler would check for Retina support based on User-Agent (the only piece of information relayed to the server). Then, based on whether the server-side logic wants to serve HiDPI assets, you load the appropriate asset (named according to some known convention).

Unfortunately, the User-Agent doesn't necessarily provide enough information to decide whether a device should receive high or low quality images. Also, it goes without saying that anything related to User-Agent is a hack and should be avoided if possible.

Use CSS media queries #

Being declarative, CSS media queries let you state your intention, and let the browser do the right thing on your behalf. In addition to the most common use of media queries — matching device size — you can also match devicePixelRatio. The associated media query is device-pixel-ratio, and has associated min and max variants, as you might expect. If you want to load high DPI images and the device pixel ratio exceeds a threshold, here's what you might do:

#my-image { background: (low.png); }

@media only screen and (min-device-pixel-ratio: 1.5) {
  #my-image { background: (high.png); }
}

It gets a little more complicated with all of the vendor prefixes mixed in, especially because of insane differences in placement of "min" and "max" prefixes:

@media only screen and (min--moz-device-pixel-ratio: 1.5),
    (-o-min-device-pixel-ratio: 3/2),
    (-webkit-min-device-pixel-ratio: 1.5),
    (min-device-pixel-ratio: 1.5) {

  #my-image {
    background:url(high.png);
  }
}

With this approach, you regain the benefits of look-ahead parsing, which was lost with the JS solution. You also gain the flexibility of choosing your responsive breakpoints (for example, you can have low, mid and high DPI images), which was lost with the server-side approach.

Unfortunately it's still a little unwieldy, and leads to strange looking CSS (or requires preprocessing). Also, this approach is restricted to CSS properties, so there's no way to set an <img src>, and your images must all be elements with a background. Finally, by relying strictly on device pixel ratio, you can end up in situations where your High-DPI smart phone ends up downloading a massive 2x image asset while on an EDGE connection. This isn't the best user experience.

Use new browser features #

There's been a lot of recent discussion around web platform support for the high DPI image problem. Apple recently broke into the space, bringing the image-set() CSS function to WebKit. As a result, both Safari and Chrome support it. Since it's a CSS function, image-set() doesn't address the problem for <img> tags. Enter @srcset, which addresses this issue but (at the time of this writing) has no reference implementations (yet!). The next section goes deeper into image-set and srcset.

Browser features for high DPI support #

Ultimately, the decision about which approach you take depends on your particular requirements. That said, keep in mind that all of the aforementioned approaches have drawbacks. Looking forward, however, once image-set and srcset are widely supported, they will be the appropriate solutions to this problem. For the time being, let's talk about some best practices that can bring us as close to that ideal future as possible.

Firstly, how are these two different? Well, image-set() is a CSS function, appropriate for use as a value of the background CSS property. srcset is an attribute specific to <img> elements, with similar syntax. Both of these tags let you specify image declarations, but the srcset attribute lets you also configure which image to load based on viewport size.

Best practices for image-set #

The image-set() CSS function is available prefixed as -webkit-image-set(). The syntax is quite simple, taking a one or more comma separated image declarations, which consist of a URL string or url() function followed by the associated resolution. For example:

background-image:  -webkit-image-set(
  url(icon1x.jpg) 1x,
  url(icon2x.jpg) 2x
);

What this tells the browser is that there are two images to choose from. One of them is optimized for 1x displays, and the other for 2x displays. The browser then gets to choose which one to load, based on a variety of factors, which might even include network speed, if the browser is smart enough (not currently implemented as far as I know).

In addition to loading the correct image, the browser will also scale it accordingly. In other words, the browser assumes that 2 images are twice as large as 1x images, and so will scale the 2x image down by a factor of 2, so that the image appears to be the same size on the page.

Instead of specifying 1x, 1.5x or Nx, you can also specify a certain device pixel density in dpi.

This works well, except in browsers that don't support the image-set property, which will show no image at all! This is clearly bad, so you must use a fallback (or series of fallbacks) to address that issue:

background-image: url(icon1x.jpg);
background-image: -webkit-image-set(
  url(icon1x.jpg) 1x,
  url(icon2x.jpg) 2x
);
/* This will be useful if image-set gets into the platform, unprefixed.
    Also include other prefixed versions of this */
background-image: image-set(
  url(icon1x.jpg) 1x,
  url(icon2x.jpg) 2x
);

The above will load the appropriate asset in browsers that support image-set, and fall back to the 1x asset otherwise. The obvious caveat is that while image-set() browser support is low, most user agents will get the 1x asset.

This demo uses the image-set() to load the correct image, falling back to the 1x asset if this CSS function isn't supported.

At this point, you may be wondering why not just polyfill (that is, build a JavaScript shim for) image-set() and call it a day? As it turns out, it's quite difficult to implement efficient polyfills for CSS functions. (For a detailed explanation why, see this www-style discussion).

Image srcset #

Here is an example of srcset:

<img alt="my awesome image"
  src="banner.jpeg"
  srcset="banner-HD.jpeg 2x, banner-phone.jpeg 640w, banner-phone-HD.jpeg 640w 2x">

As you can see, in addition to x declarations that image-set provides, the srcset element also takes w and h values which correspond to the size of the viewport, attempting to serve the most relevant version. The above would serve banner-phone.jpeg to devices with viewport width under 640px, banner-phone-HD.jpeg to small screen high DPI devices, banner-HD.jpeg to high DPI devices with screens greater than 640px, and banner.jpeg to everything else.

Using image-set for image elements #

Because the srcset attribute on img elements is not implemented in most browsers, it may be tempting to replace your img elements with <div>s with backgrounds and use the image-set approach. This will work, with caveats. The drawback here is that the <img> tag has long-time semantic value. In practice, this is important mostly for web crawlers and accessibility reasons.

If you end up using -webkit-image-set, you might be tempted to use the background CSS property. The drawback of this approach is that you need to specify image size, which is unknown if you are using a non-1x image. Rather than doing this, you can use the content CSS property as follows:

<div id="my-content-image"
  style="content: -webkit-image-set(
    url(icon1x.jpg) 1x,
    url(icon2x.jpg) 2x);">
</div>

This will automatically scale the image based on devicePixelRatio. See this example of the above technique in action, with an additional fallback to url() for browsers that don't support image-set.

Polyfilling srcset #

One handy feature of srcset is that it comes with a natural fallback. In the case where the srcset attribute is not implemented, all browsers know to process the src attribute. Also, since it's just an HTML attribute, it's possible to create polyfills with JavaScript.

This polyfill comes with unit tests to ensure that it's as close to the specification as possible. In addition, there are checks in place that prevent the polyfill from executing any code if srcset is implemented natively.

Here is a demo of the polyfill in action.

Conclusion #

There is no magic bullet for solving the problem of high DPI images.

The easiest solution is to avoid images entirely, opting for SVG and CSS instead. However, this isn't always realistic, especially if you have high quality imagery on your site.

Approaches in JS, CSS and using the server-side all have their strengths and weaknesses. The most promising approach, however, is to leverage new browser features. Though browser support for image-set and srcset is still incomplete, there are reasonable fallbacks to use today.

To summarize, my recommendations are as follows:

For background images, use image-set with the appropriate fallbacks for browsers that don't support it.
For content images, use a srcset polyfill, or fallback to using image-set (see above).
For situations where you're willing to sacrifice image quality, consider using heavily compressed 2x images.

A non-responsive approach to building cross-device webapps

2012-04-28T00:00:00Z

Media queries are great, but… #

Media queries are awesome, a godsend for website developers that want to make small tweaks to their stylesheets to give a better experience for users on devices of various sizes. Media queries essentially let you customize the CSS of your site depending on screen size. Before you dive into this article, learn more about responsive design and check out some fine examples of media queries usage here: mediaqueri.es.

As Brad Frost points out in an earlier article, changing the look is only one of many things to consider when building for the mobile web. If the only thing you do when you build your mobile website is customize your layout with media queries, then we have the following situation:

All devices get the same JavaScript, CSS, and assets (images, videos), resulting in longer than necessary load times.
All devices get the same initial DOM, potentially forcing developers to write overly complicated CSS.
Little flexibility to specify custom interactions tailored to each device.

Webapps need more than media queries #

Don't get me wrong. I don't hate responsive design via media queries, and definitely think it has a place in the world. Furthermore, some of the above mentioned issues can be resolved with approaches such as responsive images, dynamic script loading, etc. However, at a certain point, you may find yourself doing too many incremental tweaks, and may be better off serving different versions.

As the UIs you build increase in complexity, and you gravitate toward single-page webapps, you'll want to do more to customize UIs for each type of device. This article will teach you how to do these customizations with a minimal amount of effort. The general approach involves classifying your visitor's device into the right device class, and serving the appropriate version to that device, while maximizing code reuse between versions.

What device classes are you targeting? #

There are tons of internet-connected devices out there, and nearly all of them have browsers. The complication lies in their diversity: Mac laptops, Windows workstations, iPhones, iPads, Android phones with touch input, scroll wheels, keyboards, voice input, devices with pressure sensitivity, smart watches, toasters and refrigerators, and many more. Some of these devices are ubiquitous, while others are very rare.

A variety of devices (source).

To create a good user experience, you need to know who your users are and what devices they are using. If you build a user interface for a desktop user with a mouse and a keyboard and give it to a smartphone user, your interface will be a frustration because it's designed for another screen size, and another input modality.

There are two extreme ends to the spectrum of approaches:

Build one version that works on all devices. UX will suffer as a result, since different devices have different design considerations.
Build a version for each device you want to support. This will take forever, because you'll be building too many versions of your application. Also, when the next new smartphone arrives (which happens roughly weekly), you will be forced to create yet another version.

There is a fundamental tradeoff here: the more device categories you have, the better a user experience you can deliver, but the more work it will take to design, implement and maintain.

Creating a separate version for each device class you decide on may be a good idea for performance reasons or if the versions you want to serve to different device classes vary hugely. Otherwise, responsive web design is a perfectly reasonable approach.

A potential solution #

Here's a compromise: classify devices into categories, and design the best possible experience for each category. What categories you choose depend on your product and target user. Here's a sample classification that nicely spans popular web-capable devices that exist today.

small screens + touch (mostly phones)
large screens + touch (mostly tablets)
large screens + keyboard/mouse (mostly desktops/laptops)

This is only one of many possible breakdowns, but one that makes a lot of sense at the time of writing. Missing from the above list are mobile devices without touch screens (eg. feature phones, some dedicated ebook readers). However, most of these have keyboard navigation or screen reader software installed, which will work fine if you build your site with accessibility in mind.

Examples of form factor-specific web apps #

There are many examples of web properties serving entirely different versions for different form factors. Google search does this, as does Facebook. Considerations for this include both performance (fetching assets, rendering pages) and more general user experience.

In the world of native apps, many developers choose to tailor their experience to a device class. For example, Flipboard for iPad has a very different UI compared to Flipboard on iPhone. The tablet version is optimized for two hand use and horizontal flipping while the phone version is intended for single hand interaction and a vertical flip. Many other iOS applications also provide significantly different phone and tablet versions, such as Things (todo list), and Showyou (social video), featured below:

Significant UI customization for phone and tablet.

Approach #1: Server-side detection #

On the server, we have a much more limited understanding of the device that we're dealing with. Probably the most useful clue that's available is the user agent string, which is supplied via the User-Agent header on every request. Because of this, the same UA sniffing approach will work here. In fact, the DeviceAtlas and WURFL projects do this already (and give a whole lot of additional information about the device).

Unfortunately each of these present their own challenges. WURFL is very large, containing 20MB of XML, potentially incurring significant server-side overhead for each request. There are projects that split the XML for performance reasons. DeviceAtlas is not open source, and requires a paid license to use.

There are simpler, free alternatives too, like the Detect Mobile Browsers project. The drawback, of course, is that device detection will inevitably be less comprehensive. Also, it only distinguishes between mobile and non-mobile devices, providing limited tablet support only through an ad-hoc set of tweaks.

Approach #2: Client-side detection #

We can learn a lot about the user's browser and device by using feature detection. The main things we need to determine are if the device has touch capability, and if it's a large or small screen.

We need to draw the line somewhere to distinguish small and big touch devices. What about edge cases like the 5" Galaxy Note? The following graphic shows a bunch of popular Android and iOS devices overlaid (with corresponding screen resolutions). The asterisk indicates that the device comes or can come in doubled density. Though the pixel density may be doubled, CSS still reports the same sizes.

A quick aside on pixels in CSS: CSS pixels on the mobile web aren't the same as screen pixels. iOS retina devices introduced the practice of doubling pixel density (eg. iPhone 3GS vs 4, iPad 2 vs 3). The retina Mobile Safari UAs still report the same device-width to avoid breaking the web. As other devices (eg. Android) get higher resolution displays, they are doing the same device-width trick.

Device resolution (in pixels).

Complicating this decision, however, is the importance of considering both portrait and landscape modes. We don't want to reload the page or load additional scripts every time we re-orient the device, though we may want to render the page differently.

In the following diagram, squares represent the max dimensions of each device, as a result of overlaying the portrait and landscape outlines (and completing the square):

Portrait + landscape resolution (in pixels)

By setting the threshold to 650px, we classify iPhone, Galaxy Nexus as smalltouch, and iPad, Galaxy Tab as "tablet". The androgynous Galaxy Note is in this case classified as "phone", and will get the phone layout.

And so, a reasonable strategy might look like this:

if (hasTouch) {
  if (isSmall) {
    device = PHONE;
  } else {
    device = TABLET;
  }
} else {
  device = DESKTOP;
}

See a minimal sample of the feature-detection approach in action.

The alternative approach here is to use UA sniffing to detect device type. Basically you create a set of heuristics and match them against your user's navigator.userAgent. Pseudo code looks something like this:

var ua = navigator.userAgent;
for (var re in RULES) {
  if (ua.match(re)) {
    device = RULES[re];
    return;
  }
}

See a sample of the UA-detection approach in action.

A note on client-side loading #

If you're doing UA detection on your server, you can decide what CSS, JavaScript and DOM to serve when you get a new request. However, if you're doing client-side detection, the situation is more complex. You have several options:

Redirect to a device-type-specific URL that contains the version for this device type.
Dynamically load the device type-specific assets.

The first approach is straightforward, requiring a redirect such as window.location.href = '/tablet'. However, the location will now have this device type information appended to it, so you may want to use the History API to clean up your URL. Unfortunately this approach involves a redirect, which can be slow, especially on mobile devices.

The second approach is quite a bit more complex to implement. You need a mechanism to dynamically load CSS and JS, and (browser-depending), you may not be able to do things like customize <meta viewport>. Also, since there's no redirect, you're stuck with the original HTML that was served. Of course, you can manipulate it with JavaScript, but this may be slow and/or inelegant, depending on your application.

Deciding client or server #

These are the tradeoffs between the approaches:

Pro client:

More future proof since based on screen sizes/capabilities rather than UA.
No need to constantly update UA list.

Pro server:

Full control of what version to serve to what devices.
Better performance: no need for client redirects or dynamic loading.

My personal preference is to start with device.js and client-side detection. As your application evolves, if you find client-side redirect to be a significant performance drawback, you can easily remove the device.js script, and implement UA detection on the server.

Introducing device.js #

Device.js is a starting point for doing semantic, media query-based device detection without needing special server-side configuration, saving the time and effort required to do user agent string parsing.

The idea is that you provide search-engine-friendly markup (link rel=alternate) at the top of your <head> indicating which versions of your site you want to provide.

<link rel="alternate" href="http://foo.com" id="desktop"
    media="only screen and (touch-enabled: 0)">

Next, you can either do server-side UA detection and handle version redirection on your own, or use the device.js script to do feature-based client-side redirection.

For more information, see the device.js project page, and also a fake application that uses device.js for client-side redirection.

Recommendation: MVC with form-factor specific views

By now you're probably thinking that I'm telling you to build three completely separate apps, one for each device type. No! Code sharing is the key.

Hopefully you have been using an MVC-like framework, such as Backbone, Ember, etc. If you have been, you are familiar with the principle of separation of concerns, specifically that your UI (view layer) should be decoupled from your logic (model layer). If this is new to you, get started with some of these resources on MVC, and MVC in JavaScript.

The cross-device story fits neatly into your existing MVC framework. You can easily move your views into separate files, creating a custom view for each device type. Then you can serve the same code to all devices, except the view layer.

Cross-device MVC.

Your project might have the following structure (of course, you are free to choose the structure that makes the most sense depending on your application):

models/ (shared models) item.js item-collection.js

controllers/ (shared controllers) item-controller.js

versions/ (device-specific stuff) tablet/ desktop/ phone/ (phone-specific code) style.css index.html views/ item.js item-list.js

This sort of structure enables you to fully control what assets each version loads, since you have custom HTML, CSS and JavaScript for each device. This is very powerful, and can lead to the leanest, most performant way of developing for the cross-device web, without relying on tricks such as adaptive images.

Once you run your favorite build tool, you'll concatenate and minify all of your JavaScript and CSS into single files for faster loading, with your production HTML looking something like the following (for phone, using device.js):

<!doctype html>
<head>
  <title>Mobile Web Rocks! (Phone Edition)</title>

  <!-- Every version of your webapp should include a list of all
        versions. -->
  <link rel="alternate" href="http://foo.com" id="desktop"
      media="only screen and (touch-enabled: 0)">
  <link rel="alternate" href="http://m.foo.com" id="phone"
      media="only screen and (max-device-width: 650px)">
  <link rel="alternate" href="http://tablet.foo.com" id="tablet"
      media="only screen and (min-device-width: 650px)">

  <!-- Viewport is very important, since it affects results of media
        query matching. -->
  <meta name="viewport" content="width=device-width">

  <!-- Include device.js in each version for redirection. -->
  <script src="device.js"></script>

  <link rel="style" href="phone.min.css">
</head>
<body>
  <script src="phone.min.js"></script>
</body>

Note that the (touch-enabled: 0) media query is non-standard (only implemented in Firefox behind a moz vendor prefix), but is handled correctly (thanks to Modernizr.touch) by device.js.

Version override #

Device detection can sometimes go wrong, and in some cases, a user may prefer to look at the tablet layout on their phone (perhaps they are using a Galaxy Note), so it's important to give your users a choice of which version of your site to use if they want to manually override.

The usual approach is to provide a link to the desktop version from your mobile version. This is easy enough to implement, but device.js supports this functionality with the device GET parameter.

Concluding #

To summarize, when building cross-device single-page UIs, that don't fit neatly into the world of responsive design, do this:

Pick a set of device classes to support, and criteria by which to classify devices into classes.
Build your MVC app with strong separation of concerns, splitting views from the rest of the codebase.
Use device.js to do client side device class detection.
When you're ready, package your script and stylesheets into one of each per device class.
If client-side redirection performance is an issue, abandon device.js and switch to serverside UA-detection.

Developing game audio with the Web Audio API

2012-02-28T00:00:00Z

Introduction #

Audio is a huge part of what makes multimedia experiences so compelling. If you've ever tried watching a movie with the sound off, you've probably noticed this.

Games are no exception! My fondest video game memories are of the music and sound effects. Now, in many cases nearly two decades after playing my favorites, I still can't get Koji Kondo's Zelda compositions and Matt Uelmen's atmospheric Diablo soundtrack out of my head. The same catchiness applies to sound effects, such as the instantly recognizable unit click responses from Warcraft, and samples from Nintendo's classics.

Game audio presents some interesting challenges. To create convincing game music, designers need to adjust to potentially unpredictable game state a player finds themselves in. In practice, parts of the game can go on for an unknown duration, sounds can interact with the environment and mix in complex ways, such as room effects and relative sound positioning. Finally, there can be a large number of sounds playing at once, all of which need to sound good together and render without introducing performance penalties.

Game Audio on the Web #

For simple games, using the <audio> tag may be sufficient. However, many browsers provide poor implementations, which result in audio glitches and high latency. This is hopefully a temporary problem, since vendors are working hard to improve their respective implementations. For a glimpse into the state of the <audio> tag, there is a nice test suite at areweplayingyet.org.

Looking deeper into the <audio> tag specification, however, it becomes clear that there are many things that simply can't be done with it, which isn't surprising, since it was designed for media playback. Some limitations include:

No ability to apply filters to the sound signal
No way to access the raw PCM data
No concept of position and direction of sources and listeners
No fine-grained timing.

In the rest of the article, I'll dive into some of these topics in the context of game audio written with the Web Audio API. For a brief introduction to this API, take a look at the getting started tutorial.

Background music #

Games often have background music playing on a loop.

It can get very annoying if your loop is short and predictable. If a player is stuck in an area or level, and the same sample continuously plays in the background, it may be worthwhile to gradually fade out the track to prevent further frustration. Another strategy is to have mixes of various intensity that gradually crossfade into one another, depending on the context of the game.

For example, if your player is in a zone with an epic boss battle, you might have several mixes varying in emotional range from atmospheric to foreshadowing to intense. Music synthesis software often allows you to export several mixes (of the same length) based on a piece by picking the set of tracks to use in the export. That way you have some internal consistency and avoid having jarring transitions as you cross-fade from one track to another.

Then, using the Web Audio API, you can import all of these samples using something like the BufferLoader class via XHR (this is covered in depth in the introductory Web Audio API article. Loading sounds takes time, so assets that are used in the game should be loaded on page load, at the start of the level, or perhaps incrementally while the player is playing.

Next, you create a source for each node, and a gain node for each source, and connect the graph.

After doing this, you can play back all of these sources simultaneously on a loop, and since they are all the same length, the Web Audio API will guarantee that they will remain aligned. As the character gets nearer or further from the final boss battle, the game can vary the gain values for each of the respective nodes in the chain, using a gain amount algorithm like the following:

// Assume gains is an array of AudioGainNode, normVal is the intensity
// between 0 and 1.
var value = normVal - (gains.length - 1);
// First reset gains on all nodes.
for (var i = 0; i < gains.length; i++) {
    gains[i].gain.value = 0;
}
// Decide which two nodes we are currently between, and do an equal
// power crossfade between them.
var leftNode = Math.floor(value);
// Normalize the value between 0 and 1.
var x = value - leftNode;
var gain1 = Math.cos(x - 0.5*Math.PI);
var gain2 = Math.cos((1.0 - x) - 0.5*Math.PI);
// Set the two gains accordingly.
gains[leftNode].gain.value = gain1;
// Check to make sure that there's a right node.
if (leftNode < gains.length - 1) {
    // If there is, adjust its gain.
    gains[leftNode + 1].gain.value = gain2;
}

In the above approach, two sources play at once, and we crossfade between them using equal power curves (as described in the intro).

The missing link: Audio tag to Web Audio #

Many game developers today use the <audio> tag for their background music, because it is well suited to streaming content. Now you can bring content from the <audio> tag into a Web Audio context.

This technique can be useful since the <audio> tag can work with streaming content, which lets you immediately play the background music instead of having to wait for it all to download. By bringing the stream into the Web Audio API, you can manipulate or analyze the stream. The following example applies a low pass filter to the music played through the <audio> tag:

var audioElement = document.querySelector('audio');
var mediaSourceNode = context.createMediaElementSource(audioElement);
// Create the filter
var filter = context.createBiquadFilter();
// Create the audio graph.
mediaSourceNode.connect(filter);
filter.connect(context.destination);

For a more complete discussion about integrating the <audio> tag with the Web Audio API, see this short article.

Sound effects #

Games often play back sound effects in response to user input or changes in game state. Like background music, however, sound effects can get annoying very quickly. To avoid this, it's often useful to have a pool of similar but different sounds to play. This can vary from mild variations of footstep samples, to drastic variations, as seen in the Warcraft series in response to clicking on units.

Another key feature of sound effects in games is that there can be many of them simultaneously. Imagine you're in the middle of a gunfight with multiple actors shooting machine guns. Each machine gun fires many times per second, causing tens of sound effects to be played at the same time. Playing back sound from multiple, precisely timed sources simultaneously is one place the Web Audio API really shines.

The following example creates a machine gun round from multiple individual bullet samples by creating multiple sound sources whose playback is staggered in time.

var time = context.currentTime;
for (var i = 0; i < rounds; i++) {
    var source = this.makeSource(this.buffers[M4A1]);
    source.noteOn(time + i - interval);
}

Now, if all of the machine guns in your game sounded exactly like this, that would be pretty boring. Of course they would vary by sound based on distance from the target and relative position (more on this later), but even that might not be enough. Luckily the Web Audio API provides a way to easily tweak the example above in two ways:

With a subtle shift in time between bullets firing
By altering playbackRate of each sample (also changing pitch) to better simulate randomness of the real world.

For a more real-life example of these techniques in action, take a look at the Pool Table demo, which uses random sampling and varies playbackRate for a more interesting ball collision sound.

3D positional sound #

Games are often set in a world with some geometric properties, either in 2D or in 3D. If this is the case, stereo positioned audio can greatly increase the immersiveness of the experience. Luckily, Web Audio API comes with built-in hardware accelerated positional audio features that are quite straight forward to use. By the way, you should make sure that you've got stereo speakers (preferably headphones) for the following example to make sense.

In the above example, there is a listener (person icon) in the middle of the canvas, and the mouse affects the position of the source (speaker icon). The above is a simple example of using AudioPannerNode to achieve this sort of effect. The basic idea of the sample above is to respond to mouse movement by setting the position of the audio source, as follows:

PositionSample.prototype.changePosition = function(position) {
    // Position coordinates are in normalized canvas coordinates
    // with -0.5 < x, y < 0.5
    if (position) {
    if (!this.isPlaying) {
        this.play();
    }
    var mul = 2;
    var x = position.x / this.size.width;
    var y = -position.y / this.size.height;
    this.panner.setPosition(x - mul, y - mul, -0.5);
    } else {
    this.stop();
    }
};

Things to know about Web Audio's treatment of spatialization:

The listener is at the origin (0, 0, 0) by default.
Web Audio positional APIs are unitless, so I introduced a multiplier to make the demo sound better.
Web Audio uses the y-is-up cartesian coordinates (the opposite of most computer graphics systems). That's why I'm swapping the y-axis in the snippet above

Advanced: sound cones #

The positional model is very powerful and quite advanced, largely based on OpenAL. For more details, see sections 3 and 4 of the above-linked spec.

There is a single AudioListener attached to the Web Audio API context which can be configured in space through position and orientation. Each source can be passed through an AudioPannerNode, which spatializes the input audio. The panner node has position and orientation, as well as a distance and directional model.

The distance model specifies the amount of gain depending on proximity to the source, while the directional model can be configured by specifying an inner and outer cone, which determine amount of (usually negative) gain if the listener is within the inner cone, between the inner and outer cone, or outside the outer cone.

var panner = context.createPanner();
panner.coneOuterGain = 0.5;
panner.coneOuterAngle = 180;
panner.coneInnerAngle = 0;

Though my example is in 2D, this model generalizes easily to the third dimension. For an example of sound spatialized in 3D, see this positional sample. In addition to position, the Web Audio sound model also optionally includes velocity for doppler shifts. This example shows the doppler effect in more detail.

For more information on this topic, read this detailed tutorial on [mixing positional audio and WebGL][webgl].

Room effects and filters #

In reality, the way sound is perceived greatly depends on the room in which that sound is heard. The same creaky door will sound very different in a basement, compared to an large open hall. Games with high production value will want to imitate these effects, since creating a separate set of samples for each environment is prohibitively expensive, and would lead to even more assets, and a larger amount of game data.

Loosely speaking, the audio term for the difference between the raw sound and the way it sounds in reality is the impulse response. These impulse responses can be painstakingly recorded, and in fact there are sites that host many of these pre-recorded impulse response files (stored as audio) for your convenience.

For a lot more information about how impulse responses can be created from a given environment, read through the "Recording Setup" section in the Convolution part of the Web Audio API spec.

More importantly for our purposes, the Web Audio API provides an easy way to apply these impulse responses to our sounds using the ConvolverNode.

// Make a source node for the sample.
var source = context.createBufferSource();
source.buffer = this.buffer;
// Make a convolver node for the impulse response.
var convolver = context.createConvolver();
convolver.buffer = this.impulseResponseBuffer;
// Connect the graph.
source.connect(convolver);
convolver.connect(context.destination);

Also see this demo of room effects on the Web Audio API spec page, as well as this example which gives you control over dry (raw) and wet (processed via convolver) mixing of a great Jazz standard.

The final countdown #

So you've built a game, configured your positional audio, and now you have a large number of AudioNodes in your graph, all playing back simultaneously. Great, but there's still one more thing to consider:

Since multiple sounds just stack on top of one another with no normalization, you may find yourself in a situation where you are exceeding the threshold of your speaker's capability. Like images exceeding past their canvas boundaries, sounds can also clip if the waveform exceeds its maximum threshold, producing a distinct distortion. The waveform looks something like this:

Here's a real example of clipping in action. The waveform looks bad:

It's important to listen to harsh distortions like the one above, or conversely, overly subdued mixes that force your listeners to crank up the volume. If you're in this situation, you really need to fix it!

Detect clipping #

From a technical perspective, clipping happens when the value of the signal in any channel exceeds the valid range, namely between -1 and 1. Once this is detected, it's useful to give visual feedback that this is happening. To do this reliably, put a JavaScriptAudioNode into your graph. The audio graph would be setup as follows:

// Assume entire sound output is being piped through the mix node.
var meter = context.createJavaScriptNode(2048, 1, 1);
meter.onaudioprocess = processAudio;
mix.connect(meter);
meter.connect(context.destination);

And clipping could be detected in the following processAudio handler:

function processAudio(e) {
    var buffer = e.inputBuffer.getChannelData(0);

    var isClipping = false;
    // Iterate through buffer to check if any of the |values| exceeds 1.
    for (var i = 0; i < buffer.length; i++) {
    var absValue = Math.abs(buffer[i]);
    if (absValue >= 1) {
        isClipping = true;
        break;
    }
    }
}

In general, be careful not to overuse the JavaScriptAudioNode for performance reasons. In this case, an alternative implementation of metering could poll a RealtimeAnalyserNode in the audio graph for getByteFrequencyData, at render time, as determined by requestAnimationFrame. This approach is more efficient, but misses most of the signal (including places where it potentially clips), since rendering happens at most at 60 times a second, whereas the audio signal changes far more quickly.

Because clip detection is so important, it's likely that we will see a built-in MeterNode Web Audio API node in the future.

Prevent clipping #

By adjusting the gain on the master AudioGainNode, you can subdue your mix to a level that prevents clipping. However, in practice, since the sounds playing in your game may depend on a huge variety of factors, it can be difficult to decide on the master gain value that prevents clipping for all states. In general, you should tweak gains to anticipate the worst case, but this is more of an art than a science.

Add a bit of sugar #

Compressors are commonly used in music and game production to smooth over the signal and control spikes in the overall signal. This functionality is available in the Web Audio world via the DynamicsCompressorNode, which can be inserted in your audio graph to give a louder, richer and fuller sound, and also help with clipping. Directly quoting the spec, this node

Using dynamics compression is generally a good idea, especially in a game setting, where, as previously discussed, you don't know exactly what sounds will play and when. Plink from DinahMoe labs is a great example of this, since the sounds that are played back completely depends on you and other participants. A compressor is useful in most cases, except some rare ones, where you're dealing with painstakingly mastered tracks that have been tuned to sound "just right" already.

Implementing this is simply a matter of including a DynamicsCompressorNode in your audio graph, generally as the last node before the destination.:

// Assume the output is all going through the mix node.
var compressor = context.createDynamicsCompressor();
mix.connect(compressor);
compressor.connect(context.destination);

For more detail about dynamics compression, this Wikipedia article is very informative.

To summarize, listen carefully for clipping and prevent it by inserting a master gain node. Then tightening the whole mix by using a dynamics compressor node. Your audio graph might look something like this:

Conclusion #

That covers what I think are the most important aspects of game audio development using the Web Audio API. With these techniques, you can build truly compelling audio experiences right in your browser. Before I sign off, let me leave you with a browser-specific tip: be sure to pause sound if your tab goes to the background using the page visibility API, otherwise you will create a potentially frustrating experience for your user.

For additional information about Web Audio, take a look at the more introductory getting started article, and if you have a question, see if it's already answered in the web audio FAQ. Finally, if you have additional questions, ask them on Stack Overflow using the web-audio tag.

Before I sign off, let me leave you with some awesome uses of the Web Audio API in real games today:

Field Runners, and a writeup about some of the technical details.
Angry Birds, recently switched to Web Audio API. See this writeup for more information.
Skid Racer, which makes great use of spatialized audio.

Working with Custom Elements

2011-10-14T00:00:00Z

Introduction #

The web severely lacks expression. To see what I mean, take a peek at a "modern" web app like GMail:

There's nothing modern about <div> soup. And yet, this is how we build web apps. It's sad. Shouldn't we demand more from our platform?

Sexy markup. Let's make it a thing #

HTML gives us an excellent tool for structuring a document but its vocabulary is limited to elements the HTML standard defines.

What if the markup for GMail wasn't atrocious? What if it was beautiful:

<hangout-module>
    <hangout-chat from="Paul, Addy">
    <hangout-discussion>
        <hangout-message from="Paul" profile="profile.png"
            profile="118075919496626375791" datetime="2013-07-17T12:02">
        <p>Feelin' this Web Components thing.
        <p>Heard of it?
        </hangout-message>
    </hangout-discussion>
    </hangout-chat>
    <hangout-chat>...</hangout-chat>
</hangout-module>

How refreshing! This app totally makes sense too. It's meaningful, easy to understand, and best of all, it's maintainable. Future me/you will know exactly what it does just by examining its declarative backbone.

Getting started #

Custom Elements allow web developers to define new types of HTML elements. The spec is one of several new API primitives landing under the Web Components umbrella, but it's quite possibly the most important. Web Components don't exist without the features unlocked by custom elements:

Define new HTML/DOM elements
Create elements that extend from other elements
Logically bundle together custom functionality into a single tag
Extend the API of existing DOM elements

Registering new elements #

Custom elements are created using document.registerElement():

var XFoo = document.registerElement('x-foo');
document.body.appendChild(new XFoo());

The first argument to document.registerElement() is the element's tag name. The name must contain a dash (-). So for example, <x-tags>, <my-element>, and <my-awesome-app> are all valid names, while <tabs> and <foo_bar> are not. This restriction allows the parser to distinguish custom elements from regular elements but also ensures forward compatibility when new tags are added to HTML.

The second argument is an (optional) object describing the element's prototype. This is the place to add custom functionality (e.g. public properties and methods) to your elements. More on that later.

By default, custom elements inherit from HTMLElement. Thus, the previous example is equivalent to:

var XFoo = document.registerElement('x-foo', {
    prototype: Object.create(HTMLElement.prototype)
});

A call to document.registerElement('x-foo') teaches the browser about the new element, and returns a constructor that you can use to create instances of <x-foo>. Alternatively, you can use the other techniques of instantiating elements if you don't want to use the constructor.

Extending elements #

Custom elements allows you to extend existing (native) HTML elements as well as other custom elements. To extend an element, you need to pass registerElement() the name and prototype of the element to inherit from.

Extending native elements #

Say you aren't happy with Regular Joe <button>. You'd like to supercharge its capabilities to be a "Mega Button". To extend the <button> element, create a new element that inherits the prototype of HTMLButtonElement and extends the name of the element. In this case, "button":

var MegaButton = document.registerElement('mega-button', {
    prototype: Object.create(HTMLButtonElement.prototype),
    extends: 'button'
});

Custom elements that inherit from native elements are called type extension custom elements. They inherit from a specialized version of HTMLElement as a way to say, "element X is a Y".

Example:

<button is="mega-button">

Extending a custom element #

To create an <x-foo-extended> element that extends the <x-foo> custom element, simply inherit its prototype and say what tag you're inheriting from:

var XFooProto = Object.create(HTMLElement.prototype);
...

var XFooExtended = document.registerElement('x-foo-extended', {
    prototype: XFooProto,
    extends: 'x-foo'
});

See Adding JS properties and methods below for more information on creating element prototypes.

How elements are upgraded #

Have you ever wondered why the HTML parser doesn't throw a fit on non-standard tags? For example, it's perfectly happy if we declare <randomtag> on the page. According to the HTML specification:

Sorry <randomtag>! You're non-standard and inherit from HTMLUnknownElement.

The same is not true for custom elements. Elements with valid custom element names inherit from HTMLElement. You can verify this fact by firing up the Console: Ctrl + Shift + J (or Cmd + Opt + J on Mac), and paste in the following lines of code; they return true:

// "tabs" is not a valid custom element name
document.createElement('tabs').__proto__ === HTMLUnknownElement.prototype

// "x-tabs" is a valid custom element name
document.createElement('x-tabs').__proto__ == HTMLElement.prototype

Unresolved elements #

Because custom elements are registered by script using document.registerElement(), they can be declared or created before their definition is registered by the browser. For example, you can declare <x-tabs> on the page but end up invoking document.registerElement('x-tabs') much later.

Before elements are upgraded to their definition, they're called unresolved elements. These are HTML elements that have a valid custom element name but haven't been registered.

This table might help keep things straight:

Name	Inherits from	Examples
Unresolved element	`HTMLElement`	`<x-tabs>`, `<my-element>`
Unknown element	`HTMLUnknownElement`	`<tabs>`, `<foo_bar>`

Instantiating elements #

The common techniques of creating elements still apply to custom elements. As with any standard element, they can be declared in HTML or created in DOM using JavaScript.

Instantiating custom tags #

Declare them:

<x-foo></x-foo>

Create DOM in JS:

var xFoo = document.createElement('x-foo');
xFoo.addEventListener('click', function(e) {
    alert('Thanks!');
});

Use the new operator:

var xFoo = new XFoo();
document.body.appendChild(xFoo);

Instantiating type extension elements #

Instantiating type extension-style custom elements is strikingly close to custom tags.

Declare them:

<!-- <button> "is a" mega button -->
<button is="mega-button">

Create DOM in JS:

var megaButton = document.createElement('button', 'mega-button');
// megaButton instanceof MegaButton === true

As you can see, there's now an overloaded version of document.createElement() that takes the is="" attribute as its second parameter.

Use the new operator:

var megaButton = new MegaButton();
document.body.appendChild(megaButton);

So far, we've learned how to use document.registerElement() to tell the browser about a new tag…but it doesn't do much. Let's add properties and methods.

Adding JS properties and methods #

The powerful thing about custom elements is that you can bundle tailored functionality with the element by defining properties and methods on the element definition. Think of this as a way to create a public API for your element.

Here's a full example:

var XFooProto = Object.create(HTMLElement.prototype);

// 1. Give x-foo a foo() method.
XFooProto.foo = function() {
    alert('foo() called');
};

// 2. Define a property read-only "bar".
Object.defineProperty(XFooProto, "bar", {value: 5});

// 3. Register x-foo's definition.
var XFoo = document.registerElement('x-foo', {prototype: XFooProto});

// 4. Instantiate an x-foo.
var xfoo = document.createElement('x-foo');

// 5. Add it to the page.
document.body.appendChild(xfoo);

Of course there are umpteen thousand ways to construct a prototype. If you're not a fan of creating prototypes like this, here's a more condensed version of the same thing:

var XFoo = document.registerElement('x-foo', {
  prototype: Object.create(HTMLElement.prototype, {
    bar: {
      get: function () {
        return 5;
      }
    },
    foo: {
      value: function () {
        alert('foo() called');
      }
    }
  })
});

The first format allows for the use of ES5 Object.defineProperty. The second allows the use of get/set.

Lifecycle callback methods #

Elements can define special methods for tapping into interesting times of their existence. These methods are appropriately named the lifecycle callbacks. Each has a specific name and purpose:

Callback name	Called when
createdCallback	an instance of the element is created
attachedCallback	an instance was inserted into the document
detachedCallback	an instance was removed from the document
attributeChangedCallback(attrName, oldVal, newVal)	an attribute was added, removed, or updated

Example: defining createdCallback() and attachedCallback() on <x-foo>:

var proto = Object.create(HTMLElement.prototype);

proto.createdCallback = function() {...};
proto.attachedCallback = function() {...};

var XFoo = document.registerElement('x-foo', {prototype: proto});

All of the lifecycle callbacks are optional, but define them if/when it makes sense. For example, say your element is sufficiently complex and opens a connection to IndexedDB in createdCallback(). Before it gets removed from the DOM, do the necessary cleanup work in detachedCallback(). Note: you shouldn't rely on this, for example, if the user closes the tab, but think of it as a possible optimization hook.

Another use case lifecycle callbacks is for setting up default event listeners on the element:

proto.createdCallback = function() {
  this.addEventListener('click', function(e) {
    alert('Thanks!');
  });
};

Adding markup #

We've created <x-foo>, given it a JavaScript API, but it's blank! Shall we give it some HTML to render?

Lifecycle callbacks come in handy here. Particularly, we can use createdCallback() to endow an element with some default HTML:

var XFooProto = Object.create(HTMLElement.prototype);

XFooProto.createdCallback = function() {
    this.innerHTML = "**I'm an x-foo-with-markup!**";
};

var XFoo = document.registerElement('x-foo-with-markup', {prototype: XFooProto});

Instantiating this tag and inspecting in the DevTools (right-click, select Inspect Element) should show:

▾<x-foo-with-markup>
  **I'm an x-foo-with-markup!**
</x-foo-with-markup>

Encapsulating the internals in Shadow DOM #

By itself, Shadow DOM is a powerful tool for encapsulating content. Use it in conjunction with custom elements and things get magical!

Shadow DOM gives custom elements:

A way to hide their guts, thus shielding users from gory implementation details.
Style encapsulation…fo' free.

Creating an element from Shadow DOM is like creating one that renders basic markup. The difference is in createdCallback():

var XFooProto = Object.create(HTMLElement.prototype);

XFooProto.createdCallback = function() {
    // 1. Attach a shadow root on the element.
    var shadow = this.createShadowRoot();

    // 2. Fill it with markup goodness.
    shadow.innerHTML = "**I'm in the element's Shadow DOM!**";
};

var XFoo = document.registerElement('x-foo-shadowdom', {prototype: XFooProto});

Instead of setting the element's .innerHTML, I've created a Shadow Root for <x-foo-shadowdom> and then filled it with markup. With the "Show Shadow DOM" setting enabled in the DevTools, you'll see a #shadow-root that can be expanded:

▾<x-foo-shadowdom>
  ▾#shadow-root
    **I'm in the element's Shadow DOM!**
</x-foo-shadowdom>

That's the Shadow Root!

Creating elements from a template #

HTML Templates are another new API primitive that fits nicely into the world of custom elements.

Example: registering an element created from a <template> and Shadow DOM:

<template id="sdtemplate">
  <style>
    p { color: orange; }
  </style>
  <p>I'm in Shadow DOM. My markup was stamped from a <template&gt;.
</template>

<script>
  var proto = Object.create(HTMLElement.prototype, {
    createdCallback: {
      value: function() {
        var t = document.querySelector('#sdtemplate');
        var clone = document.importNode(t.content, true);
        this.createShadowRoot().appendChild(clone);
      }
    }
  });
  document.registerElement('x-foo-from-template', {prototype: proto});
</script>

<template id="sdtemplate">
  <style>:host p { color: orange; }</style>
  <p>I'm in Shadow DOM. My markup was stamped from a <template&gt;.
</template>

<div class="demoarea">
  <x-foo-from-template></x-foo-from-template>
</div>

These few lines of code pack a lot of punch. Let's understanding everything that is happening:

We've registered a new element in HTML: <x-foo-from-template>

The element's DOM was created from a <template>
The element's scary details are hidden away using Shadow DOM
Shadow DOM gives the element style encapsulation (e.g. p {color: orange;} isn't turning the entire page orange)

So good!

Styling custom elements #

As with any HTML tag, users of your custom tag can style it with selectors:

<style>
  app-panel {
    display: flex;
  }
  [is="x-item"] {
    transition: opacity 400ms ease-in-out;
    opacity: 0.3;
    flex: 1;
    text-align: center;
    border-radius: 50%;
  }
  [is="x-item"]:hover {
    opacity: 1.0;
    background: rgb(255, 0, 255);
    color: white;
  }
  app-panel > [is="x-item"] {
    padding: 5px;
    list-style: none;
    margin: 0 7px;
  }
</style>

<app-panel>
    <li is="x-item">Do</li>
    <li is="x-item">Re</li>
    <li is="x-item">Mi</li>
</app-panel>

Styling elements that use Shadow DOM #

The rabbit hole goes much much deeper when you bring Shadow DOM into the mix. Custom elements that use Shadow DOM inherit its great benefits.

Shadow DOM infuses an element with style encapsulation. Styles defined in a Shadow Root don't leak out of the host and don't bleed in from the page. In the case of a custom element, the element itself is the host. The properties of style encapsulation also allow custom elements to define default styles for themselves.

Shadow DOM styling is a huge topic! If you want to learn more about it, I recommend a few of my other articles:

"A Guide to Styling Elements" on Polymer's documentation.
"Shadow DOM 201: CSS & Styling" here.

FOUC prevention using :unresolved #

To mitigate FOUC, custom elements spec out a new CSS pseudo class, :unresolved. Use it to target unresolved elements, right up until the point where the browser invokes your createdCallback() (see lifecycle methods). Once that happens, the element is no longer an unresolved element. The upgrade process is complete and the element has transformed into its definition.

Example: fade in "x-foo" tags when they're registered:

<style>
  x-foo {
    opacity: 1;
    transition: opacity 300ms;
  }
  x-foo:unresolved {
    opacity: 0;
  }
</style>

Keep in mind that :unresolved only applies to unresolved elements, not to elements that inherit from HTMLUnknownElement (see How elements are upgraded).

<style>
  /* apply a dashed border to all unresolved elements */
  :unresolved {
    border: 1px dashed red;
    display: inline-block;
  }
  /* x-panel's that are unresolved are red */
  x-panel:unresolved {
    color: red;
  }
  /* once the definition of x-panel is registered, it becomes green */
  x-panel {
    color: green;
    display: block;
    padding: 5px;
    display: block;
  }
</style>

<panel>
    I'm black because :unresolved doesn't apply to "panel".
    It's not a valid custom element name.
</panel>

<x-panel>I'm red because I match x-panel:unresolved.</x-panel>

History and browser support #

Feature detection #

Feature detecting is a matter of checking if document.registerElement() exists:

function supportsCustomElements() {
    return 'registerElement' in document;
}

if (supportsCustomElements()) {
    // Good to go!
} else {
    // Use other libraries to create components.
}

Browser support #

document.registerElement() first started landing behind a flag in Chrome 27 and Firefox ~23. However, the specification has evolved quite a bit since then. Chrome 31 is the first to have true support for the updated spec.

Until browser support is stellar, there's a polyfill which is used by Google's Polymer and Mozilla's X-Tag.

What happened to HTMLElementElement? #

For those that have followed the standardization work, you know there was once <element>. It was the bees knees. You could use it to declaratively register new elements:

<element name="my-element">
    ...
</element>

Unfortunately, there were too many timing issues with the upgrade process, corner cases, and Armageddon-like scenarios to work it all out. <element> had to be shelved. In August 2013, Dimitri Glazkov posted to public-webapps announcing its removal, at least for now.

It's worth noting that Polymer implements a declarative form of element registration with <polymer-element>. How? It uses document.registerElement('polymer-element') and the techniques I described in Creating elements from a template.

Conclusion #

Custom elements give us the tool to extend HTML's vocabulary, teach it new tricks, and jump through the wormholes of the web platform. Combine them with the other new platform primitives like Shadow DOM and <template>, and we start to realize the picture of Web Components. Markup can be sexy again!

If you're interested in getting started with web components, I recommend checking out Polymer. It's got more than enough to get you going.

Getting started with Web Audio API

2011-10-14T00:00:00Z

Before the HTML5 <audio> element, Flash or another plugin was required to break the silence of the web. While audio on the web no longer requires a plugin, the audio tag brings significant limitations for implementing sophisticated games and interactive applications.

The Web Audio API is a high-level JavaScript API for processing and synthesizing audio in web applications. The goal of this API is to include capabilities found in modern game audio engines and some of the mixing, processing, and filtering tasks that are found in modern desktop audio production applications. What follows is a gentle introduction to using this powerful API.

Getting started with the AudioContext #

An AudioContext is for managing and playing all sounds. To produce a sound using the Web Audio API, create one or more sound sources and connect them to the sound destination provided by the AudioContext instance. This connection doesn't need to be direct, and can go through any number of intermediate AudioNodes which act as processing modules for the audio signal. This routing is described in greater detail at the Web Audio specification.

A single instance of AudioContext can support multiple sound inputs and complex audio graphs, so we will only need one of these for each audio application we create.

The following snippet creates an AudioContext:

var context;
window.addEventListener('load', init, false);
function init() {
    try {
    context = new AudioContext();
    }
    catch(e) {
    alert('Web Audio API is not supported in this browser');
    }
}

For older WebKit-based browsers, use the webkit prefix, as with webkitAudioContext.

Many of the interesting Web Audio API functionality such as creating AudioNodes and decoding audio file data are methods of AudioContext.

Loading sounds #

The Web Audio API uses an AudioBuffer for short- to medium-length sounds. The basic approach is to use XMLHttpRequest for fetching sound files.

The API supports loading audio file data in multiple formats, such as WAV, MP3, AAC, OGG and others. Browser support for different audio formats varies.

The following snippet demonstrates loading a sound sample:

var dogBarkingBuffer = null;
var context = new AudioContext();

function loadDogSound(url) {
    var request = new XMLHttpRequest();
    request.open('GET', url, true);
    request.responseType = 'arraybuffer';

    // Decode asynchronously
    request.onload = function() {
    context.decodeAudioData(request.response, function(buffer) {
        dogBarkingBuffer = buffer;
    }, onError);
    }
    request.send();
}

The audio file data is binary (not text), so we set the responseType of the request to 'arraybuffer'. For more information about ArrayBuffers, see this article about XHR2.

Once the (undecoded) audio file data has been received, it can be kept around for later decoding, or it can be decoded right away using the AudioContext decodeAudioData() method. This method takes the ArrayBuffer of audio file data stored in request.response and decodes it asynchronously (not blocking the main JavaScript execution thread).

When decodeAudioData() is finished, it calls a callback function which provides the decoded PCM audio data as an AudioBuffer.

Playing sounds #

A simple audio graph

Once one or more AudioBuffers are loaded, then we're ready to play sounds. Let's assume we've just loaded an AudioBuffer with the sound of a dog barking and that the loading has finished. Then we can play this buffer with a the following code.

var context = new AudioContext();

function playSound(buffer) {
    var source = context.createBufferSource(); // creates a sound source
    source.buffer = buffer;                    // tell the source which sound to play
    source.connect(context.destination);       // connect the source to the context's destination (the speakers)
    source.noteOn(0);                          // play the source now
}

This playSound() function could be called every time somebody presses a key or clicks something with the mouse.

The noteOn(time) function makes it easy to schedule precise sound playback for games and other time-critical applications. However, to get this scheduling working properly, ensure that your sound buffers are pre-loaded.

Abstracting the Web Audio API #

Of course, it would be better to create a more general loading system which isn't hard-coded to loading this specific sound. There are many approaches for dealing with the many short- to medium-length sounds that an audio application or game would use–here's one way using a BufferLoader (not part of web standard).

The following is an example of how you can use the BufferLoader class. Let's create two AudioBuffers; and, as soon as they are loaded, let's play them back at the same time.

window.onload = init;
var context;
var bufferLoader;

function init() {
    context = new AudioContext();

    bufferLoader = new BufferLoader(
    context,
    [
        '../sounds/hyper-reality/br-jam-loop.wav',
        '../sounds/hyper-reality/laughter.wav',
    ],
    finishedLoading
    );

    bufferLoader.load();
}

function finishedLoading(bufferList) {
    // Create two sources and play them both together.
    var source1 = context.createBufferSource();
    var source2 = context.createBufferSource();
    source1.buffer = bufferList[0];
    source2.buffer = bufferList[1];

    source1.connect(context.destination);
    source2.connect(context.destination);
    source1.noteOn(0);
    source2.noteOn(0);
}

Dealing with time: playing sounds with rhythm #

The Web Audio API lets developers precisely schedule playback. To demonstrate this, let's set up a simple rhythm track. Probably the most widely known drumkit pattern is the following:

A simple rock drum pattern

in which a hihat is played every eighth note, and kick and snare are played alternating every quarter, in 4/4 time.

Supposing we have loaded the kick, snare and hihat buffers, the code to do this is simple:

for (var bar = 0; bar < 2; bar++) {
    var time = startTime + bar * 8 * eighthNoteTime;
    // Play the bass (kick) drum on beats 1, 5
    playSound(kick, time);
    playSound(kick, time + 4 * eighthNoteTime);

    // Play the snare drum on beats 3, 7
    playSound(snare, time + 2 * eighthNoteTime);
    playSound(snare, time + 6 * eighthNoteTime);

    // Play the hi-hat every eighth note.
    for (var i = 0; i < 8; ++i) {
    playSound(hihat, time + i * eighthNoteTime);
    }
}

Here, we make only one repeat instead of the unlimited loop we see in the sheet music. The function playSound is a method that plays a buffer at a specified time, as follows:

function playSound(buffer, time) {
    var source = context.createBufferSource();
    source.buffer = buffer;
    source.connect(context.destination);
    source.noteOn(time);
}

Changing the volume of a sound #

One of the most basic operations you might want to do to a sound is change its volume. Using the Web Audio API, we can route our source to its destination through an AudioGainNode in order to manipulate the volume:

Audio graph with a gain node

This connection setup can be achieved as follows:

// Create a gain node.
var gainNode = context.createGainNode();
// Connect the source to the gain node.
source.connect(gainNode);
// Connect the gain node to the destination.
gainNode.connect(context.destination);

After the graph has been set up, you can programmatically change the volume by manipulating the gainNode.gain.value as follows:

// Reduce the volume.
gainNode.gain.value = 0.5;

Cross-fading between two sounds #

Now, suppose we have a slightly more complex scenario, where we're playing multiple sounds but want to cross fade between them. This is a common case in a DJ-like application, where we have two turntables and want to be able to pan from one sound source to another.

This can be done with the following audio graph:

Audio graph with two sources connected through gain nodes

To set this up, we simply create two AudioGainNodes, and connect each source through the nodes, using something like this function:

function createSource(buffer) {
    var source = context.createBufferSource();
    // Create a gain node.
    var gainNode = context.createGainNode();
    source.buffer = buffer;
    // Turn on looping.
    source.loop = true;
    // Connect source to gain.
    source.connect(gainNode);
    // Connect gain to destination.
    gainNode.connect(context.destination);

    return {
    source: source,
    gainNode: gainNode
    };
}

Equal power crossfading #

A naive linear crossfade approach exhibits a volume dip as you pan between the samples.

A linear crossfade

To address this issue, we use an equal power curve, in which the corresponding gain curves are non-linear, and intersect at a higher amplitude. This minimizes volume dips between audio regions, resulting in a more even crossfade between regions that might be slightly different in level.

An equal power crossfade

Playlist crossfading #

Another common crossfader application is for a music player application. When a song changes, we want to fade the current track out, and fade the new one in, to avoid a jarring transition. To do this, schedule a crossfade into the future. While we could use setTimeout to do this scheduling, this is not precise. With the Web Audio API, we can use the AudioParam interface to schedule future values for parameters such as the gain value of an AudioGainNode.

Thus, given a playlist, we can transition between tracks by scheduling a gain decrease on the currently playing track, and a gain increase on the next one, both slightly before the current track finishes playing:

function playHelper(bufferNow, bufferLater) {
    var playNow = createSource(bufferNow);
    var source = playNow.source;
    var gainNode = playNow.gainNode;
    var duration = bufferNow.duration;
    var currTime = context.currentTime;
    // Fade the playNow track in.
    gainNode.gain.linearRampToValueAtTime(0, currTime);
    gainNode.gain.linearRampToValueAtTime(1, currTime + ctx.FADE_TIME);
    // Play the playNow track.
    source.noteOn(0);
    // At the end of the track, fade it out.
    gainNode.gain.linearRampToValueAtTime(1, currTime + duration-ctx.FADE_TIME);
    gainNode.gain.linearRampToValueAtTime(0, currTime + duration);
    // Schedule a recursive track change with the tracks swapped.
    var recurse = arguments.callee;
    ctx.timer = setTimeout(function() {
    recurse(bufferLater, bufferNow);
    }, (duration - ctx.FADE_TIME) - 1000);
}

The Web Audio API provides a convenient set of RampToValue methods to gradually change the value of a parameter, such as linearRampToValueAtTime and exponentialRampToValueAtTime.

While the transition timing function can be picked from built-in linear and exponential ones (as above), you can also specify your own value curve via an array of values using the setValueCurveAtTime function.

Applying a simple filter effect to a sound #

An audio graph with a BiquadFilterNode

The Web Audio API lets you pipe sound from one audio node into another, creating a potentially complex chain of processors to add complex effects to your soundforms.

One way to do this is to place BiquadFilterNodes between your sound source and destination. This type of audio node can do a variety of low-order filters which can be used to build graphic equalizers and even more complex effects, mostly to do with selecting which parts of the frequency spectrum of a sound to emphasize and which to subdue.

Supported types of filters include:

Low pass filter
High pass filter
Band pass filter
Low shelf filter
High shelf filter
Peaking filter
Notch filter
All pass filter

And all of the filters include parameters to specify some amount of gain, the frequency at which to apply the filter, and a quality factor. The low-pass filter keeps the lower frequency range, but discards high frequencies. The break-off point is determined by the frequency value, and the Q factor is unitless, and determines the shape of the graph. The gain only affects certain filters, such as the low-shelf and peaking filters, and not this low-pass filter.

Let's setup a simple low-pass filter to extract only the bases from a sound sample:

// Create the filter
var filter = context.createBiquadFilter();
// Create the audio graph.
source.connect(filter);
filter.connect(context.destination);
// Create and specify parameters for the low-pass filter.
filter.type = 0; // Low-pass filter. See BiquadFilterNode docs
filter.frequency.value = 440; // Set cutoff to 440 HZ
// Playback the sound.
source.noteOn(0);

In general, frequency controls need to be tweaked to work on a logarithmic scale since human hearing itself works on the same principle (that is, A4 is 440hz, and A5 is 880hz). For more details, see the FilterSample.changeFrequency function in the source code link above.

Lastly, note that the sample code lets you connect and disconnect the filter, dynamically changing the AudioContext graph. We can disconnect AudioNodes from the graph by calling node.disconnect(outputNumber). For example, to re-route the graph from going through a filter, to a direct connection, we can do the following:

// Disconnect the source and filter.
source.disconnect(0);
filter.disconnect(0);
// Connect the source directly.
source.connect(context.destination);

Further listening #

We've covered the basics of the API, including loading and playing audio samples. We've built audio graphs with gain nodes and filters, and scheduled sounds and audio parameter tweaks to enable some common sound effects. At this point, you are ready to go and build some sweet web audio applications!

If you are seeking inspiration, many developers have already created great work using the Web Audio API. Some of my favorite include:

AudioJedit, an in-browser sound splicing tool that uses SoundCloud permalinks.
ToneCraft, a sound sequencer where sounds are created by stacking 3D blocks.
Plink, a collaborative music-making game using Web Audio and Web Sockets.

Multi-touch web development

2011-08-21T00:00:00Z

Introduction #

Mobile devices such as smartphones and tablets usually have a capacitive touch-sensitive screen to capture interactions made with the user's fingers. As the mobile web evolves to enable increasingly sophisticated applications, web developers need a way to handle these events. For example, nearly any fast-paced game requires the player to press multiple buttons at once, which, in the context of a touchscreen, implies multi-touch.

Apple introduced their touch events API in iOS 2.0. Android has been catching up to this de-facto standard and closing the gap. Recently a W3C working group has come together to work on this touch events specification.

In this article I’ll dive into the touch events API provided by iOS and Android devices, as well as desktop Chrome on hardware that supports touch, and explore what sorts of applications you can build, present some best practices, and cover useful techniques that make it easier to develop touch-enabled applications.

Touch events #

Three basic touch events are outlined in the spec and implemented widely across mobile devices:

touchstart: a finger is placed on a DOM element.
touchmove: a finger is dragged along a DOM element.
touchend: a finger is removed from a DOM element.

Each touch event includes three lists of touches:

touches: a list of all fingers currently on the screen.
targetTouches: a list of fingers on the current DOM element.
changedTouches: a list of fingers involved in the current event. For example, in a touchend event, this will be the finger that was removed.

These lists consist of objects that contain touch information:

identifier: a number that uniquely identifies the current finger in the touch session.
target: the DOM element that was the target of the action.
client/page/screen coordinates: where on the screen the action happened.
radius coordinates and rotationAngle: describe the ellipse that approximates finger shape.

Touch-enabled apps #

The touchstart, touchmove, and touchend events provide a rich enough feature set to support virtually any kind of touch-based interaction – including all of the usual multi-touch gestures like pinch-zoom, rotation, and so on.

This snippet lets you drag a DOM element around using single-finger touch:

var obj = document.getElementById('id');
obj.addEventListener('touchmove', function(event) {
  // If there's exactly one finger inside this element
  if (event.targetTouches.length == 1) {
    var touch = event.targetTouches[0];
    // Place element where the finger is
    obj.style.left = touch.pageX + 'px';
    obj.style.top = touch.pageY + 'px';
  }
}, false);

Below is a sample that displays all current touches on the screen. It’s useful just to get a feeling for the responsiveness of the device.

// Setup canvas and expose context via ctx variable
canvas.addEventListener('touchmove', function(event) {
  for (var i = 0; i < event.touches.length; i++) {
    var touch = event.touches[i];
    ctx.beginPath();
    ctx.arc(touch.pageX, touch.pageY, 20, 0, 2*Math.PI, true);
    ctx.fill();
    ctx.stroke();
  }
}, false);

Demos #

A number of interesting multi-touch demos are already in the wild, such as this canvas-based drawing demo by Paul Irish and others.

And Browser Ninja, a tech demo that is a Fruit Ninja clone using CSS3 transforms and transitions, as well as canvas:

Best practices #

Prevent zooming #

Default settings don't work very well for multi-touch, since your swipes and gestures are often associated with browser behavior, such as scrolling and zooming.

To disable zooming, setup your viewport so that it is not user scalable using the following meta tag:

<meta name="viewport" 
  content="width=device-width, initial-scale=1.0, user-scalable=no>

Check out this mobile HTML5 article for more information on setting your viewport.

Prevent scrolling #

Some mobile devices have default behaviors for touchmove, such as the classic iOS overscroll effect, which causes the view to bounce back when scrolling exceeds the bounds of the content. This is confusing in many multi-touch applications, and can easily be disabled:

document.body.addEventListener('touchmove', function(event) {
  event.preventDefault();
}, false);

Render carefully #

If you are writing a multi-touch application that involves complex multi-finger gestures, be careful how you react to touch events, since you will be handling so many at once. Consider the sample in the previous section that draws all touches on the screen. You could draw as soon as there is a touch input:

canvas.addEventListener('touchmove', function(event) {
  renderTouches(event.touches);
}, false);

But this technique does not scale with number of fingers on the screen. Instead, you could track all of the fingers, and render in a loop to get far better performance:

var touches = []
canvas.addEventListener('touchmove', function(event) {
  touches = event.touches;
}, false);

// Setup a 60fps timer
timer = setInterval(function() {
  renderTouches(touches);
}, 15);

Make use of targetTouches and changedTouches #

Remember that event.touches is an array of ALL fingers in contact with the screen, not just the ones on the DOM element's target. You might find it much more useful to use event.targetTouches or event.changedTouches instead.

Finally, since you are developing for mobile, you should be aware of general mobile best practices, which are covered in Eric Bidelman's article, as well as this W3C document.

Device support #

Unfortunately, touch event implementations vary greatly in completeness and quality. I wrote a diagnostics script that displays some basic information about the touch API implementation, including which events are supported, and touchmove firing resolution. I tested Android 2.3.3 on Nexus One and Nexus S hardware, Android 3.0.1 on Xoom, and iOS 4.2 on iPad and iPhone.

In a nutshell, all tested browsers support the touchstart, touchend, and touchmove events.

The spec provides three additional touch events, but no tested browsers support them:

touchenter: a moving finger enters a DOM element.
touchleave: a moving finger leaves a DOM element.
touchcancel: a touch is interrupted (implementation specific).

Within each touch list, the tested browsers also provide the touches, targetTouches and changedTouches touch lists. However, no tested browsers support radiusX, radiusY or rotationAngle, which specify the shape of the finger touching the screen.

During a touchmove, events fire at roughly 60 times a second across all tested devices.

Android 2.3.3 (Nexus) #

On the Android Gingerbread Browser (tested on Nexus One and Nexus S), there is no multi-touch support. This is a known issue.

Android 3.0.1 (Xoom) #

On Xoom's browser, there is basic multi-touch support, but it only works on a single DOM element. The browser does not correctly respond to two simultaneous touches on different DOM elements. In other words, the following will react to two simultaneous touches:

obj1.addEventListener('touchmove', function(event) {
  for (var i = 0; i < event.targetTouches; i++) {
    var touch = event.targetTouches[i];
    console.log('touched ' + touch.identifier);
  }
}, false);

But the following will not:

var objs = [obj1, obj2];
for (var i = 0; i < objs.length; i++) {
  var obj = objs[i];
  obj.addEventListener('touchmove', function(event) {
    if (event.targetTouches.length == 1) {
      console.log('touched ' + event.targetTouches[0].identifier);
    }
  }, false);
}

iOS 4.x (iPad, iPhone) #

iOS devices fully support multi-touch, are capable of tracking quite a few fingers and provide a very responsive touch experience in the browser.

Developer tools #

In mobile development, it's often easier to start prototyping on the desktop and then tackle the mobile-specific parts on the devices you intend to support. Multi-touch is one of those features that's difficult to test on the PC, since most PCs don't have touch input.

Having to test on mobile can lengthen your development cycle, since every change you make needs to be pushed out to a server and then loaded on the device. Then, once running, there’s little you can do to debug your application, since tablets and smartphones lack web developer tooling.

A solution to this problem is to simulate touch events on your development machine. For single-touches, touch events can be simulated based on mouse events. Multi-touch events can be simulated if you have a device with touch input, such as a modern Apple MacBook.

Single-touch events #

If you would like to simulate single-touch events on your desktop, Chrome provides touch event emulation from the developer tools. Open up the Developer tools, then select the Settings gear, then "Overrides" or "Emulation", and turn on "Emulate touch events".

For other browsers, you may wish to try out Phantom Limb, which simulates touch events on pages and also gives a giant hand to boot.

There's also the Touchable jQuery plugin that unifies touch and mouse events across platforms.

Multi-touch events #

To enable your multi-touch web application to work in your browser on your multi-touch trackpad (such as a Apple MacBook or MagicPad), I've created the MagicTouch.js polyfill. It captures touch events from your trackpad and turns them into standard-compatible touch events.

Download and install the npTuioClient NPAPI plugin into ~/Library/Internet Plug-Ins/.
Download the TongSeng TUIO app for Mac’s MagicPad and start the server.
Download MagicTouch.js, a javascript library to simulate spec-compatible touch events based on npTuioClient callbacks.
Include the magictouch.js script and npTuioClient plugin in your application as follows:

<head>
  ...
  <script src="/path/to/magictouch.js"></script>
</head>

<body>
  ...
  <object id="tuio" type="application/x-tuio" style="width: 0px; height: 0px;">
    Touch input plugin failed to load!
  </object>
</body>

You may need to enable the plugin.

A live demo with magictouch.js is available at paulirish.com/demo/multi:

I tested this approach only with Chrome 10, but it should work on other modern browsers with only minor tweaks.

If your computer does not have multi-touch input, you can simulate touch events using other TUIO trackers, such as the reacTIVision. For more information, see the TUIO project page.

Note that your gestures might be identical to OS-level multi-touch gestures. On OS X, you can configure system-wide events by going to the Trackpad preference pane in System Preferences.

As multi-touch features become more widely supported across mobile browsers, I'm very excited to see new web applications take full advantage of this rich API.

Improving HTML5 Canvas performance

2011-08-16T00:00:00Z

Introduction #

HTML5 canvas, which started as an experiment from Apple, is the most widely supported standard for 2D immediate mode graphics on the web. Many developers now rely on it for a wide variety of multimedia projects, visualizations, and games. However, as the applications we build increase in complexity, developers inadvertently hit the performance wall. There’s a lot of disconnected wisdom about optimizing canvas performance. This article aims to consolidate some of this body into a more readily digestible resource for developers. This article includes fundamental optimizations that apply to all computer graphics environments as well as canvas-specific techniques that are subject to change as canvas implementations improve. In particular, as browser vendors implement canvas GPU acceleration, some of the outlined performance techniques discussed will likely become less impactful. This will be noted where appropriate. Note that this article does not go into usage of HTML5 canvas. For that, check out these canvas related articles on HTML5Rocks, this chapter on the Dive into HTML5 site or the MDN Canvas. tutorial.

Performance testing #

To address the quickly changing world of HTML5 canvas, JSPerf (jsperf.com) tests verify that every proposed optimization still works. JSPerf is a web application that allows developers to write JavaScript performance tests. Each test focuses on a result that you’re trying to achieve (for example, clearing the canvas), and includes multiple approaches that achieve the same result. JSPerf runs each approach as many times as possible over a short time period and gives a statistically meaningful number of iterations per second. Higher scores are always better! Visitors to a JSPerf performance test page can run the test on their browser, and let JSPerf store the normalized test results on Browserscope (browserscope.org). Because the optimization techniques in this article are backed up by a JSPerf result, you can return to see up-to-date information about whether or not the technique still applies. I’ve written a small helper application that renders these results as graphs, embedded throughout this article.

All of the performance results in this article are keyed on the browser version. This turns out to be a limitation, since we don't know what OS the browser was running on, or even more importantly, whether or not HTML5 canvas was hardware accelerated when the performance test ran. You can find out if Chrome's HTML5 canvas is hardware accelerated by visiting about:gpu in the address bar.

Pre-render to an off-screen canvas #

If you’re re-drawing similar primitives to the screen across multiple frames, as is often the case when writing a game, you can make large performance gains by pre-rendering large parts of the scene. Pre-rendering means using a separate off-screen canvas (or canvases) on which to render temporary images, and then rendering the off-screen canvases back onto the visible one. For example, suppose you’re redrawing Mario running at 60 frames a second. You could either redraw his hat, moustache, and “M” at each frame, or pre-render Mario before running the animation. no pre-rendering:

// canvas, context are defined
function render() {
  drawMario(context);
  requestAnimationFrame(render);
}

pre-rendering:

var m_canvas = document.createElement('canvas');
m_canvas.width = 64;
m_canvas.height = 64;
var m_context = m_canvas.getContext('2d');
drawMario(m_context);

function render() {
  context.drawImage(m_canvas, 0, 0);
  requestAnimationFrame(render);
}

Note the use of requestAnimationFrame, which is discussed in more detail in a later section.

This technique is especially effective when the rendering operation (drawMario in the above example) is expensive. A good example of this is text rendering, which is a very expensive operation.

However, the poor performance of the “pre-rendered loose” test case. When pre-rendering, it’s important to make sure that your temporary canvas fits snugly around the image you are drawing, otherwise the performance gain of off-screen rendering is counterweighted by the performance loss of copying one large canvas onto another (which varies as a function of source target size). A snug canvas in the above test is simply smaller:

can2.width = 100;
can2.height = 40;

Compared to the loose one that yields poorer performance:

can3.width = 300;
can3.height = 100;

Batch canvas calls together #

Since drawing is an expensive operation, it’s more efficient to load the drawing state machine with a long set of commands, and then have it dump them all onto the video buffer.

For example, when drawing multiple lines, it's more efficient to create one path with all the lines in it and draw it with a single draw call. In other words, rather than drawing separate lines:

for (var i = 0; i < points.length - 1; i++) {
  var p1 = points[i];
  var p2 = points[i+1];
  context.beginPath();
  context.moveTo(p1.x, p1.y);
  context.lineTo(p2.x, p2.y);
  context.stroke();
}

We get better performance from drawing a single polyline:

context.beginPath();
for (var i = 0; i < points.length - 1; i++) {
  var p1 = points[i];
  var p2 = points[i+1];
  context.moveTo(p1.x, p1.y);
  context.lineTo(p2.x, p2.y);
}
context.stroke();

This applies to the world of HTML5 canvas as well. When drawing a complex path, for example, it’s better to put all of the points into the path, rather than rendering the segments separately (jsperf).

Note, however, that with Canvas, there’s an important exception to this rule: if the primitives involved in drawing the desired object have small bounding boxes (for example, horizontal and vertical lines), it may actually be more efficient to render them separately (jsperf).

Avoid unnecessary canvas state changes #

The HTML5 canvas element is implemented on top of a state machine that tracks things like fill and stroke styles, as well as previous points that make up the current path. When trying to optimize graphics performance, it’s tempting to focus solely on the graphics rendering. However, manipulating the state machine can also incur a performance overhead. If you use multiple fill colors to render a scene, for example, it’s cheaper to render by color rather than by placement on the canvas. To render a pinstripe pattern, you could render a stripe, change colors, render the next stripe, etc:

for (var i = 0; i < STRIPES; i++) {
  context.fillStyle = (i % 2 ? COLOR1 : COLOR2);
  context.fillRect(i * GAP, 0, GAP, 480);
}

Or render all odd stripes and then all even stripes:

context.fillStyle = COLOR1;
for (var i = 0; i < STRIPES/2; i++) {
  context.fillRect((i*2) * GAP, 0, GAP, 480);
}
context.fillStyle = COLOR2;
for (var i = 0; i < STRIPES/2; i++) {
  context.fillRect((i*2+1) * GAP, 0, GAP, 480);
}

As expected, the interlaced approach is slower because changing the state machine is expensive.

Render screen differences only, not the whole new state #

As one would expect, rendering less on the screen is cheaper than rendering more. If you have only incremental differences between redraws, you can get a significant performance boost by just drawing the difference. In other words, rather than clearing the whole screen before drawing:

context.fillRect(0, 0, canvas.width, canvas.height);

Keep track of the drawn bounding box, and only clear that.

context.fillRect(last.x, last.y, last.width, last.height);

If you are familiar with computer graphics, you might also know this technique as “redraw regions”, where the previously rendered bounding box is saved, and then cleared on each rendering. This technique also applies to pixel-based rendering contexts, as is illustrated by this JavaScript Nintendo emulator talk.

Use multiple layered canvases for complex scenes #

As mentioned before, drawing large images is expensive and should be avoided if possible. In addition to using another canvas for rendering off screen, as illustrated in the pre-rendering section, we can also use canvases layered on top of one another. By using transparency in the foreground canvas, we can rely on the GPU to composite the alphas together at render time. You might set this up as follows, with two absolutely positioned canvases one on top of the other.

<canvas id="bg" width="640" height="480" style="position: absolute; z-index: 0">
</canvas>
<canvas id="fg" width="640" height="480" style="position: absolute; z-index: 1">
</canvas>

The advantage over having just one canvas here, is that when we draw or clear the foreground canvas, we don’t ever modify the background. If your game or multimedia app can be split up into a foreground and background, consider rendering these on separate canvases to get a significant performance boost.

You can often take advantage of imperfect human perception and render the background just once or at a slower speed compared to the foreground (which is likely to occupy most of your user’s attention). For example, you can render the foreground every time you render, but render the background only every Nth frame. Also note that this approach generalizes well for any number of composite canvases if your application works better with a this sort of structure.

Avoid shadowBlur #

Like many other graphics environments, HTML5 canvas allows developers to blur primitives, but this operation can be very expensive:

context.shadowOffsetX = 5;
context.shadowOffsetY = 5;
context.shadowBlur = 4;
context.shadowColor = 'rgba(255, 0, 0, 0.5)';
context.fillRect(20, 20, 150, 100);

Know various ways to clear the canvas #

Since HTML5 canvas is an immediate mode drawing paradigm, the scene needs to be redrawn explicitly at each frame. Because of this, clearing the canvas is a fundamentally important operation for HTML5 canvas apps and games. As mentioned in the Avoid canvas state changes section, clearing the entire canvas is often undesirable, but if you must do it, there are two options: calling context.clearRect(0, 0, width, height) or using a canvas-specific hack to do it: canvas.width = canvas.width;.At the time of writing, clearRect generally outperforms the width reset version, but in some cases using the canvas.width resetting hack is significantly faster in Chrome 14

Be careful with this tip, since it depends heavily on the underlying canvas implementation and is very much subject to change. For more information, see Simon Sarris' article on clearing the canvas.

Avoid floating point coordinates #

HTML5 canvas supports sub-pixel rendering, and there’s no way to turn it off. If you draw with coordinates that are not integers, it automatically uses anti-aliasing to try to to smooth out the lines. Here’s the visual effect, taken from this sub-pixel canvas performance article by Seb Lee-Delisle:

If the smoothed sprite is not the effect you seek, it can be much faster to convert your coordinates to integers using Math.floor or Math.round (jsperf):

To convert your floating point coordinates to integers, you can use several clever techniques, the most performant of which involve adding one half to the target number, and then performing bitwise operations on the result to eliminate the fractional part.

// With a bitwise or.
rounded = (0.5 + somenum) | 0;
// A double bitwise not.
rounded = ~~ (0.5 + somenum);
// Finally, a left bitwise shift.
rounded = (0.5 + somenum) << 0;

The full performance breakdown is here (jsperf).

Note that this sort of optimization should no longer matter once canvas implementations are GPU accelerated which will be able to quickly render non-integer coordinates.

Optimize your animations with `requestAnimationFrame` #

The relatively new requestAnimationFrame API is the recommended way of implementing interactive applications in the browser. Rather than command the browser to render at a particular fixed tick rate, you politely ask the browser to call your rendering routine and get called when the browser is available. As a nice side effect, if the page is not in the foreground, the browser is smart enough not to render. The requestAnimationFrame callback aims for a 60 FPS callback rate but doesn’t guarantee it, so you need to keep track of how much time passed since the last render. This can look something like the following:

var x = 100;
var y = 100;
var lastRender = Date.now();
function render() {
  var delta = Date.now() - lastRender;
  x += delta;
  y += delta;
  context.fillRect(x, y, W, H);
  requestAnimationFrame(render);
}
render();

Note that this use of requestAnimationFrame applies to canvas as well as other rendering technologies such as WebGL. At the time of writing, this API is only available in Chrome, Safari and Firefox, so you should use this shim.

Most mobile canvas implementations are slow #

Let’s talk about mobile. Unfortunately at the time of writing, only iOS 5.0 beta running Safari 5.1 has GPU accelerated mobile canvas implementation. Without GPU acceleration, mobile browsers don’t generally have powerful enough CPUs for modern canvas-based applications. A number of the JSPerf tests described above perform an order of magnitude worse on mobile compared to desktop, greatly restricting the kinds of cross-device apps you can expect to successfully run.

Conclusion #

To recap, this article covered a comprehensive set of useful optimization techniques that will help you develop performant HTML5 canvas-based projects. Now that you’ve learned something new here, go forth and optimize your awesome creations. Or, if you don’t currently have a game or application to optimize, check out Chrome Experiments and Creative JS for inspiration.

References #

Immediate mode vs. retained mode.
Other HTML5Rocks canvas articles.
The Canvas section of Dive into HTML5.
JSPerf lets developers create JS performance tests.
Browserscope stores browser performance data.
JSPerfView, which renders JSPerf tests as charts.
Simon's blog post on clearing the canvas, and his book, HTML5 Unleashed which includes chapters on Canvas performance.
Sebastian's blog post on sub-pixel rendering performance.
Ben's talk about optimizing a JS NES emulator.
The new canvas profiler in the Chrome DevTools.