Chris Wilson on web.dev

Touch and mouse

2013-03-13T00:00:00Z

Introduction #

For close to thirty years, desktop computing experiences have centered around a keyboard and a mouse or trackpad as our main user input devices. Over the last decade, however, smartphones and tablets have brought a new interaction paradigm: touch. With the introduction of touch-enabled Windows 8 machines, and now with the release of the awesome touch-enabled Chromebook Pixel, touch is now becoming part of the expected desktop experience. One of the biggest challenges is building experiences that work not only on touch devices and mouse devices, but also on these devices where the user will use both input methods - sometimes simultaneously!

This article will help you understand how touch capabilities are built into the browser, how you can integrate this new interface mechanism into your existing apps and how touch can play nicely with mouse input.

The State of Touch in the Web Platform #

The iPhone was the first popular platform to have dedicated touch APIs built in to the web browser. Several other browser vendors have created similar API interfaces built to be compatible with the iOS implementation, which is now described by the "Touch Events version 1" specification. Touch events are supported by Chrome and Firefox on desktop, and by Safari on iOS and Chrome and the Android browser on Android, as well as other mobile browsers like the Blackberry browser.

My colleague Boris Smus wrote a great HTML5Rocks tutorial on Touch events that is still a good way to get started if you haven’t looked at Touch events before. In fact, if you haven’t worked with touch events before, go read that article now, before you continue. Go on, I’ll wait.

All done? Now that you have a basic grounding in touch events, the challenge with writing touch-enabled interactions is that the touch interactions can be quite a bit different from mouse (and mouse-emulating trackpad and trackball) events - and although touch interfaces typically try to emulate mice, that emulation isn’t perfect or complete; you really need to work through both interaction styles, and may have to support each interface independently.

Most Importantly: The User May Have Touch And a Mouse #

Many developers have built sites that statically detect whether an environment supports touch events, and then make the assumption that they only need to support touch (and not mouse) events. This is now a faulty assumption - instead, just because touch events are present does not mean the user is primarily using that touch input device. Devices such as the Chromebook Pixel and some Windows 8 laptops now support BOTH Mouse and Touch input methods, and more will in the near future. On these devices, it is quite natural for users to use both the mouse and the touch screen to interact with applications, so "supports touch" is not the same as "doesn’t need mouse support." You can’t think of the problem as "I have to write two different interaction styles and switch between them," you need to think through how both interactions will work together as well as independently. On my Chromebook Pixel, I frequently use the trackpad, but I also reach up and touch the screen - on the same application or page, I do whatever feels most natural at the moment. On the other hand, some touchscreen laptop users will rarely if ever use the touchscreen at all - so the presence of touch input shouldn’t disable or hinder mouse control.

Unfortunately, it can be hard to know if a user’s browser environment supports touch input or not; ideally, a browser on a desktop machine would always indicate support for touch events so a touchscreen display could be attached at any time (e.g. if a touchscreen attached through a KVM becomes available). For all these reasons, your applications shouldn’t attempt to switch between touch and mouse - just support both!

In IE10 on Windows 8, Microsoft introduced a new model called Pointer Events. Pointer Events are a unification of Mouse Events and touch input, as well as other input methods such as pen input. There is work to standardize the Pointer Event model at the W3C, and in the short term, there are libraries out there like PointerEvents and Hand.js that you can use to prototype how Pointer Events could work in your code to remove some of the need to independently support mouse and touch. For really great touch and mouse interaction, you may need to customize your user experience for mouse and touch separately, but unified event handling can make this easier in many scenarios. However, there are significant challenges with this model - namely, it requires supporting redundant input models, and it’s not broadly supported yet - and it will take some time to settle down into a solid, cross-browser standard. In the meantime, the best advice is to support both mouse and touch interaction models. There are a lot of challenges with simultaneously supporting touch and mouse events, so this article explains those challenges and the strategies to overcome them. Additionally, some of this advice is just general "implementing touch" advice, so it may be redundant if you are already used to implementing touch in a mobile context.

Supporting Mouse and Touch Together #

#1 - Clicking and Tapping - the "Natural" Order of Things #

The first problem is that touch interfaces typically try to emulate mouse clicks - obviously, since touch interfaces need to work on applications that have only interacted with mouse events before! You can use this as a shortcut - because "click" events will continue to be fired, whether the user clicked with a mouse or tapped their finger on the screen. However, there are a couple of problems with this shortcut.

First, you have to be careful when designing more advanced touch interactions: when the user uses a mouse it will respond via a click event, but when the user touches the screen both touch and click events will occur. For a single click the order of events is:

touchstart
touchmove
touchend
mouseover
mousemove
mousedown
mouseup
click

This, of course, means that if you are processing touch events like touchstart, you need to make sure that you don’t process the corresponding mousedown and/or click event as well. If you can cancel the touch events (call preventDefault() inside the event handler), then no mouse events will get generated for touch. One of the most important rules of touch handlers is:

However, this also prevents other default browser behavior (like scrolling) - although usually you’re handling the touch event entirely in your handler, and you will WANT to disable the default actions. In general, you’ll either want to handle and cancel all touch events, or avoid having a handler for that event.

Secondly, when a user taps on an element in a web page on a mobile device, pages that haven’t been designed for mobile interaction have a delay of at least 300 milliseconds between the touchstart event and the processing of mouse events (mousedown). It could be done using Chrome, you can turn on "Emulate touch events" in Chrome Developer Tools to help you test touch interfaces on a non-touch system!

This delay is to allow the browser time to determine if the user is performing another gesture - in particular, double-tap zooming. Obviously, this can be problematic in cases where you want to have instantaneous response to a finger touch. There is ongoing work to try to limit the scenarios in which this delay occurs automatically.

	Chrome for Android	Android Browser	Opera Mobile for Android)	Firefox for Android	Safari iOS
Non-scalable viewport	No delay	300ms	300ms	No delay	300ms
No Viewport	300ms	300ms	300ms	300ms	300ms

The first and easiest way to avoid this delay is to "tell" the mobile browser that your page is not going to need zooming - which can be done using a fixed viewport, e.g. by inserting into your page:

<meta name="viewport" content="width=device-width,user-scalable=no">

This isn’t always appropriate, of course - this disables pinch-zooming, which may be required for accessibility reasons, so use it sparingly if at all (if you do disable user scaling, you may want to provide some other way to increase text readability in your application). Also, for Chrome on desktop class devices that support touch, and other browsers on mobile platforms when the page has viewports that are not scalable, this delay does not apply.

#2: Mousemove Events Aren’t Fired by Touch #

It’s important to note at this point that the emulation of mouse events in a touch interface does not typically extend to emulating mousemove events - so if you build a beautiful mouse-driven control that uses mousemove events, it probably won’t work with a touch device unless you specifically add touchmove handlers too.

Browsers typically automatically implement the appropriate interaction for touch interactions on the HTML controls - so, for example, HTML5 Range controls will just work when you use touch interactions. However, if you’ve implemented your own controls, they will likely not work on click-and-drag type interactions; in fact, some commonly used libraries (like jQueryUI) do not yet natively support touch interactions in this way (although for jQueryUI, there are several monkey-patch fixes to this issue). This was one of the first problems I ran into when upgrading my Web Audio Playground application to work with touch - the sliders were jQueryUI-based, so they did not work with click-and-drag interactions. I changed over to HTML5 Range controls, and they worked. Alternately, of course, I could have simply added touchmove handlers to update the sliders, but there’s one problem with that…

#3: Touchmove and MouseMove Aren’t the Same Thing #

A pitfall I've seen a few developers fall into is having touchmove and mousemove handlers call into the same codepaths. The behavior of these events is very close, but subtly different - in particular, touch events always target the element where that touch STARTED, while mouse events target the element currently under the mouse cursor. This is why we have mouseover and mouseout events, but there are no corresponding touchover and touchout events - only touchend.

The most common way this can bite you is if you happen to remove (or relocate) the element that the user started touching. For example, imagine an image carousel with a touch handler on the entire carousel to support custom scrolling behavior. As available images change, you remove some <img> elements and add others. If the user happens to start touching on one of those images and then you remove it, your handler (which is on an ancestor of the img element) will just stop receiving touch events (because they’re being dispatched to a target that’s no longer in the tree) - it'll look like the user is holding their finger in one place even though they may have moved and eventually removed it.

You can of course avoid this problem by avoiding removing elements that have (or have ancestors that have) touch handlers while a touch is active. Alternately, the best guidance is rather than register static touchend/touchmove handlers, wait until you get a touchstart event and then add touchmove/touchend/touchcancel handlers to the target of the touchstart event (and remove them on end/cancel). This way you'll continue to receive events for the touch even if the target element is moved/removed. You can play with this a little here - touch the red box and while holding hit escape to remove it from the DOM.

#4: Touch and :Hover #

The mouse pointer metaphor separated cursor position from actively selecting, and this allowed developers to use hover states to hide and show information that might be pertinent to the users. However, most touch interfaces right now do not detect a finger "hovering" over a target - so providing semantically important information (e.g. providing "what is this control?" popup) based on hovering is a no-no, unless you also give a touch-friendly way to access this information. You need to be careful about how you use hovering to relay information to users.

Interestingly enough, though, the CSS :hover pseudoclass CAN be triggered by touch interfaces in some cases - tapping an element makes it :active while the finger is down, and it also acquires the :hover state. (With Internet Explorer, the :hover is only in effect while the user’s finger is down - other browsers keep the :hover in effect until the next tap or mouse move.) This is a good approach to making pop-out menus work on touch interfaces - a side effect of making an element active is that the :hover state is also applied. For example:

<style>
img ~ .content {
  display:none;
}

img:hover ~ .content {
  display:block;
}
</style>

<img src="/awesome.png">
<div class="content">This is an awesome picture of me</div>

Once another element is tapped the element is no longer active, and the hover state disappears, just as if the user was using a mouse pointer and moved it off the element. You may wish to wrap the content in an <a> element in order to make it a tabstop as well - that way the user can toggle the extra information on a mouse hover or click, a touch tap, or a keypress, with no JavaScript required. I was pleasantly surprised as I began work to make my Web Audio Playground to work well with touch interfaces that my pop-out menus already worked well on touch, because I’d used this kind of structure!

The above method works well for mouse pointer based interfaces, as well as for touch interfaces. This is in contrast to using "title" attributes on hover, which will NOT show up when the element is activated:

<img src="/awesome.png" title="this doesn't show up in touch">

#5: Touch vs. Mouse Precision #

While mice have a conceptual disassociation from reality, it turns out that they are extremely accurate, as the underlying operating system generally tracks exact pixel precision for the cursor. Mobile developers on the other hand have learned that finger touches on a touch screen are not as accurate, mostly because of the size of the surface area of the finger when in contact with the screen (and partly because your fingers obstruct the screen).

Many individuals and companies have done extensive user research on how to design applications and sites that are accommodating of finger based interaction, and many books have been written on the topic. The basic advice is to increase the size of the touch targets by increasing the padding, and reduce the likelihood of incorrect taps by increasing the margin between elements. (Margins are not included in the hit detection handling of touch and click events, while padding is.) One of the primary fixes I had to make to the Web Audio Playground was to increase the sizes of the connection points so they were more easily touched accurately.

Many browser vendors who are handling touch based interfaces have also introduced logic into the browser to help target the correct element when a user touches the screen and reduce the likelihood of incorrect clicks - although this usually only corrects click events, not moves (although Internet Explorer appears to modify mousedown/mousemove/mouseup events as well).

#6: Keep Touch Handlers Contained, or They’ll Jank Your Scroll #

It’s also important to keep touch handlers confined only to the elements where you need them; touch elements can be very high-bandwidth, so it’s important to avoid touch handlers on scrolling elements (as your processing may interfere with browser optimizations for fast jank-free touch scrolling - modern browsers try to scroll on a GPU thread, but this is impossible if they have to check with javascript first to see if each touch event is going to be handled by the app). You can check out an example of this behavior.

One piece of guidance to follow to avoid this problem is to make sure that if you are only handling touch events in a small portion of your UI, you only attach touch handlers there (not, e.g., on the <body> of the page); in short, limit the scope of your touch handlers as much as possible.

#7: Multi-touch #

The final interesting challenge is that although we’ve been referring to it as "Touch" user interface, nearly universally the support is actually for Multi-touch - that is, the APIs provide more than one touch input at a time. As you begin to support touch in your applications, you should consider how multiple touches might affect your application.

If you have been building apps primarily driven by mouse, then you are used to building with at most one cursor point - systems don’t typically support multiple mice cursors. For many applications, you will be just mapping touch events to a single cursor interface, but most of the hardware that we have seen for desktop touch input can handle at least 2 simultaneous inputs, and most new hardware appears to support at least 5 simultaneous inputs. For developing an onscreen piano keyboard, of course, you would want to be able to support multiple simultaneous touch inputs.

The currently implemented W3C Touch APIs have no API to determine how many touch points the hardware supports, so you’ll have to use your best estimation for how many touch points your users will want - or, of course, pay attention to how many touch points you see in practice and adapt. For example, in a piano application, if you never see more than two touch points you may want to add some "chords" UI. The PointerEvents API does have an API to determine the capabilities of the device.

Touching Up #

Hopefully this article has given you some guidance on common challenges in implementing touch alongside mouse interactions. More important than any other advice, of course, is that you need to test your app on mobile, tablet, and combined mouse-and-touch desktop environments. If you don’t have touch+mouse hardware, use Chrome’s "Emulate touch events" to help you test the different scenarios.

It’s not only possible, but relatively easy following these pieces of guidance, to build engaging interactive experiences that work well with touch input, mouse input, and even both styles of interaction at the same time.

A tale of two clocks

2013-01-09T00:00:00Z

Introduction #

One of the biggest challenges in building great audio and music software using the web platform is managing time. Not as in “time to write code”, but as in clock time - one of the least well-understood topics about Web Audio is how to properly work with the audio clock. The Web Audio AudioContext object has a currentTime property that exposes this audio clock.

Particularly for musical applications of web audio - not just writing sequencers and synthesizers, but any rhythmic use of audio events such as drum machines, games, and other applications - it’s very important to have consistent, precise timing of audio events; not just starting and stopping sounds, but also scheduling changes to the sound (like changing frequency or volume). Sometimes it’s desirable to have slightly time-randomized events - for example, in the machine-gun demo in Developing Game Audio with the Web Audio API - but usually, we want to have consistent and accurate timing for musical notes.

We’ve already shown you how to schedule notes using the Web Audio noteOn and noteOff (now renamed start and stop) methods’ time parameter in Getting Started with Web Audio and also in Developing Game Audio with the Web Audio API; however, we haven’t deeply explored more complex scenarios, such as playing long musical sequences or rhythms. To dive into that, first we need a little background on clocks.

The Best of Times - the Web Audio Clock #

The Web Audio API exposes access to the audio subsystem’s hardware clock. This clock is exposed on the AudioContext object through its .currentTime property, as a floating-point number of seconds since the AudioContext was created. This enables this clock (hereafter called the “audio clock”) to be very high-precision; it’s designed to be able to specify alignment at an individual sound sample level, even with a high sample rate. Since there are around 15 decimal digits of precision in a “double”, even if the audio clock has been running for days, it should still have plenty of bits left over to point to a specific sample even at a high sample rate.

The audio clock is used for scheduling parameters and audio events throughout the Web Audio API - for start() and stop(), of course, but also for set*ValueAtTime() methods on AudioParams. This lets us set up very precisely-timed audio events in advance. In fact, it’s tempting to just set up everything in Web Audio as start/stop times - however, in practice there’s a problem with that.

For example, look at this reduced code snippet from our Web Audio Intro, which sets up two bars of an eighth note hi-hat pattern:

for (var bar = 0; bar < 2; bar++) {
  var time = startTime + bar * 8 * eighthNoteTime;

  // Play the hi-hat every eighth note.
  for (var i = 0; i < 8; ++i) {
    playSound(hihat, time + i * eighthNoteTime);
  }

This code will work great. However, if you want to change the tempo in the middle of those two bars - or stop playing before the two bars are up - you’re out of luck. (I’ve seen developers do things like insert a gain node between their pre-scheduled AudioBufferSourceNodes and the output, just so they can mute their own sounds!)

In short, because you will need the flexibility to change tempo or parameters like frequency or gain (or stop scheduling altogether), you don’t want to push too many audio events into the queue - or, more accurately, you don’t want to look ahead too far in time, because you may want to change that scheduling entirely.

The Worst of Times - the JavaScript Clock #

We also have our much-beloved and much-maligned JavaScript clock, represented by Date.now() and setTimeout(). The good side of the JavaScript clock is that it has a couple of very useful call-me-back-later window.setTimeout() and window.setInterval() methods, which let us have the system call our code back at specific times.

The bad side of the JavaScript clock is that it is not very precise. For starters, Date.now() returns a value in milliseconds - an integer number of milliseconds - so the best precision you could hope for is one millisecond. This isn’t incredibly bad in some musical contexts - if your note started a millisecond early or late, you might not even notice - but even at a relatively low audio hardware rate of 44.1kHz, it’s about 44.1 times too slow to use as an audio scheduling clock. Remember that dropping any samples at all can cause audio glitching - so if we’re chaining samples together, we may need them to be precisely sequential.

The up-and-coming High Resolution Time specification actually does give us a much better precision current time through window.performance.now(); it’s even implemented (albeit prefixed) in many current browsers. That can help in some situations, although it’s not really relevant to the worst part of the JavaScript timing APIs.

The worst part of the JavaScript timing APIs are that although Date.now()’s millisecond precision doesn’t sound too bad to live with, the actual callback of timer events in JavaScript (through window.setTimeout() or window.setInterval) can easily be skewed by tens of milliseconds or more by layout, rendering, garbage collection, and XMLHTTPRequest and other callbacks - in short, by any number of things happening on the main execution thread. Remember how I mentioned “audio events” that we could schedule using the Web Audio API? Well, those are all getting processed on a separate thread - so even if the main thread is temporarily stalled doing a complex layout or other long task, the audio will still happen at exactly the times they were told to happen - in fact, even if you’re stopped at a breakpoint in the debugger, the audio thread will continue to play scheduled events!

Using JavaScript setTimeout() in Audio Apps #

Since the main thread can easily get stalled for multiple milliseconds at a time, it is a bad idea to use JavaScript’s setTimeout to directly start playing audio events, because at best your notes will fire within a millisecond or so of when they really should, and at worst they will be delayed for even longer. Worst of all, for what should be rhythmic sequences, they won’t fire at precise intervals as the timing will be sensitive to other things happening on the main JavaScript thread.

To demonstrate this, I wrote a sample “bad” metronome application - that is, one that uses setTimeout directly to schedule notes - and also does a lot of layout. Open this application, click “play”, and then resize the window quickly while it’s playing; you’ll notice that the timing is noticeably jittery (you can hear the rhythm doesn't stay consistent). “But this is contrived!” you say? Well, of course - but that doesn’t mean it doesn’t happen in the real world too. Even relatively static user interface will have timing issues in setTimeout due to relayouts - for example, I noticed that resizing the window quickly will cause the timing on the otherwise excellent WebkitSynth to stutter noticeably. Now picture what will happen when you’re trying to smooth-scroll a full musical score along with your audio, and you can easily imagine how this would affect complex music apps in the real world.

One of the most frequently-asked questions I hear is “Why can’t I get callbacks from audio events?” Although there may be uses for these types of callbacks, they wouldn’t solve the particular problem at hand - it’s important to understand that those events would be fired in the main JavaScript thread, so they would be subject to all the same potential delays as setTimeout; that is, they could be delayed for some unknown and variable number of milliseconds from the precise time they were scheduled before they were actually processed.

So what can we do? Well, the best way to handle timing is to set up a collaboration between JavaScript timers (setTimeout(), setInterval() or requestAnimationFrame() - more on that later) and the audio hardware scheduling.

Obtaining Rock-Solid Timing By Looking Ahead #

Let’s go back to that metronome demo - in fact, I wrote the first version of this simple metronome demo correctly to demonstrate this collaborative scheduling technique. (The code is also available on Github This demo plays beep sounds (generated by an Oscillator) with high precision on every sixteenth, eighth, or quarter note, altering the pitch depending on the beat. It also lets you change the tempo and note interval while it’s playing, or stop the playback at any time - which is a key feature for any real-world rhythmic sequencer. It would be pretty easy to add code to change the sounds this metronome uses on the fly as well.

The way that it manages to allow temp control while maintaining rock-solid timing is a collaboration: a setTimeout timer that fires once every so often, and sets up Web Audio scheduling in the future for individual notes. The setTimeout timer basically just checks to see if any notes are going to need to be scheduled “soon” based on the current tempo, and then schedules them, like so:

setTimeout() and audio event interaction.

In practice, setTimeout() calls may get delayed, so the timing of the scheduling calls may jitter (and skew, depending on how you use setTimeout) over time - although the events in this example fire approximately 50ms apart, they’re frequently slightly more than that (and sometimes much more). However, during each call, we schedule Web Audio events not only for any notes that need to be played now (e.g. the very first note), but also any notes that need to be played between now and the next interval.

In fact, we don’t want to just look ahead by precisely the interval between setTimeout() calls - we also need some scheduling overlap between this timer call and the next, in order to accommodate the worst case main thread behavior - that is, the worst case of garbage collection, layout, rendering or other code happening on the main thread delaying our next timer call. We also need to account for the audio block-scheduling time - that is, how much audio the operating system keeps in its processing buffer - which varies across operating systems and hardware, from low single digits of milliseconds to around 50ms. Each setTimeout() call shown above has a blue interval that shows the entire range of times during which it will attempt to schedule events; for example, the fourth web audio event scheduled in the diagram above might have been played “late” if we’d waited to play it until the next setTimeout call happened, if that setTimeout call was just a few milliseconds later. In real life, the jitter in these times can be even more extreme than that, and this overlap becomes even more important as your app becomes more complex.

The overall lookahead latency affects how tight the tempo control (and other real-time controls) can be; the interval between scheduling calls is a tradeoff between the minimum latency and how often your code impacts the processor. How much the lookahead overlaps with the next interval’s start time is determines how resilient your app will be across different machines, and as it becomes more complex (and layout and garbage collection may take longer). In general, to be resilient to slower machines and operating systems, it’s best to have a large overall lookahead and a reasonably short interval. You can adjust to have shorter overlaps and longer intervals, in order to process fewer callbacks, but at some point, you might start hearing that a large latency causes tempo changes, etc., to not take effect immediately; conversely, if you lessened the lookahead too much, you might start hearing some jittering (as a scheduling call might have to “make up” events that should have happened in the past).

The following timing diagram shows what the metronome demo code actually does: it has a setTimeout interval of 25ms, but a much more resilient overlap: each call will schedule for the next 100ms. The downside of this long lookahead is that tempo changes, etc., will take a tenth of a second to take effect; however, we are much more resilient to interruptions:

scheduling with long overlaps

In fact, you can tell in this example we had a setTimeout interruption in the middle - we should have had a setTimeout callback at approximately 270ms, but it was delayed for some reason until approximately 320ms - 50ms later than it should have been! However, the large lookahead latency kept the timing going with no problem, and we didn’t miss a beat, even though we increased the tempo just before that to playing sixteenth notes at 240bpm (beyond even hardcore drum & bass tempos!)

It’s also possible that each scheduler call might end up scheduling multiple notes - let’s take a look at what happens if we use a longer scheduling interval (250ms lookahead, spaced 200ms apart), and a tempo increase in the middle:

setTimeout() with long lookahead and long intervals

This case demonstrates that each setTimeout() call may end up scheduling multiple audio events - in fact, this metronome is a simple one-note-at-a-time application, but you can easily see how this approach works for a drum machine (where there are frequently multiple simultaneous notes) or a sequencer (that may frequently have non-regular intervals between notes).

In practice, you’ll want to tune your scheduling interval and the lookahead to see how affected it is by layout, garbage collection and other things going on in the main JavaScript execution thread, and to tune the granularity of control over tempo, etc. If you have a very complex layout that happens frequently, for example, you’ll probably want to make the lookahead larger. The main point is that we want the amount of “scheduling ahead” that we’re doing to be large enough to avoid any delays, but not so large as to be create noticeable delay when tweaking the tempo control. Even the case above has a very small overlap, so it won’t be very resilient on a slow machine with a complex web application. A good place to start is probably 100ms of “lookahead” time, with intervals set to 25ms. This may still have problems in complex applications on machines with a lot of audio system latency, in which case you should up the lookahead time; or, if you need tighter control with the loss of some resilience, use a shorter lookahead.

The core code of the scheduling process is in the scheduler() function -

while (nextNoteTime < audioContext.currentTime + scheduleAheadTime ) {
  scheduleNote( current16thNote, nextNoteTime );
  nextNote();
}

This function just gets the current audio hardware time, and compares it against the time for the next note in the sequence - most of the time* in this precise scenario this will do nothing (as there are no metronome “notes” waiting to be scheduled, but when it succeeds it will schedule that note using the Web Audio API, and advance to the next note.

The scheduleNote() function is responsible for actually scheduling the next Web Audio “note” to be played. In this case, I used oscillators to make beeping sounds at different frequencies; you could just as easily create AudioBufferSource nodes and set their buffers to drum sounds, or any other sounds you wish.

currentNoteStartTime = time;

// create an oscillator
var osc = audioContext.createOscillator();
osc.connect( audioContext.destination );

if (! (beatNumber % 16) )         // beat 0 == low pitch
  osc.frequency.value = 220.0;
else if (beatNumber % 4)          // quarter notes = medium pitch
  osc.frequency.value = 440.0;
else                              // other 16th notes = high pitch
  osc.frequency.value = 880.0;
osc.start( time );
osc.stop( time + noteLength );

Once those oscillators are scheduled and connected, this code can forget about them entirely; they will start, then stop, then get garbage-collected automatically.

The nextNote() method is responsible for advancing to the next sixteenth note - that is, setting the nextNoteTime and current16thNote variables to the next note:

function nextNote() {
  // Advance current note and time by a 16th note...
  var secondsPerBeat = 60.0 / tempo;	// picks up the CURRENT tempo value!
  nextNoteTime += 0.25 * secondsPerBeat;	// Add 1/4 of quarter-note beat length to time

  current16thNote++;	// Advance the beat number, wrap to zero
  if (current16thNote == 16) {
    current16thNote = 0;
  }
}

This is pretty straightforward - although it’s important to understand that in this scheduling example, I’m not keeping track of “sequence time” - that is, time since the beginning of starting the metronome. All we have to do is remember when we played the last note, and figure out when the next note is scheduled to play. That way, we can change the tempo (or stop playing) very easily.

This scheduling technique is used by a number of other audio applications on the web - for example, the Web Audio Drum Machine, the very fun Acid Defender game, and even more in-depth audio examples like the Granular Effects demo.

Yet Another Timing System #

Now, as any good musician knows, what every audio application needs is more cowbell - er, more timers. It’s worth mentioning that the right way to do visual display is making use of a THIRD timing system!

Why, why, oh dear heavens why do we need another timing system? Well, this one is synchronized to the visual display - that is, the graphics refresh rate - via the requestAnimationFrame API. For drawing boxes in our metronome example, this may not seem like a really a big deal, but as your graphics get more and more complex, it becomes more and more critical to be using requestAnimationFrame() to sync with the visual refresh rate - and it’s actually just as easy to use from the beginning as using setTimeout()! With very complex synced graphics (e.g. precise display of dense musical notes as they play in a musical notation package), requestAnimationFrame() will give you the smoothest, most precise graphic and audio synchronization.

We kept track of the beats in the queue in the scheduler:

notesInQueue.push( { note: beatNumber, time: time } );

The interaction with our metronome’s current time can be found in the draw() method, which is called (using requestAnimationFrame) whenever the graphics system is ready for an update:

var currentTime = audioContext.currentTime;

while (notesInQueue.length && notesInQueue[0].time < currentTime) {
  currentNote = notesInQueue[0].note;
  notesInQueue.splice(0,1);   // remove note from queue
}

Again, you’ll notice that we are checking the audio system’s clock - because that’s really the one we want to synchronize with, since it will actually play the notes - to see if we should be drawing a new box or not. In fact, we’re not really using the requestAnimationFrame timestamps at all, since we’re using the audio system clock to figure out where we are in time.

Of course, I could have just skipped using a setTimeout() callback altogether, and put my note scheduler into the requestAnimationFrame callback - then we’d be back down to two timers again. That’s okay to do, too, but it’s important to understand that requestAnimationFrame is just a stand-in for setTimeout() in this case; you’ll still want the scheduling accuracy of Web Audio timing for the actual notes.

Conclusion #

I hope this tutorial has been helpful in explaining clocks, timers and how to build great timing into web audio applications. These same techniques can be extrapolated easily to build sequence players, drum machines, and more. Until next time…

Performance tips for JavaScript in V8

2012-10-11T00:00:00Z

Introduction #

Daniel Clifford gave an excellent talk at Google I/O on tips and tricks to improve JavaScript performance in V8. Daniel encouraged us to "demand faster" - to carefully analyze performance differences between C++ and JavaScript, and write code mindfully of how JavaScript works. A summary of the most important points of Daniel's talk are captured in this article, and we'll also keep this article updated as performance guidance changes.

The Most Important Advice #

It's important to put any performance advice into context. Performance advice is addictive, and sometimes focusing on deep advice first can be quite distracting from the real issues. You need to take a holistic view of the performance of your web application - before focusing on these performance tip, you should probably analyze your code with tools like PageSpeed and get your score up. This will help you avoid premature optimization.

The best basic advice for getting good performance in Web applications is:

Be prepared before you have (or notice) a problem
Then, identify and understand the crux of your problem
Finally, fix what matters

In order to accomplish these steps, it can be important to understand how V8 optimizes JS, so you can write code mindful of the JS runtime design. It's also important to learn about the tools available and how they can help you. Daniel goes into some more explanation of how to use the developer tools in his talk; this document just captures some of the most important points of the V8 engine design.

So, on to the V8 tips!

Hidden Classes #

JavaScript has limited compile-time type information: types can be changed at runtime, so it's natural to expect that it is expensive to reason about JS types at compile time. This might lead you to question how JavaScript performance could ever get anywhere close to C++. However, V8 has hidden types created internally for objects at runtime; objects with the same hidden class can then use the same optimized generated code.

For example:

function Point(x, y) {
  this.x = x;
  this.y = y;
}

var p1 = new Point(11, 22);
var p2 = new Point(33, 44);
// At this point, p1 and p2 have a shared hidden class
p2.z = 55;
// warning! p1 and p2 now have different hidden classes!```

Until the object instance p2 has additional member ".z" added, p1 and p2 internally have the same hidden class - so V8 can generate a single version of optimized assembly for JavaScript code that manipulates either p1 or p2. The more you can avoid causing the hidden classes to diverge, the better performance you'll obtain.

Therefore #

Initialize all object members in constructor functions (so the instances don't change type later)
Always initialize object members in the same order

Numbers #

V8 uses tagging to represent values efficiently when types can change. V8 infers from the values that you use what number type you are dealing with. Once V8 has made this inference, it uses tagging to represent values efficiently, because these types can change dynamically. However, there is sometimes a cost to changing these type tags, so it's best to use number types consistently, and in general it is most optimal to use 31-bit signed integers where appropriate.

For example:

var i = 42;  // this is a 31-bit signed integer
var j = 4.2;  // this is a double-precision floating point number```

Therefore #

Prefer numeric values that can be represented as 31-bit signed integers.

Arrays #

In order to handle large and sparse arrays, there are two types of array storage internally:

Fast Elements: linear storage for compact key sets
Dictionary Elements: hash table storage otherwise

It's best not to cause the array storage to flip from one type to another.

Therefore #

Use contiguous keys starting at 0 for Arrays
Don't pre-allocate large Arrays (e.g. > 64K elements) to their maximum size, instead grow as you go
Don't delete elements in arrays, especially numeric arrays
Don't load uninitialized or deleted elements:

for (var b = 0; b &lt; 10; b++) {
  a[0] |= b;  // Oh no!
}
//vs.
a = new Array();
a[0] = 0;
for (var b = 0; b &lt; 10; b++) {
  a[0] |= b;  // Much better! 2x faster.
}

Also, Arrays of doubles are faster - the array's hidden class tracks element types, and arrays containing only doubles are unboxed (which causes a hidden class change).However, careless manipulation of Arrays can cause extra work due to boxing and unboxing - e.g.

var a = new Array();
a[0] = 77;   // Allocates
a[1] = 88;
a[2] = 0.5;   // Allocates, converts
a[3] = true; // Allocates, converts```

is less efficient than:

var a = [77, 88, 0.5, true];

because in the first example the individual assignments are performed one after another, and the assignment of a[2] causes the Array to be converted to an Array of unboxed doubles, but then the assignment of a[3] causes it to be re-converted back to an Array that can contain any values (Numbers or objects). In the second case, the compiler knows the types of all of the elements in the literal, and the hidden class can be determined up front.

Initialize using array literals for small fixed-sized arrays
Preallocate small arrays (<64k) to correct size before using them
Don't store non-numeric values (objects) in numeric arrays
Be careful not to cause re-conversion of small arrays if you do initialize without literals.

JavaScript Compilation #

Although JavaScript is a very dynamic language, and original implementations of it were interpreters, modern JavaScript runtime engines use compilation. V8 (Chrome's JavaScript) has two different Just-In-Time (JIT) compilers, in fact:

The "Full" compiler, which can generate good code for any JavaScript
The Optimizing compiler, which produces great code for most JavaScript, but takes longer to compile.

The Full Compiler #

In V8, the Full compiler runs on all code, and starts executing code as soon as possible, quickly generating good but not great code. This compiler assumes almost nothing about types at compilation time - it expects that types of variables can and will change at runtime. The code generated by the Full compiler uses Inline Caches (ICs) to refine knowledge about types while program runs, improving efficiency on the fly.

The goal of Inline Caches is to handle types efficiently, by caching type-dependent code for operations; when the code runs, it will validate type assumptions first, then use the inline cache to shortcut the operation. However, this means that operations that accept multiple types will be less performant.

Therefore #

Monomorphic use of operations is preferred over polymorphic operations

Operations are monomorphic if the hidden classes of inputs are always the same - otherwise they are polymorphic, meaning some of the arguments can change type across different calls to the operation. For example, the second add() call in this example causes polymorphism:

function add(x, y) {
  return x + y;
}

add(1, 2);      // + in add is monomorphic
add("a", "b");  // + in add becomes polymorphic```

The Optimizing Compiler #

In parallel with the full compiler, V8 re-compiles "hot" functions (that is, functions that are run many times) with an optimizing compiler. This compiler uses type feedback to make the compiled code faster - in fact, it uses the types taken from ICs we just talked about!

In the optimizing compiler, operations get speculatively inlined (directly placed where they are called). This speeds execution (at the cost of memory footprint), but also enables other optimizations. Monomorphic functions and constructors can be inlined entirely (that's another reason why monomorphism is a good idea in V8).

You can log what gets optimized using the standalone "d8" version of the V8 engine:

d8 --trace-opt primes.js

(this logs names of optimized functions to stdout.)

Not all functions can be optimized, however - some features prevent the optimizing compiler from running on a given function (a "bail-out"). In particular, the optimizing compiler currently bails out on functions with try {} catch {} blocks!

Therefore #

Put perf-sensitive code into a nested function if you have try {} catch {} blocks:

function perf_sensitive() {
  // Do performance-sensitive work here
}

try {
  perf_sensitive()
} catch (e) {
  // Handle exceptions here
}

This guidance will probably change in the future, as we enable try/catch blocks in the optimizing compiler. You can examine how the optimizing compiler is bailing out on functions by using the "--trace-opt" option with d8 as above, which gives you more information on which functions were bailed out:

d8 --trace-opt primes.js

De-optimization #

Finally, the optimization performed by this compiler is speculative - sometimes it doesn't work out, and we back off. The process of "deoptimization" throws away optimized code, and resumes execution at the right place in "full" compiler code. Reoptimization might be triggered again later, but for the short term, execution slows down. In particular, causing changes in the hidden classes of variables after the functions have been optimized will cause this deoptimization to occur.

Therefore #

Avoid hidden class changes in functions after they are optimized

You can, as with other optimizations, get a log of functions that V8 had to deoptimize with a logging flag:

d8 --trace-deopt primes.js

Other V8 Tools #

By the way, you can also pass V8 tracing options to Chrome on startup:

"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --js-flags="--trace-opt --trace-deopt"```

In addition to using the developer tools profiling, you can also use d8 to do profiling:

% out/ia32.release/d8 primes.js --prof

This uses the built-in sampling profiler, which takes a sample every millisecond and writes v8.log.

In Summary #

It's important to indentify and understand how the V8 engine works with your code to prepare to build performant JavaScript. Once more, the basic advice is:

Be prepared before you have (or notice) a problem
Then, identify and understand the crux of your problem
Finally, fix what matters

This means you should ensure the problem is in your JavaScript, by using other tools like PageSpeed first; possibly reducing to to pure JavaScript (no DOM) before collecting metrics, and then use those metrics to locate bottlenecks and eliminate the important ones. Hopefully Daniel's talk (and this article) will help you understand better how V8 runs JavaScript - but be sure to focus on optimizing your own algorithms, too!

Chris Wilson on web.dev

Touch and mouse

Introduction #

The State of Touch in the Web Platform #

Most Importantly: The User May Have Touch And a Mouse #

Supporting Mouse and Touch Together #

#1 - Clicking and Tapping - the "Natural" Order of Things #

#2: Mousemove Events Aren’t Fired by Touch #

#3: Touchmove and MouseMove Aren’t the Same Thing #

#4: Touch and :Hover #

#5: Touch vs. Mouse Precision #

#6: Keep Touch Handlers Contained, or They’ll Jank Your Scroll #

#7: Multi-touch #

Touching Up #

A tale of two clocks

Introduction #

The Best of Times - the Web Audio Clock #

The Worst of Times - the JavaScript Clock #

Using JavaScript setTimeout() in Audio Apps #

Obtaining Rock-Solid Timing By Looking Ahead #

Yet Another Timing System #

Conclusion #

Performance tips for JavaScript in V8

Introduction #

The Most Important Advice #

Hidden Classes #

Therefore #

Numbers #

Therefore #

Arrays #

Therefore #

JavaScript Compilation #

The Full Compiler #

Therefore #

The Optimizing Compiler #

Therefore #

De-optimization #

Therefore #

Other V8 Tools #

In Summary #

References #