Ilmari Heikkinen on web.dev

Making of the World Wonders 3D Globe

2012-07-26T00:00:00Z

Introduction to the World Wonders 3D globe #

If you've viewed the recently launched Google World Wonders site on a WebGL-capable browser, you might have spotted a fancy spinning globe at the bottom of the screen. This article lets you in on how the globe works and what we used to build it.

To give you a quick overview, the World Wonders globe is a heavily tweaked version of the WebGL Globe by the Google Data Arts Team. We took the original globe, stripped out the bar graph bits, changed the shaders, added fancy clickable HTML markers and Natural Earth continent geometry from Mozilla's GlobeTweeter demo (big thanks to Cedric Pinson!) All to make a nice animated globe that matches the site's color scheme and adds an extra layer of sophistication to the site.

The design brief for the globe was to have a nice-looking animated map with clickable markers placed on top of World Heritage Sites. With that in mind, I started looking for something suitable. The first thing that came to mind was the WebGL Globe built by the Google Data Arts Team. It's a globe and it looks cool. What else do you need, eh?

Setting up the WebGL Globe #

The first step in making the globe widget was to download the WebGL Globe and get it up and running. The WebGL Globe is online at Google Code, and it's simple to download and run. Download and extract the zip, cd into it and run a basic webserver: python -m SimpleHTTPServer. (Do note, this doesn't have UTF-8 on by default; you can use this.) Now if you navigate to http://localhost:8000/globe/globe.html you should see the WebGL Globe.

With the WebGL Globe up and running it was time to cut off all the unneeded parts. I edited the HTML to excise the UI bits and removed the globe bar graph setup stuff from the globe initialization function. At the end of that process, I had a very barebones WebGL Globe on my screen. You can spin it around and it looks cool, but that's about it.

To cut the unneeded stuff, I deleted all the UI elements from the globe's index.html and edited the initialization script in to look like this:

if(!Detector.webgl){
  Detector.addGetWebGLMessage();
} else {
  var container = document.getElementById('container');
  var globe = new DAT.Globe(container);
  globe.animate();
}

Adding the continent geometry #

We wanted to have the camera close to the globe surface, but when we tested the globe zoomed in the lack of texture resolution became apparent. Zoomed in, the WebGL Globe's texture becomes blocky and blurry. We could have used a larger image, but that would make the globe slower to download and run, so we opted to go with a vector representation of the landmasses and borders.

For the landmass geometry, I turned to the open source GlobeTweeter demo and loaded the 3D model in that to Three.js. With the model loaded and rendering, it was time to start polishing the look for the globe. First problem was that the globe landmass model wasn't spherical enough to be flush with the WebGL Globe, so I ended up writing a quick mesh splitting algorithm that made the landmass model more spherical.

With a spherical landmass model, I was able to place it just slightly offset from the globe surface, creating floating continents outlined with a black 2px line below them for a shadow of sorts. I also experimented with neon-colored outlines to make for a sort of Tron-like look.

With the globe and the landmasses rendering, I started to experiment with different looks for the globe. As we wanted to go with a understated monochrome look, I stuck with a greyscale globe and landmasses. In addition to the aforementioned neon outlines, I tried out a dark globe with dark landmasses on a light background, which actually looks pretty cool. But it was too low-contrast to be easily readable and it didn't fit the feel of the project so I scrapped that.

Another early thought I had for the globe look was to make it look like glazed porcelain. That one I didn't manage to try out as I didn't manage to write a shader to do the porcelain look (visual material editor would be nice). The closest thing I tried was this white glowing globe with black landmasses. It's kinda neat but too high-contrast. And it doesn't look super nice. So another one for the scrapheap.

The shaders in the black and white globes are using a sort of bogus diffuse backlit lighting. The lightness of the globe depends on the distance of the surface normal to the screen plane. So pixels at the middle of the globe that are pointing at the screen are dark and pixels at the edges of the globe are light. Combined with a light background you get a look where the globe is reflecting the diffuse bright background, creating a classy showroom look. The black globe is also using the WebGL Globe texture as a gloss map, so that the continental shelves (shallow water areas) look shiny compared to the other parts of the globe.

Here's what the ocean shader for the black globe looks like. Very basic vertex shader and a hacky "oh that looks kinda neat tweak tweak" fragment shader.

    'ocean' : {
      uniforms: {
        'texture': { type: 't', value: 0, texture: null }
      },
      vertexShader: [
        'varying vec3 vNormal;',
        'varying vec2 vUv;',
        'void main() {',
          'gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );',
          'vNormal = normalize( normalMatrix * normal );',
          'vUv = uv;',
        '}'
      ].join('\n'),
      fragmentShader: [
        'uniform sampler2D texture;',
        'varying vec3 vNormal;',
        'varying vec2 vUv;',
        'void main() {',
          'vec3 diffuse = texture2D( texture, vUv ).xyz;',
          'float intensity = pow(1.05 - dot( vNormal, vec3( 0.0, 0.0, 1.0 ) ), 4.0);',
          'float i = 0.8-pow(clamp(dot( vNormal, vec3( 0, 0, 1.0 )), 0.0, 1.0), 1.5);',
          'vec3 atmosphere = vec3( 1.0, 1.0, 1.0 ) * intensity;',
          'float d = clamp(pow(max(0.0,(diffuse.r-0.062)*10.0), 2.0)*5.0, 0.0, 1.0);',
          'gl_FragColor = vec4( (d*vec3(i)) + ((1.0-d)*diffuse) + atmosphere, 1.0 );',
        '}'
      ].join('\n')
    }

In the end we went with a dark globe with light-grey landmasses lit from above. It was closest to the design brief and looked nice and readable. In addition, having the globe be a bit low-contrast makes the markers and the rest of the content stand out more in comparison. The version below is using completely black oceans, whereas the production version has dark gray oceans and slightly different markers.

Creating the markers with CSS #

Speaking of markers, with the globe and landmasses working, I started work on the placemarkers. I decided to go with CSS-style HTML elements for the markers, to make it easier to make and style the markers, and to potentially reuse the markers in the 2D map the team was working on. At the time I also didn't know of an easy way to make the WebGL markers clickable and didn't want to write extra code for loading / creating the marker models. In hindsight, the CSS markers worked well but had a tendency to occasionally run into performance issues when browser compositors and renderers were in periods of flux. From a performance point of view, doing the markers in WebGL would've been a better option. Then again, the CSS markers saved a good deal of dev time.

The CSS markers consist of a couple of divs absolute-positioned with the CSS transform property. The background of the markers is a CSS gradient and the triangle part of the marker is a rotated div. The markers have a small drop shadow to pop them from the background. The biggest problem with the markers was to make them perform well enough. Sad as it may sound, drawing a few dozen divs that move around and change their z-index on every frame is a pretty good way to trigger all sorts of browser rendering pitfalls.

The way the markers are synchronized with the 3D scene is not too complicated. Each marker has a corresponding Object3D in the Three.js scene, which are used to track the markers. To get screen space coordinates, I take the Three.js matrices for the globe and the marker, and multiply a zero vector with those. From that I get the scene position of the marker. To get the screen position of the marker, I project the scene position through the camera. The resulting projected vector has the screen space coordinates for the marker, ready for use in CSS.

var mat = new THREE.Matrix4();
var v = new THREE.Vector3();

for (var i=0; i<locations.length; i++) {
  mat.copy(scene.matrix);
  mat.multiplySelf(locations[i].point.matrix);
  v.set(0,0,0);
  mat.multiplyVector3(v);
  projector.projectVector(v, camera);
  var x = w * (v.x + 1) / 2; // Screen coords are between -1 .. 1, so we transform them to pixels.
  var y = h - h * (v.y + 1) / 2; // The y coordinate is flipped in WebGL.
  var z = v.z;
}

In the end, the fastest approach was to use CSS transforms for moving the markers, not use opacity fading as it was triggering a slow path on Firefox and keeping all the markers in the DOM, not removing them when they went behind the globe. We also experimented with using 3D transforms instead of z-indexes, but for some reason it didn't work right in the app (but it did work in a reduced test case, go figure), and we were a few days from the launch at that point so had to leave that part to post-launch maintenance.

When you click on a marker it expands into a list of clickable placenames. This is all normal HTML DOM stuff, so it was super easy to write. All the links and text rendering just work with no extra work on our part.

Squeezing the filesize #

With the demo working and hooked up to the rest of the World Wonders site, there was still one big issue to solve. The JSON-format mesh for the globe landmasses was about 3 megs in size. Not good for the front page of a showcase site. The good thing was that compressing the mesh with gzip brought it down to 350 kB. But hey, 350 kB is still a bit big. A couple emails later we managed to recruit Won Chun -- who worked on compressing the huge Google Body meshes -- to give us a hand at compressing the mesh. He squeezed the mesh down from a big flat list of triangles given as JSON coordinates to compressed 11-bit coords with indexed triangles and got the file size down to 95 kB gzipped.

Using compressed meshes does not only save bandwidth, but the meshes are also faster to parse. Turning 3 megs of stringified numbers into native numbers takes a good deal more work than parsing a hundred kB of binary data. And the resulting 250 kB size reduction for the page is very nifty as well as it gets the initial load time below a second on a 2 Mbps connection. Faster and smaller, awesomesauce!

At the same time, I was doodling around with loading the original Natural Earth Shapefiles from which the GlobeTweeter mesh is derived. I managed to load the Shapefiles but rendering them as flat landmasses requires triangulating them (with holes for lakes, natch.) I got the shapes triangulated using THREE.js utils but not the holes. And the resulting meshes had very long edges, which required splitting the mesh down to smaller tris. Long story short, I didn't manage to get it working in time, but the neat thing was that the further-compressed Shapefile format would've gotten you a 8 kB landmass model. Oh well, maybe next time.

Future work #

One thing that could use a bit of extra work are making the marker animations nicer. Now when they go over the horizon, the effect is a bit tacky. Additionally, having a cool animation for the marker opening would be nice.

Performance-wise the two things lacking are optimizing the mesh splitting algorithm and making the markers faster. Apart from that, things are dandy. Hurrah!

Summary #

In this article I described how we built the 3D globe for the Google World Wonders project. I hope you enjoyed the examples and will try out building your own custom globe widget.

References #

Typed arrays - Binary data in the browser

2012-07-20T00:00:00Z

Introduction #

Typed Arrays are a relatively recent addition to browsers, born out of the need to have an efficient way to handle binary data in WebGL. A Typed Array is a slab of memory with a typed view into it, much like how arrays work in C. Because a Typed Array is backed by raw memory, the JavaScript engine can pass the memory directly to native libraries without having to painstakingly convert the data to a native representation. As a result, typed arrays perform a lot better than JavaScript arrays for passing data to WebGL and other APIs dealing with binary data.

Typed array views act like single-type arrays to a segment of an ArrayBuffer. There are views for all the usual numeric types, with self-descriptive names like Float32Array, Float64Array, Int32Array and Uint8Array. There's also a special view which has replaced the pixel array type in Canvas's ImageData: Uint8ClampedArray.

DataView is the second type of view and it is meant for handling heterogeneous data. Instead of having an array-like API, the DataView object provides you a get/set API to read and write arbitrary data types at arbitrary byte offsets. DataView works great for reading and writing file headers and other such struct-like data.

Basics of using Typed Arrays #

Typed array views #

To use Typed Arrays, you need to create an ArrayBuffer and a view to it. The easiest way is to create a typed array view of the desired size and type.

// Typed array views work pretty much like normal arrays.
var f64a = new Float64Array(8);
f64a[0] = 10;
f64a[1] = 20;
f64a[2] = f64a[0] + f64a[1];

There are several different types of typed array views. They all share the same API, so once you know how to use one, you pretty much know how to use them all. I'm going to create one of each currently existing typed array views in the next example.

// Floating point arrays.
var f64 = new Float64Array(8);
var f32 = new Float32Array(16);

// Signed integer arrays.
var i32 = new Int32Array(16);
var i16 = new Int16Array(32);
var i8 = new Int8Array(64);

// Unsigned integer arrays.
var u32 = new Uint32Array(16);
var u16 = new Uint16Array(32);
var u8 = new Uint8Array(64);
var pixels = new Uint8ClampedArray(64);

The last one is a bit special, it clamps input values between 0 and 255. This is especially handy for Canvas image processing algorithms since now you don’t have to manually clamp your image processing math to avoid overflowing the 8-bit range.

For example, here’s how you’d apply a gamma factor to an image stored in a Uint8Array. Not very pretty:

u8[i] = Math.min(255, Math.max(0, u8[i] * gamma));

With Uint8ClampedArray you can skip the manual clamping:

pixels[i] *= gamma;

The other way to create typed array views is to create an ArrayBuffer first and then create views that point to it. The APIs that get you external data usually deal in ArrayBuffers, so this is the way you get a typed array view to those.

var ab = new ArrayBuffer(256); // 256-byte ArrayBuffer.
var faFull = new Uint8Array(ab);
var faFirstHalf = new Uint8Array(ab, 0, 128);
var faThirdQuarter = new Uint8Array(ab, 128, 64);
var faRest = new Uint8Array(ab, 192);

You can also have several views to the same ArrayBuffer.

var fa = new Float32Array(64);
var ba = new Uint8Array(fa.buffer, 0, Float32Array.BYTES_PER_ELEMENT); // First float of fa.

To copy a typed array to another typed array, the fastest way is to use the typed array set method. For a memcpy-like use, create Uint8Arrays to the buffers of the views and use set to copy the data over.

function memcpy(dst, dstOffset, src, srcOffset, length) {
  var dstU8 = new Uint8Array(dst, dstOffset, length);
  var srcU8 = new Uint8Array(src, srcOffset, length);
  dstU8.set(srcU8);
};

DataView #

To use ArrayBuffers that contain data with heterogenous types, the easiest way is to use a DataView to the buffer. Suppose we have a file format that has a header with an 8-bit unsigned int followed by two 16-bit ints, followed by a payload array of 32-bit floats. Reading this back with typed array views is doable but a bit of a pain. With a DataView we can read the header and use a typed array view for the float array.

var dv = new DataView(buffer);
var vector_length = dv.getUint8(0);
var width = dv.getUint16(1); // 0+uint8 = 1 bytes offset
var height = dv.getUint16(3); // 0+uint8+uint16 = 3 bytes offset
var vectors = new Float32Array(width*height*vector_length);
for (var i=0, off=5; i<vectors.length; i++, off+=4) {
  vectors[i] = dv.getFloat32(off);
}

In the above example, all the values I read are big-endian. If the values in the buffer are little-endian, you can pass the optional littleEndian parameter to the getter:

...
var width = dv.getUint16(1, true);
var height = dv.getUint16(3, true);
...
vectors[i] = dv.getFloat32(off, true);
...

Note that typed array views are always in the native byte order. This is to make them fast. You should use a DataView to read and write data where endianness is going to be an issue.

The DataView also has methods for writing values to buffers. These setters are named in the same fashion as the getters, "set" followed by the data type.

dv.setInt32(0, 25, false); // set big-endian int32 at byte offset 0 to 25
dv.setInt32(4, 25); // set big-endian int32 at byte offset 4 to 25
dv.setFloat32(8, 2.5, true); // set little-endian float32 at byte offset 8 to 2.5

A discussion of endianness #

Endianness, or byte order, is the order in which multi-byte numbers are stored in the computer's memory. The term big-endian describes a CPU architecture which stores the most significant byte first; little-endian, the least significant byte first. Which endianness is used in a given CPU architecture is completely arbitrary; there are good reasons to choose either one. In fact, some CPUs can be configured to support both big-endian and little-endian data.

Why do you need to be concerned about endianness? The reason is simple. When reading or writing data from the disk or the network, the endianness of the data must be specified. This ensures that the data is interpreted properly, regardless of the endianness of the CPU that is working with it. In our increasingly networked world, it is essential to properly support all kinds of devices, big- or little-endian, that might need to work with binary data coming from servers or other peers on the network.

The DataView interface is specifically designed to read and write data to and from files and the network. DataView operates upon data with a specified endianness. The endianness, big or little, must be specified with every access of every value, ensuring that you get consistent and correct results when reading or writing binary data, no matter what the endianness of the CPU on which the browser is running.

Typically, when your application reads binary data from a server, you’ll need to scan through it once in order to convert it into the data structures your application uses internally. DataView should be used during this phase. It’s not a good idea to use the multi-byte typed array views (Int16Array, Uint16Array, etc.) directly with data fetched via XMLHttpRequest, FileReader, or any other input/output API, because the typed array views use the CPU’s native endianness. More on this later.

Let’s look at a couple of simple examples. The Windows BMP file format used to be the standard format for storing images in the early days of Windows. The documentation linked above clearly indicates that all of the integer values in the file are stored in little-endian format. Here is a snippet of code which parses the beginning of the BMP header using the DataStream.js library which accompanies this article:

function parseBMP(arrayBuffer) {
  var stream = new DataStream(arrayBuffer, 0,
    DataStream.LITTLE_ENDIAN);
  var header = stream.readUint8Array(2);
  var fileSize = stream.readUint32();
  // Skip the next two 16-bit integers
  stream.readUint16();
  stream.readUint16();
  var pixelOffset = stream.readUint32();
  // Now parse the DIB header
  var dibHeaderSize = stream.readUint32();
  var imageWidth = stream.readInt32();
  var imageHeight = stream.readInt32();
  // ...
}

Here’s another example, this one from the High Dynamic Range rendering demo in the WebGL samples project. This demo downloads raw, little-endian floating-point data representing high dynamic range textures, and needs to upload it to WebGL. Here is the snippet of code which properly interprets the floating-point values on all CPU architectures. Assume the variable “arrayBuffer” is an ArrayBuffer which has just been downloaded from the server via XMLHttpRequest:

var arrayBuffer = ...;
var data = new DataView(arrayBuffer);
var tempArray = new Float32Array(
  data.byteLength / Float32Array.BYTES_PER_ELEMENT);
var len = tempArray.length;
// Incoming data is raw floating point values
// with little-endian byte ordering.
for (var jj = 0; jj < len; ++jj) {
  tempArray[jj] =
    data.getFloat32(jj * Float32Array.BYTES_PER_ELEMENT, true);
}
gl.texImage2D(...other arguments...,
  gl.RGB, gl.FLOAT, tempArray);

The rule of thumb is: upon receiving binary data from the web server, make one pass over it with a DataView. Read the individual numeric values and store them in some other data structure, either a JavaScript object (for small amounts of structured data) or a typed array view (for large blocks of data). This will ensure that your code works correctly on all kinds of CPUs. Also use DataView to write data to a file or the network, and make sure to appropriately specify the littleEndian argument to the various set methods in order to produce the file format you are creating or using.

Remember, all data that goes over the network implicitly has a format and an endianness (at least, for any multi-byte values). Make sure to clearly define and document the format of all data your application sends over the network.

Browser APIs that use Typed Arrays #

I'm going to give you a brief overview of the different browser APIs that are currently using Typed Arrays. The current crop includes WebGL, Canvas, Web Audio API, XMLHttpRequests, WebSockets, Web Workers, Media Source API and File APIs. From the list of APIs you can see that Typed Arrays are well suited for performance-sensitive multimedia work as well as passing data around in an efficient fashion.

WebGL #

The first use of Typed Arrays was in WebGL, where it is used to pass around buffer data and image data. To set the contents of a WebGL buffer object, you use the gl.bufferData() call with a Typed Array.

var floatArray = new Float32Array([1,2,3,4,5,6,7,8]);
gl.bufferData(gl.ARRAY_BUFFER, floatArray);

Typed Arrays are also used to pass around texture data. Here's a basic example of passing in texture content using a Typed Array.

var pixels = new Uint8Array(16*16*4); // 16x16 RGBA image
gl.texImage2D(
  gl.TEXTURE_2D, // target
  0, // mip level
  gl.RGBA, // internal format
  16, 16, // width and height
  0, // border
  gl.RGBA, //format
  gl.UNSIGNED_BYTE, // type
  pixels // texture data
);

You also need Typed Arrays to read pixels from the WebGL context.

var pixels = new Uint8Array(320*240*4); // 320x240 RGBA image
gl.readPixels(0, 0, 320, 240, gl.RGBA, gl.UNSIGNED_BYTE, pixels);

Canvas 2D #

Recently the Canvas ImageData object was made to work with the Typed Arrays spec. Now you can get a Typed Arrays representation of the pixels on a canvas element. This is helpful since now you can also create and edit canvas pixel arrays without having to fiddle around with the canvas element.

var imageData = ctx.getImageData(0,0, 200, 100);
var typedArray = imageData.data // data is a Uint8ClampedArray

XMLHttpRequest2 #

XMLHttpRequest got a Typed Array boost and now you can receive a Typed Array response instead of having to parse a JavaScript string into a Typed Array. This is really neat for passing fetched data directly to multimedia APIs and for parsing binary files fetched from the network.

All you have to do is set the responseType of the XMLHttpRequest object to 'arraybuffer'.

xhr.responseType = 'arraybuffer';

Recall that you must be aware of endianness issues when downloading data from the network! See the section on endianness above.

File APIs #

The FileReader can read file contents as an ArrayBuffer. You can then attach typed array views and DataViews to the buffer to manipulate its contents.

reader.readAsArrayBuffer(file);

You should keep endianness in mind here as well. Check out the endianness section for details.

Transferable objects #

Transferable objects in postMessage make passing binary data to other windows and Web Workers a great deal faster. When you send an object to a Worker as a Transferable, the object becomes inaccessible in the sending thread and the receiving Worker gets ownership of the object. This allows for a highly optimized implementation where the sent data is not copied, just the ownership of the Typed Array is transferred to the receiver.

To use Transferable objects with Web Workers, you need to use the webkitPostMessage method on the worker. The webkitPostMessage method works just like postMessage, but it takes two arguments instead of just one. The added second argument is an array of objects you wish to transfer to the worker.

worker.webkitPostMessage(oneGBTypedArray, [oneGBTypedArray]);

To get the objects back from the worker, the worker can pass them back to the main thread in the same fashion.

webkitPostMessage({results: grand, youCanHaveThisBack: oneGBTypedArray}, [oneGBTypedArray]);

Zero copies, woo!

Media Source API #

Recently, the media elements also got some Typed Array goodness in the form of the Media Source API. You can directly pass a Typed Array containing video data to a video element using webkitSourceAppend. This makes the video element append the video data after the existing video. SourceAppend is awesome for doing interstitials, playlists, streaming and other uses where you might want to play several videos using a single video element.

video.webkitSourceAppend(uint8Array);

Binary WebSockets #

You can also use Typed Arrays with WebSockets to avoid having to stringify all your data. Great for writing efficient protocols and minimizing network traffic.

socket.binaryType = 'arraybuffer';

Whew! That wraps up the API review. Let's move onto looking at third-party libraries for handling Typed Arrays.

Third-party libraries #

jDataView #

jDataView implements a DataView shim for all browsers. DataView used to be a WebKit-only feature, but now it's supported by most other browsers. The Mozilla developer team is in the process of landing a patch to enable DataView on Firefox as well.

Eric Bidelman on the Chrome Developer Relations team wrote a small MP3 ID3 tag reader example that uses jDataView. Here's a usage example from the blog post:

var dv = new jDataView(arraybuffer);

// "TAG" starts at byte -128 from EOF.
// See http://en.wikipedia.org/wiki/ID3
if (dv.getString(3, dv.byteLength - 128) == 'TAG') {
  var title = dv.getString(30, dv.tell());
  var artist = dv.getString(30, dv.tell());
  var album = dv.getString(30, dv.tell());
  var year = dv.getString(4, dv.tell());
} else {
  // no ID3v1 data found.
}

stringencoding #

Working with strings in Typed Arrays is a bit of a pain at the moment, but there’s the stringencoding library that helps there. Stringencoding implements the proposed Typed Array string encoding spec, so it’s also a good way to get a feel for what’s coming.

Here's a basic usage example of stringencoding:

var uint8array = new TextEncoder(encoding).encode(string);
var string = new TextDecoder(encoding).decode(uint8array);

BitView.js #

I have written a small bit manipulation library for Typed Arrays called BitView.js. As the name says, it works much like the DataView, except it works with bits. With BitView you can get and set the value of a bit at a given bit offset in an ArrayBuffer. BitView also has methods for storing and loading 6-bit and 12-bit ints at arbitrary bit offsets.

12-bit ints are nice for working with screen coordinates, as displays tend to have fewer than 4096 pixels along the longer dimension. By using 12-bit ints instead of 32-bit ints, you get a 62% size reduction. For a more extreme example, I was working with Shapefiles that use 64-bit floats for the coordinates, but I didn't need the precision because the model was only going to be shown at screen size. Switching over to 12-bit base coordinates with 6-bit deltas to encode changes from the previous coordinate brought the filesize down to a tenth. You can see the demo of that at here.

Here's an example of using BitView.js:

var bv = new BitView(arrayBuffer);
bv.setBit(4, 1); // Set fourth bit of arrayBuffer to 1.
bv.getBit(17); // Get 17th bit of arrayBuffer.

bv.getBit(50*8 + 3); // Get third bit of 50th byte in arrayBuffer.

bv.setInt6(3, 18); // Write 18 as a 6-bit int to bit position 3 in arrayBuffer.
bv.getInt12(9); // Read a 12-bit int from bit position 9 in arrayBuffer.

DataStream.js #

One of the most exciting things about typed arrays is how they make it easier to deal with binary files in JavaScript. Instead of parsing a string character by character and manually converting the characters into binary numbers and such, you can now get an ArrayBuffer with XMLHttpRequest and directly process it using a DataView. This makes it easy to e.g. load in an MP3 file and read the metadata tags for use in your audio player. Or load in a shapefile and turn it into a WebGL model. Or read the EXIF tags off a JPEG and show them in your slideshow app.

The problem with ArrayBuffer XHRs is that reading struct-like data from the buffer is a bit of a pain. DataView is good for reading a few numbers at a time in an endian-safe fashion, typed array views are good for reading arrays of element-size-aligned native endian numbers. What we felt missing is a way to read in arrays and structs of data in a convenient endian-safe fashion. Enter DataStream.js.

DataStream.js is a Typed Arrays library that read and writes scalars, strings, arrays and structs of data from ArrayBuffers in a file-like fashion.

Example of reading in an array of floats from an ArrayBuffer:

// without DataStream.js
var dv = new DataView(buffer);
var f32 = new Float32Array(buffer.byteLength / 4);
var littleEndian = true;
for (var i = 0; i<f32.length; i++) {
  f32[i] = dv.getFloat32(i*4, littleEndian);
}

// with DataStream.js
var ds = new DataStream(buffer);
ds.endianness = DataStream.LITTLE_ENDIAN;
var f32 = ds.readFloat32Array(ds.byteLength / 4);

Where DataStream.js gets really useful is in reading more complex data. Suppose you have a method that reads in JPEG markers:

// without DataStream.js
var dv = new DataView(buffer);
var objs = [];
for (var i=0; i<buffer.byteLength;) {
  var obj = {};
  obj.tag = dv.getUint16(i);
  i += 2;
  obj.length = dv.getUint16(i);
  i += 2;
  obj.data = new Uint8Array(obj.length - 2);
  for (var j=0; j<obj.data.length; j++,i++) {
    obj.data[j] = dv.getUint8(i);
  }
  objs.push(obj);
}

// with DataStream.js
var ds = new DataStream(buffer);
ds.endianness = ds.BIG_ENDIAN;
var objs = [];
while (!ds.isEof()) {
  var obj = {};
  obj.tag = ds.readUint16();
  obj.length = ds.readUint16();
  obj.data = ds.readUint8Array(obj.length - 2);
  objs.push(obj);
}

Or use the DataStream.readStruct method to read in structs of data. The readStruct method takes in a struct definition array that contains the types of the struct members. It’s got callback functions for handling complex types and handles arrays of data and nested structs as well:

// with DataStream.readStruct
ds.readStruct([
  'objs', ['[]', [ // objs: array of tag,length,data structs
    'tag', 'uint16',
    'length', 'uint16',
    'data', ['[]', 'uint8', function(s,ds){ return s.length - 2; }], // get length with a function
  '*'] // read in as many struct as there are
]);

As you can see, the struct definition is a flat array of [name, type]-pairs. Nested structs are done by having an array for the type. Arrays are defined by using a three-element array where the second element is the array element type and the third element is the array length (either as a number, as a reference to previously read field or as a callback function). The first element of the array definition is unused.

The possible values for the type are as follows:

Number types

Unsuffixed number types use DataStream endianness.
To explicitly specify endianness, suffix the type with
'le' for little-endian or 'be' for big-endian,
e.g. 'int32be' for big-endian int32.

  'uint8' -- 8-bit unsigned int
  'uint16' -- 16-bit unsigned int
  'uint32' -- 32-bit unsigned int
  'int8' -- 8-bit int
  'int16' -- 16-bit int
  'int32' -- 32-bit int
  'float32' -- 32-bit float
  'float64' -- 64-bit float

String types

  'cstring' -- ASCII string terminated by a zero byte.
  'string:N' -- ASCII string of length N.
  'string,CHARSET:N' -- String of byteLength N encoded with given CHARSET.
  'u16string:N' -- UCS-2 string of length N in DataStream endianness.
  'u16stringle:N' -- UCS-2 string of length N in little-endian.
  'u16stringbe:N' -- UCS-2 string of length N in big-endian.

Complex types

  [name, type, name_2, type_2, ..., name_N, type_N] -- Struct

  function(dataStream, struct) {} -- Callback function to read and return data.

  {get: function(dataStream, struct) {}, set: function(dataStream, struct) {}}
  -- Getter/setter functions to reading and writing data. Handy for using the
     same struct definition for both reading and writing.

  ['', type, length] -- Array of given type and length. The length can be either
                        a number, a string that references a previously-read
                        field, or a callback function(struct, dataStream, type){}.
                        If length is set to '*', elements are read from the
                        DataStream until a read fails.

You can see a live example of reading in JPEG metadata here. The demo's using DataStream.js for reading the tag-level structure of the JPEG file (along with some EXIF parsing), and jpg.js for decoding and displaying the JPEG image in JavaScript.

History of Typed Arrays #

Typed Arrays got their start in the early implementation stage of WebGL, when we found that passing JavaScript arrays to the graphics driver was causing performance problems. With JavaScript arrays, the WebGL binding had to allocate a native array and fill it by walking over the JavaScript array and cast every JavaScript object in the array to the required native type.

To fix the data conversion bottleneck, Mozilla's Vladimir Vukicevic wrote CanvasFloatArray: a C-style float array with a JavaScript interface. Now you could edit the CanvasFloatArray in JavaScript and pass it directly to WebGL without having to do any extra work in the binding. In further iterations, CanvasFloatArray was renamed to WebGLFloatArray, which was further renamed to Float32Array and split into a backing ArrayBuffer and the typed Float32Array-view to access the buffer. Types were also added for other integer and floating-point sizes and signed/unsigned variants.

Design considerations #

From the beginning, the design of Typed Arrays was driven by the need to efficiently pass binary data to native libraries. For this reason, the typed array views operate upon aligned data in the host CPU’s native endianness. These decisions make it possible for JavaScript to reach maximum performance during operations such as sending vertex data to the graphics card.

DataView is specifically designed for file and network I/O, where the data always has a specified endianness, and might not be aligned for maximum performance.

The design split between in-memory data assembly (using the typed array views) and I/O (using DataView) was a conscious one. Modern JavaScript engines optimize the typed array views heavily, and achieve high performance on numerical operations with them. The current levels of performance of the typed array views were made possible by this design decision.

References #

The amazing powers of CSS

2012-06-19T00:00:00Z

Yesterday at the office, we were coming up with strange and magical CSS tricks. Take this one for instance, it makes empty links very visible:

a[href = ""] {
    background: red;
    color: white;
    font-size: x-large;
}

Check out the live example at jsFiddle

You can also style absolute links differently from relative links:

a[href ^= http] {
    display: inline-block;
    color: red;
    transform: rotate(180deg);
}

Check out the live example at jsFiddle

If you want to have a different style for links pointing out of your domain, you can use the :not() selector. This is actually how we do the little arrows next to external links at HTML5Rocks.

a[href ^= 'http']:not([href *= 'html5rocks.']) {
    background: transparent url(arrow.png) no-repeat center right;
    padding-right: 16px;
}

Check out the live example at jsFiddle

Just to remind you that you're not limited to styling links, here's how to make all PNG images inverted:

img[src $= .png] {
    filter: invert(100%);
}

Moving on from attribute selectors, did you know that you can make the document head visible, along with the other elements there?

head {
    display: block;
    border-bottom: 5px solid red;
}
script, style, link {
    display: block;
    white-space: pre;
    font-family: monospace;
}

Or that you can use the awesome power of CSS attr-function to fill in the :after and :before content?

script:before {
    content: "<script src=\"" attr(src) "\" type=\"" attr(type) "\">";
}
script:after {
    content: "</script>";
}

style:before {
    content: "<style type=\"" attr(type) "\">";
}
style:after {
    content: "< /style>";
}

/* And for a finish, <link> */
link:before {
    content: "<link rel=\"" attr(rel) "\" type=\"" attr(type) "\" href=\"" attr(href) "\" />";
}

Check out the live example at jsFiddle

Note that attr() reads in the attribute values of the matching element, so if you use it for #foo:before, it reads the attributes from #foo.

Writing augmented reality applications using JSARToolKit

2012-02-28T00:00:00Z

Introduction #

This article is about using the JSARToolKit library with the WebRTC getUserMedia API to do augmented reality applications on the web. For rendering, I'm using WebGL due to the increased performance it offers. The end result of this article is a demo application that puts a 3D model on top of an augmented reality marker in webcam video.

JSARToolKit is an augmented reality library for JavaScript. It's an open source library released under the GPL and a direct port of the Flash FLARToolKit that I made for the Mozilla Remixing Reality demo. FLARToolKit itself is port of the Java NyARToolKit, which is a port of the C ARToolKit. Long way, but here we are.

JSARToolKit operates on canvas elements. As it needs to read the image off the canvas, the image needs to come from the same origin as the page or use CORS to get around same-origin policy. In a nutshell, set the crossOrigin-property on the image or video element you want to use as a texture to '' or 'anonymous'.

When you pass a canvas to JSARToolKit for analysis, JSARToolKit returns a list of AR markers found in the image and the corresponding transformation matrices. To draw a 3D object on top of a marker, you pass the transformation matrix to whatever 3D rendering library you're using so that your object is transformed using the matrix. Then, draw the video frame in your WebGL scene and draw the object on top of that and you're good to go.

To analyze video using the JSARToolKit, draw the video on a canvas, then pass the canvas to JSARToolKit. Do this for every frame and you've got video AR tracking. JSARToolKit is fast enough on modern JavaScript engines to do this in realtime even on 640x480 video frames. However, the larger the video frame, the longer it takes to process. A good video frame size is 320x240, but if you expect to use small markers or multiple markers, 640x480 is preferable.

Demo #

To view the webcam demo, you need to have WebRTC enabled in your browser (on Chrome, go the about:flags and enable MediaStream). You also need to print out the AR marker below. You can also try opening the marker image on your phone or tablet and showing it to the webcam.

AR marker.

Setting up JSARToolKit #

The JSARToolKit API is quite Java-like, so you'll have to do some contortions to use it. The basic idea is that you have a detector object that operates on a raster object. Between the detector and the raster is a camera parameter object that transforms raster coordinates to camera coordinates. To get the detected markers from the detector, you iterate over them and copy their transformation matrices over to your code.

The first step is to create the raster object, the camera parameter object and the detector object.

// Create a RGB raster object for the 2D canvas.
// JSARToolKit uses raster objects to read image data.
// Note that you need to set canvas.changed = true on every frame.
var raster = new NyARRgbRaster_Canvas2D(canvas);

// FLARParam is the thing used by FLARToolKit to set camera parameters.
// Here we create a FLARParam for images with 320x240 pixel dimensions.
var param = new FLARParam(320, 240);

// The FLARMultiIdMarkerDetector is the actual detection engine for marker detection.
// It detects multiple ID markers. ID markers are special markers that encode a number.
var detector = new FLARMultiIdMarkerDetector(param, 120);

// For tracking video set continue mode to true. In continue mode, the detector
// tracks markers across multiple frames.
detector.setContinueMode(true);

// Copy the camera perspective matrix from the FLARParam to the WebGL library camera matrix.
// The second and third parameters determine the zNear and zFar planes for the perspective matrix.
param.copyCameraMatrix(display.camera.perspectiveMatrix, 10, 10000);

Using getUserMedia to access the webcam #

Next, I'm going to create a video element that's getting webcam video through the WebRTC APIs. For pre-recorded videos, just set the source attribute of the video to the video URL. If you're doing marker detection from still images, you can use a image element in much the same way.

As WebRTC and getUserMedia are still new emerging technologies, you need to feature detect them. For more details, check out Eric Bidelman's article on Capturing Audio & Video in HTML5.

var video = document.createElement('video');
video.width = 320;
video.height = 240;

var getUserMedia = function(t, onsuccess, onerror) {
  if (navigator.getUserMedia) {
    return navigator.getUserMedia(t, onsuccess, onerror);
  } else if (navigator.webkitGetUserMedia) {
    return navigator.webkitGetUserMedia(t, onsuccess, onerror);
  } else if (navigator.mozGetUserMedia) {
    return navigator.mozGetUserMedia(t, onsuccess, onerror);
  } else if (navigator.msGetUserMedia) {
    return navigator.msGetUserMedia(t, onsuccess, onerror);
  } else {
    onerror(new Error("No getUserMedia implementation found."));
  }
};

var URL = window.URL || window.webkitURL;
var createObjectURL = URL.createObjectURL || webkitURL.createObjectURL;
if (!createObjectURL) {
  throw new Error("URL.createObjectURL not found.");
}

getUserMedia({'video': true},
  function(stream) {
    var url = createObjectURL(stream);
    video.src = url;
  },
  function(error) {
    alert("Couldn't access webcam.");
  }
);

Detecting markers #

Once we have the detector running a-ok, we can start feeding it images to detect AR matrices. First draw the image onto the raster object canvas, then run the detector on the raster object. The detector returns the number of markers found in the image.

// Draw the video frame to the raster canvas, scaled to 320x240.
canvas.getContext('2d').drawImage(video, 0, 0, 320, 240);

// Tell the raster object that the underlying canvas has changed.
canvas.changed = true;

// Do marker detection by using the detector object on the raster object.
// The threshold parameter determines the threshold value
// for turning the video frame into a 1-bit black-and-white image.
//
var markerCount = detector.detectMarkerLite(raster, threshold);

The last step is to iterate through the detected markers and get their transformation matrices. You use the transformation matrices for putting 3D objects on top of the markers.

// Create a NyARTransMatResult object for getting the marker translation matrices.
var resultMat = new NyARTransMatResult();

var markers = {};

// Go through the detected markers and get their IDs and transformation matrices.
for (var idx = 0; idx &lt; markerCount; idx++) {
  // Get the ID marker data for the current marker.
  // ID markers are special kind of markers that encode a number.
  // The bytes for the number are in the ID marker data.
  var id = detector.getIdMarkerData(idx);

  // Read bytes from the id packet.
  var currId = -1;
  // This code handles only 32-bit numbers or shorter.
  if (id.packetLength <= 4) {
    currId = 0;
    for (var i = 0; i &lt; id.packetLength; i++) {
      currId = (currId << 8) | id.getPacketData(i);
    }
  }

  // If this is a new id, let's start tracking it.
  if (markers[currId] == null) {
    markers[currId] = {};
  }
  // Get the transformation matrix for the detected marker.
  detector.getTransformMatrix(idx, resultMat);

  // Copy the result matrix into our marker tracker object.
  markers[currId].transform = Object.asCopy(resultMat);
}

Matrix mapping #

Here's the code to copy JSARToolKit matrices over to glMatrix matrices (which are 16-element FloatArrays with the translation column in the last four elements). It works by magic (read: I don't know how ARToolKit matrices are setup. Inverted Y-axis is my guess.) Anyway, this bit of sign-reversing voodoo makes a JSARToolKit matrix work the same as a glMatrix.

To use the library with another library, such as Three.js, you need to write a function that converts the ARToolKit matrices to the library's matrix format. You also need to hook into the FLARParam.copyCameraMatrix method. The copyCameraMatrix method writes the FLARParam perspective matrix into a glMatrix-style matrix.

function copyMarkerMatrix(arMat, glMat) {
  glMat[0] = arMat.m00;
  glMat[1] = -arMat.m10;
  glMat[2] = arMat.m20;
  glMat[3] = 0;
  glMat[4] = arMat.m01;
  glMat[5] = -arMat.m11;
  glMat[6] = arMat.m21;
  glMat[7] = 0;
  glMat[8] = -arMat.m02;
  glMat[9] = arMat.m12;
  glMat[10] = -arMat.m22;
  glMat[11] = 0;
  glMat[12] = arMat.m03;
  glMat[13] = -arMat.m13;
  glMat[14] = arMat.m23;
  glMat[15] = 1;
}

Three.js integration #

Three.js is a popular JavaScript 3D engine. I'm going to go through how to use JSARToolKit output in Three.js. You need three things: a full screen quad with the video image drawn onto it, a camera with the FLARParam perspective matrix and an object with marker matrix as its transform. I'll walk you through the integration in the code below.

// I'm going to use a glMatrix-style matrix as an intermediary.
// So the first step is to create a function to convert a glMatrix matrix into a Three.js Matrix4.
THREE.Matrix4.prototype.setFromArray = function(m) {
  return this.set(
    m[0], m[4], m[8], m[12],
    m[1], m[5], m[9], m[13],
    m[2], m[6], m[10], m[14],
    m[3], m[7], m[11], m[15]
  );
};

// glMatrix matrices are flat arrays.
var tmp = new Float32Array(16);

// Create a camera and a marker root object for your Three.js scene.
var camera = new THREE.Camera();
scene.add(camera);

var markerRoot = new THREE.Object3D();
markerRoot.matrixAutoUpdate = false;

// Add the marker models and suchlike into your marker root object.
var cube = new THREE.Mesh(
  new THREE.CubeGeometry(100,100,100),
  new THREE.MeshBasicMaterial({color: 0xff00ff})
);
cube.position.z = -50;
markerRoot.add(cube);

// Add the marker root to your scene.
scene.add(markerRoot);

// Next we need to make the Three.js camera use the FLARParam matrix.
param.copyCameraMatrix(tmp, 10, 10000);
camera.projectionMatrix.setFromArray(tmp);


// To display the video, first create a texture from it.
var videoTex = new THREE.Texture(videoCanvas);

// Then create a plane textured with the video.
var plane = new THREE.Mesh(
  new THREE.PlaneGeometry(2, 2, 0),
  new THREE.MeshBasicMaterial({map: videoTex})
);

// The video plane shouldn't care about the z-buffer.
plane.material.depthTest = false;
plane.material.depthWrite = false;

// Create a camera and a scene for the video plane and
// add the camera and the video plane to the scene.
var videoCam = new THREE.Camera();
var videoScene = new THREE.Scene();
videoScene.add(plane);
videoScene.add(videoCam);

...

// On every frame do the following:
function tick() {
  // Draw the video frame to the canvas.
  videoCanvas.getContext('2d').drawImage(video, 0, 0);
  canvas.getContext('2d').drawImage(videoCanvas, 0, 0, canvas.width, canvas.height);

  // Tell JSARToolKit that the canvas has changed.
  canvas.changed = true;

  // Update the video texture.
  videoTex.needsUpdate = true;

  // Detect the markers in the video frame.
  var markerCount = detector.detectMarkerLite(raster, threshold);
  for (var i=0; i&lt;markerCount; i++) {
    // Get the marker matrix into the result matrix.
    detector.getTransformMatrix(i, resultMat);

    // Copy the marker matrix to the tmp matrix.
    copyMarkerMatrix(resultMat, tmp);

    // Copy the marker matrix over to your marker root object.
    markerRoot.matrix.setFromArray(tmp);
  }

  // Render the scene.
  renderer.autoClear = false;
  renderer.clear();
  renderer.render(videoScene, videoCam);
  renderer.render(scene, camera);
}

Summary #

In this article we went through the basics of JSARToolKit. Now you are ready to build your own webcam-using augmented reality applications with JavaScript.

Integrating JSARToolKit with Three.js is a bit of a hassle, but it is certainly possible. I'm not 100% certain if I'm doing it right in my demo, so please let me know if you know of a better way of achieving the integration. Patches are welcome :)

References #

Mixing positional audio and WebGL

2012-02-16T00:00:00Z

Introduction #

In this article I’m going to talk about how to use the positional audio feature in the Web Audio API to add 3D sound into your WebGL scenes. To make the audio more believable, I will also introduce you to the environmental effects possible with the Web Audio API. To get a more thorough introduction to the Web Audio API, check out the Getting started with Web Audio API article by Boris Smus.

To do positional audio, you use the AudioPannerNode in the Web Audio API. The AudioPannerNode defines the position, orientation and velocity of a sound. Additionally, the Web Audio API audio context has a listener attribute that lets you define the position, orientation and velocity of the listener. With these two things you can create directional sounds with doppler effects and 3D panning.

Let's see what the audio code looks like for the above scene. This is very basic Audio API code. You create a bunch of Audio API nodes and connect them together. The audio nodes are individual sounds, volume controllers, effect nodes and analyzers and suchlike. After you have built this graph, you need to hook it up to the audio context destination to make it audible.

// Detect if the audio context is supported.
window.AudioContext = (
  window.AudioContext ||
  window.webkitAudioContext ||
  null
);

if (!AudioContext) {
  throw new Error("AudioContext not supported!");
} 

// Create a new audio context.
var ctx = new AudioContext();

// Create a AudioGainNode to control the main volume.
var mainVolume = ctx.createGain();
// Connect the main volume node to the context destination.
mainVolume.connect(ctx.destination);

// Create an object with a sound source and a volume control.
var sound = {};
sound.source = ctx.createBufferSource();
sound.volume = ctx.createGain();

// Connect the sound source to the volume control.
sound.source.connect(sound.volume);
// Hook up the sound volume control to the main volume.
sound.volume.connect(mainVolume);

// Make the sound source loop.
sound.source.loop = true;

// Load a sound file using an ArrayBuffer XMLHttpRequest.
var request = new XMLHttpRequest();
request.open("GET", soundFileName, true);
request.responseType = "arraybuffer";
request.onload = function(e) {

  // Create a buffer from the response ArrayBuffer.
  ctx.decodeAudioData(this.response, function onSuccess(buffer) {
    sound.buffer = buffer;

    // Make the sound source use the buffer and start playing it.
    sound.source.buffer = sound.buffer;
    sound.source.start(ctx.currentTime);
  }, function onFailure() {
    alert("Decoding the audio buffer failed");
  });
};
request.send();

Position #

Positional audio uses the position of your audio sources and the position of the listener to determine how to mix the sound to the speakers. An audio source on the left side of the listener would be louder in the left speaker, and vice versa for the right side.

To get started, create an audio source and attach it to an AudioPannerNode. Then set the position of the AudioPannerNode. Now you have a movable 3D sound. The audio context listener position is at (0,0,0) by default, so when used in this way, the AudioPannerNode position is relative to the camera position. Whenever you move the camera, you need to update the AudioPannerNode position. To make the AudioPannerNode position relative to the world, you need to change the audio context listener position to your camera position.

To set up the position tracking, we need to create an AudioPannerNode and wire it up to the main volume.

...
sound.panner = ctx.createPanner();
// Instead of hooking up the volume to the main volume, hook it up to the panner.
sound.volume.connect(sound.panner);
// And hook up the panner to the main volume.
sound.panner.connect(mainVolume);
...

On every frame, update the positions of the AudioPannerNodes. I'm going to be using Three.js in the examples below.

...
// In the frame handler function, get the object's position.
object.position.set(newX, newY, newZ);
object.updateMatrixWorld();
var p = new THREE.Vector3();
p.setFromMatrixPosition(object.matrixWorld);

// And copy the position over to the sound of the object.
sound.panner.setPosition(p.x, p.y, p.z);
...

To track the listener position, set the audio context's listener position to match the camera position.

...
// Get the camera position.
camera.position.set(newX, newY, newZ);
camera.updateMatrixWorld();
var p = new THREE.Vector3();
p.setFromMatrixPosition(camera.matrixWorld);

// And copy the position over to the listener.
ctx.listener.setPosition(p.x, p.y, p.z);
...

Velocity #

Now that we have the positions of the listener and the AudioPannerNode, let’s turn our attention to their velocities. By changing the velocity properties of the listener and the AudioPannerNode, you can add a doppler effect to the sound. There are some nice doppler effect examples on the Web Audio API examples page.

The easiest way to get the velocities for the listener and the AudioPannerNode is to keep track of their per-frame positions. The velocity of the listener is the camera’s current position minus the camera’s position in the previous frame. Similarly, the velocity of the AudioPannerNode is its current position minus its previous position.

Tracking the velocity can be done by getting the object's previous position, subtracting it from the current position and dividing the result by the time elapsed since last frame. Here's how to do it in Three.js:

...
var dt = secondsSinceLastFrame;

var p = new THREE.Vector3();
p.setFromMatrixPosition(object.matrixWorld);
var px = p.x, py = p.y, pz = p.z;

object.position.set(newX, newY, newZ);
object.updateMatrixWorld();

var q = new THREE.Vector3();
q.setFromMatrixPosition(object.matrixWorld);
var dx = q.x-px, dy = q.y-py, dz = q.z-pz;

sound.panner.setPosition(q.x, q.y, q.z);
sound.panner.setVelocity(dx/dt, dy/dt, dz/dt);
...

Orientation #

Orientation is the direction the sound source is pointing to and the direction the listener is facing. With orientation, you can simulate directional sound sources. For an example, think of a directional speaker. If you stand in front of the speaker, the sound is going to be louder than if you stand behind the speaker. More importantly, you need listener orientation to determine which side of the listener the sounds are coming from. A sound coming from your left needs to switch to the right when you turn around.

To get the orientation vector for the AudioPannerNode, you need to take the rotation part of the model matrix of the sound-emitting 3D object and multiply a vec3(0,0,1) with it to see where it ends up pointing. For the context listener orientation, you need to get the camera’s orientation vector. The listener orientation also needs an up vector, since it needs to know the roll angle of the listener's head. To compute the listener orientation, get the rotation part of the camera's view matrix and multiply a vec3(0,0,1) for the orientation and a vec3(0,-1,0) for the up-vector.

For orientation to have an effect on your sounds, you also need to define the cone for the sound. The sound cone takes an inner angle, outer angle and outer gain. The sound plays at the normal volume inside the inner angle and gradually changes gain to outer gain as you approach the outer angle. Outside the outer angle the sound plays at outer gain.

Tracking the orientation in Three.js is a bit trickier as it involves some vector math and zeroing the translation part of the 4x4 world matrices. Still, not many lines of code.

...
var vec = new THREE.Vector3(0,0,1);
var m = object.matrixWorld;

// Save the translation column and zero it.
var mx = m.elements[12], my = m.elements[13], mz = m.elements[14];
m.elements[12] = m.elements[13] = m.elements[14] = 0;

// Multiply the 0,0,1 vector by the world matrix and normalize the result.
vec.applyProjection(m);
vec.normalize();

sound.panner.setOrientation(vec.x, vec.y, vec.z);

// Restore the translation column.
m.elements[12] = mx;
m.elements[13] = my;
m.elements[14] = mz;
...

Camera orientation tracking also requires the up vector, so you need to multiply an up vector with the transformation matrix.

...
// The camera's world matrix is named "matrix".
var m = camera.matrix;

var mx = m.elements[12], my = m.elements[13], mz = m.elements[14];
m.elements[12] = m.elements[13] = m.elements[14] = 0;

// Multiply the orientation vector by the world matrix of the camera.
var vec = new THREE.Vector3(0,0,1);
vec.applyProjection(m);
vec.normalize();

// Multiply the up vector by the world matrix.
var up = new THREE.Vector3(0,-1,0);
up.applyProjection(m);
up.normalize();

// Set the orientation and the up-vector for the listener.
ctx.listener.setOrientation(vec.x, vec.y, vec.z, up.x, up.y, up.z);

m.elements[12] = mx;
m.elements[13] = my;
m.elements[14] = mz;
...

To set the sound cone for your sound, you set the appropriate properties of the panner node. The cone angles are in degrees and run from 0 to 360.

...
sound.panner.coneInnerAngle = innerAngleInDegrees;
sound.panner.coneOuterAngle = outerAngleInDegrees;
sound.panner.coneOuterGain = outerGainFactor;
...

All together #

Putting it all together, the audio context listener follows the position, orientation and velocity of the camera, and the AudioPannerNodes follow the positions, orientations and velocities of their respective audio sources. You need to update the positions, velocities and orientations of the AudioPannerNodes and the audio context listener on every frame.

Environmental effects #

After you’ve got positional audio set up, you can set the environmental effects for your audio to enhance the immersiveness of your 3D scene. Suppose your scene is set inside a large cathedral. On default settings, the sounds in your scene sound like you’re standing outdoors. This discrepancy between the visuals and audio breaks immersion and makes your scene less impressive.

The Web Audio API has a ConvolverNode that lets you set the environmental effect for a sound. Add it to the processing graph for the audio source and you’ll be able to make the sound fit the setting. You can find impulse response samples on the web that you can use with ConvolverNodes, and you can also make your own. It may be slightly cumbersome experience as you need to record the impulse response of the place you wish to simulate, but the capability is there if you need it.

Using ConvolverNodes to do environmental audio requires rewiring the audio processing graph. Instead of passing sound directly to the main volume, you need to route it through the ConvolverNode. And as you may want to control the strength of the environmental effect, you also need to route the audio around the ConvolverNode. To control the mix volumes the ConvolverNode and the plain audio need to have GainNodes attached to them.

The final audio processing graph I'm using has the audio from the objects going through a GainNode used as a pass-through mixer. From the mixer I pass the audio into the ConvolverNode and another GainNode, which is used to control the volume of the plain audio. The ConvolverNode is hooked into its own GainNode to control the convolved audio volume. The outputs of the GainNodes are connected to the main volume controller.

...
var ctx = new webkitAudioContext();
var mainVolume = ctx.createGain();

// Create a convolver to apply environmental effects to the audio.
var convolver = ctx.createConvolver();

// Create a mixer that receives sound from the panners.
var mixer = ctx.createGain();

sounds.forEach(function(sound){
  sound.panner.connect(mixer);
});

// Create volume controllers for the plain audio and the convolver.
var plainGain = ctx.createGain();
var convolverGain = ctx.createGain();

// Send audio from the mixer to plainGain and the convolver node.
mixer.connect(plainGain);
mixer.connect(convolver);

// Hook up the convolver to its volume control.
convolver.connect(convolverGain);

// Send audio from the volume controls to the main volume control.
plainGain.connect(mainVolume);
convolverGain.connect(mainVolume);

// Finally, connect the main volume to the audio context's destination.
volume.connect(ctx.destination);
...

To make the ConvolverNode work, you need to load an impulse response sample into a buffer and make the ConvolverNode use it. Loading the sample happens in the same way as with normal sound samples. Below is an example of one way to do it:

...
loadBuffer(ctx, "impulseResponseExample.wav", function(buffer){
  convolver.buffer = buffer;
  convolverGain.gain.value = 0.7;
  plainGain.gain.value = 0.3;
})
...
function loadBuffer(ctx, filename, callback) {
  var request = new XMLHttpRequest();
  request.open("GET", soundFileName, true);
  request.responseType = "arraybuffer";
  request.onload = function() {
    // Create a buffer and keep the channels unchanged.
    ctx.decodeAudioData(request.response, callback, function() {
      alert("Decoding the audio buffer failed");
    });
  };
  request.send();
}

Summary #

In this article you learned how to add positional audio to your 3D scenes using the Web Audio API. The Web Audio API gives you a way to set the position, orientation and velocity of audio sources and the listener. By setting those to track the objects in your 3D scene, you can create a rich soundscape for your 3D applications.

To make the audio experience even more compelling, you can use the ConvolverNode in the Web Audio API to set up the general sound of the environment. From cathedrals to closed rooms, you can simulate a variety of effects and environments using the Web Audio API.

References #

Web Audio API specification
Impulse response
Three.js
A cool 3D Positional Audio example
Web Audio Examples has a lot of good examples on using Web Audio API features.

Google Photography Prize Gallery

2012-02-07T00:00:00Z

Creating the Google Photography Prize Gallery #

We recently launched the Gallery section on the Google Photography Prize site. The gallery shows an infinite scrolling list of photos fetched from Google+. It gets the list of photos from an AppEngine app that we use for moderating the list of photos in the gallery. We also released the gallery app as an open source project on Google Code.

The backend of the gallery is an AppEngine app that uses the Google+ API to search for posts with one of the Google Photography Prize hashtags on it (e.g. #megpp and #travelgpp). The app then adds those posts to its list of unmoderated photos. Once a week, our content team goes through the list of unmoderated photos and flags ones that break our content guidelines. After hitting the Moderate-button, the unflagged photos are added to the list of photos shown on the gallery page.

Gallery frontend #

The Gallery frontend is built using the Google Closure library. The Gallery widget itself is a Closure component. At the top of the source file we tell Closure that this file provides a component named photographyPrize.Gallery and require the parts of the Closure library used by the app:

goog.provide('photographyPrize.Gallery');

goog.require('goog.debug.Logger');
goog.require('goog.dom');
goog.require('goog.dom.classes');
goog.require('goog.events');
goog.require('goog.net.Jsonp');
goog.require('goog.style');

The gallery page has a bit of JavaScript that uses JSONP to retrieve the list of photos from the AppEngine app. JSONP is a simple cross-origin JavaScript hack that injects a script that looks like jsonpcallback("responseValue"). To handle the JSONP stuff, we’re using the goog.net.Jsonp component in the Closure library.

The gallery script goes through the list of photos and generates HTML elements for them to show them on the gallery page. The infinite scrolling works by hooking up to the window scroll event and loading a new batch of photos when the window scroll is close to the bottom of the page. After loading the new photo list segment, the gallery script creates elements for the photos and adds them to the gallery element to display them.

Displaying the list of images #

The image list display method is pretty basic stuff. It goes through the image list, generates HTML elements and +1 buttons. The next step is to add the generated list segment to the gallery's main gallery element. You can see some Closure compiler conventions in the code below, note the type definitions in the JSDoc comment and the @private visibility. Private methods have an underscore (_) after their name.

/**
 * Displays images in imageList by putting them inside the section element.
 * Edits image urls to scale them down to imageSize x imageSize bounding
 * box.
 *
 * @param {Array.&lt;Object>} imageList List of image objects to show. Retrieved
 *                                   by loadImages.
 * @return {Element} The generated image list container element.
 * @private
 */
photographyPrize.Gallery.prototype.displayImages_ = function(imageList) {
  
  // find the images and albums from the image list
  for (var j = 0; j &lt; imageList.length; j++) {
    // change image urls to scale them to photographyPrize.Gallery.MAX_IMAGE_SIZE
  }

  // Go through the image list and create a gallery photo element for each image.
  // This uses the Closure library DOM helper, goog.dom.createDom:
  // element = goog.dom.createDom(tagName, className, var_childNodes);

  var category = goog.dom.createDom('div', 'category');
  for (var k = 0; k &lt; items.length; k++) {
    var plusone = goog.dom.createDom('g:plusone');
    plusone.setAttribute('href', photoPageUrl);
    plusone.setAttribute('size', 'standard');
    plusone.setAttribute('annotation', 'none');

    var photo = goog.dom.createDom('div', {className: 'gallery-photo'}, ...)
    photo.appendChild(plusone);

    category.appendChild(photo);
  }
  this.galleryElement_.appendChild(category);
  return category;
};

Handling scroll events #

To see when the visitor has scrolled the page to the bottom and we need to load new images, the gallery hooks up to the window object's scroll event. To paper over differences in browser implementations, we're using some handy utility functions from the Closure library: goog.dom.getDocumentScroll() returns an {x, y} object with the current document scroll position, goog.dom.getViewportSize() returns the window size, and goog.dom.getDocumentHeight() the height of the HTML document.

/**
 * Handle window scroll events by loading new images when the scroll reaches
 * the last screenful of the page.
 *
 * @param {goog.events.BrowserEvent} ev The scroll event.
 * @private
 */
photographyPrize.Gallery.prototype.handleScroll_ = function(ev) {
  var scrollY = goog.dom.getDocumentScroll().y;
  var height = goog.dom.getViewportSize().height;
  var documentHeight = goog.dom.getDocumentHeight();
  if (scrollY + height >= documentHeight - height / 2) {
    this.tryLoadingNextImages_();
  }
};

/**
 * Try loading the next batch of images objects from the server.
 * Only fires if we have already loaded the previous batch.
 *
 * @private
 */
photographyPrize.Gallery.prototype.tryLoadingNextImages_ = function() {
  // ...
};

Loading images #

To load the images from the server, we're using the goog.net.Jsonp component. It takes a goog.Uri to query. Once created, you can send a query to the Jsonp provider with a query parameter object and a success callback function.

/**
 * Loads image list from the App Engine page and sets the callback function
 * for the image list load completion.
 *
 * @param {string} tag Fetch images tagged with this.
 * @param {number} limit How many images to fetch.
 * @param {number} offset Offset for the image list.
 * @param {function(Array.&lt;Object>=)} callback Function to call
 *        with the loaded image list.
 * @private
 */
photographyPrize.Gallery.prototype.loadImages_ = function(tag, limit, offset, callback) {
  var jsonp = new goog.net.Jsonp(
      new goog.Uri(photographyPrize.Gallery.IMAGE_LIST_URL));
  jsonp.send({'tag': tag, 'limit': limit, 'offset': offset}, callback);
};

As mentioned above, the gallery script uses the Closure compiler for compiling and minifying the code. The Closure compiler is also useful in enforcing correct typing (you use @type foo JSDoc notation in your comments to set the type of a property) and it also tells you when you don’t have comments for a method.

Unit tests #

We also needed unit tests for the gallery script, so it’s handy that the Closure library has an unit testing framework built into it. It follows the jsUnit conventions, so it’s easy to get started with.

To help me in writing the unit tests, I wrote a small Ruby script that parses the JavaScript file and generates a failing unit test for each method and property in the gallery component. Given a script like:

Foo = function() {}
Foo.prototype.bar = function() {}
Foo.prototype.baz = "hello";

The test generator generates an empty test for each of the properties:

function testFoo() {
  fail();
  Foo();
}

function testFooPrototypeBar = function() {
  fail();
  instanceFoo.bar();
}

function testFooPrototypeBaz = function() {
  fail();
  instanceFoo.baz;
}

These autogenerated tests gave me an easy start in writing tests for the code, and all the methods and properties were covered by default. The failing tests create a nice psychological effect where I had to go through the tests one by one and write proper tests. Coupled with a code coverage meter, it’s a fun game to make the tests and coverage all green.

Summary #

Gallery+ is an open source project to display a moderated list of Google+ photos matching a #hashtag. It was built using Go and the Closure library. The backend runs on App Engine. Gallery+ is used on the Google Photography Prize website to display the submission gallery. In this article we walked through the juicy bits of the frontend script. My collague Johan Euphrosine from the App Engine Developer Relations team is writing a second article talking about the backend app. The backend is written in Go, Google's new server-side language. So if you're interested in seeing a production example of Go code, stay tuned!

References #

Animating a million letters using Three.js

2012-01-30T00:00:00Z

Introduction #

My goal in this article is to draw a million animated letters on the screen at a smooth frame rate. This task should be quite possible with modern GPUs. Each letter consists of two textured triangles, so we're only talking about two million triangles per frame.

If you're coming from a traditional JavaScript animation background, this all sounds like madness. Two million triangles updated every frame is definitely not something you would like to do with JavaScript today. But thankfully we have WebGL, which lets us tap into the awesome power of modern GPUs. And two million animated triangles is quite doable with a modern GPU and some shader magic.

Writing efficient WebGL code #

Writing efficient WebGL code requires a certain mindset. The usual way to draw using WebGL is to set up your uniforms, buffers and shaders for each object, followed by a call to draw the object. This way of drawing works when drawing a small number of objects. To draw a large number of objects, you should minimize the amount of WebGL state changes. To start with, draw all objects using the same shader after each other, so that you don't have to change shaders between objects. For simple objects like particles, you could bundle several objects into a single buffer and edit it using JavaScript. That way you'd only have to reupload the vertex buffer instead of changing shader uniforms for every single particle.

But to go really fast, you need to push most of your computation to the shaders. That's what I'm trying to do here. Animate a million letters using shaders.

The article's code uses the Three.js library, which abstracts away all the tedious boilerplate from writing WebGL code. Instead of having to write hundreds of lines of WebGL state setup and error handling, with Three.js you only need to write a couple lines of code. It's also easy to tap into the WebGL shader system from Three.js.

Drawing multiple objects using a single draw call #

Here's a small pseudo-code example of how you might draw multiple objects using a single draw call. The traditional way is to draw one object at a time like this:

for (var i=0; i<objects.length; i++) {
  // each added object requires a separate WebGL draw call
  scene.add(createNewObject(objects[i]));
}
renderer.render(scene, camera);

But the above method requires a separate draw call for each object. To draw multiple objects at once, you can bundle the objects into a single geometry and get away with a single draw call:

var geo = new THREE.Geometry();
for (var i=0; i<objects.length; i++) {
  // bundle the objects into a single geometry
  // so that they can be drawn with a single draw call
  addObjectToGeometry(geo, objects[i]);
}
// GOOD! Only one object to add to the scene!
scene.add(new THREE.Mesh(geo, material));
renderer.render(scene, camera);

Alright, now that you've got the basic idea, let's get back to writing the demo and start animating those million letters!

Setting up the geometry and textures #

As the first step, I'm going to create a texture with the letter bitmaps on it. I'm using the 2D canvas for this. The resulting texture has all the letters I want to draw. The next step is to create a buffer with the texture coordinates to the letter sprite sheet. While this is an easy and straightforward method to set up the letters, it’s a bit wasteful as it uses two floats per vertex for the texture coordinates. A shorter way - left as an exercise to the reader - would be to pack the letter index and corner index into one number and convert that back to texture coordinates in the vertex shader.

Here's how I build the letter texture using Canvas 2D:

var fontSize = 16;

// The square letter texture will have 16 * 16 = 256 letters, enough for all 8-bit characters.
var lettersPerSide = 16;

var c = document.createElement('canvas');
c.width = c.height = fontSize*lettersPerSide;
var ctx = c.getContext('2d');
ctx.font = fontSize+'px Monospace';

// This is a magic number for aligning the letters on rows. YMMV.
var yOffset = -0.25;

// Draw all the letters to the canvas.
for (var i=0,y=0; y<lettersPerSide; y++) {
  for (var x=0; x<lettersPerSide; x++,i++) {
    var ch = String.fromCharCode(i);
    ctx.fillText(ch, x*fontSize, yOffset*fontSize+(y+1)*fontSize);
  }
}

// Create a texture from the letter canvas.
var tex = new THREE.Texture(c);
// Tell Three.js not to flip the texture.
tex.flipY = false;
// And tell Three.js that it needs to update the texture.
tex.needsUpdate = true;

I also upload the triangle array to the GPU. These vertices are used by the vertex shader to put the letters on the screen. The vertices are set to the letter positions in the text so that if you render the triangle array as-is, you get a basic layout rendering of the text.

Creating the geometry for the book:

var geo = new THREE.Geometry();

var i=0, x=0, line=0;
for (i=0; i<BOOK.length; i++) {
  var code = BOOK.charCodeAt(i); // This is the character code for the current letter.
  if (code > lettersPerSide * lettersPerSide) {
    code = 0; // Clamp character codes to letter map size.
  }
  var cx = code % lettersPerSide; // Cx is the x-index of the letter in the map.
  var cy = Math.floor(code / lettersPerSide); // Cy is the y-index of the letter in the map.

  // Add letter vertices to the geometry.
  var v,t;
  geo.vertices.push(
    new THREE.Vector3( x*1.1+0.05, line*1.1+0.05, 0 ),
    new THREE.Vector3( x*1.1+1.05, line*1.1+0.05, 0 ),
    new THREE.Vector3( x*1.1+1.05, line*1.1+1.05, 0 ),
    new THREE.Vector3( x*1.1+0.05, line*1.1+1.05, 0 )
  );
  // Create faces for the letter.
  var face = new THREE.Face3(i*4+0, i*4+1, i*4+2);
  geo.faces.push(face);
  face = new THREE.Face3(i*4+0, i*4+2, i*4+3);
  geo.faces.push(face);

  // Compute texture coordinates for the letters.
  var tx = cx/lettersPerSide, 
      ty = cy/lettersPerSide,
      off = 1/lettersPerSide;
  var sz = lettersPerSide*fontSize;
  geo.faceVertexUvs[0].push([
    new THREE.Vector2( tx, ty+off ),
    new THREE.Vector2( tx+off, ty+off ),
    new THREE.Vector2( tx+off, ty )
  ]);
  geo.faceVertexUvs[0].push([
    new THREE.Vector2( tx, ty+off ),
    new THREE.Vector2( tx+off, ty ),
    new THREE.Vector2( tx, ty )
  ]);

  // On newline, move to the line below and move the cursor to the start of the line.
  // Otherwise move the cursor to the right.
  if (code == 10) {
    line--;
    x=0;
  } else {
    x++;
  }
}

Vertex shader for animating the letters #

With a simple vertex shader, I get a flat view of the text. Nothing fancy. Runs well, but if I want to animate it, I need to do the animation in JavaScript. And JavaScript is kinda slow for animating the six million vertices involved, especially if you want to do it on every frame. Maybe there is there a faster way.

Why yes, we can do procedural animation. What that means is that we do all our position and rotation math in the vertex shader. Now I don't need to run any JavaScript to update the positions of the vertices. The vertex shader runs very fast and I get a smooth frame rate even with a million triangles being individually animated every frame. To address the individual triangles, I round down the vertex coordinates so that all four points of a letter quad map to a single unique coordinate. Now I can use this coordinate to set the animation parameters for the letter in question.

To be able to successfully round down coordinates, coordinates from two different letters can't overlap. The easiest way to do this is by using square letter quads with a small offset separating the letter from the one on its right side and the line above it. For example, you could use a width and height of 0.5 for the letters and align the letters on integer coordinates. Now when you round down the coordinate of any letter vertex, you get the bottom-left coordinate of the letter.

Rounding down vertex coordinates to find the top-left corner of a letter.

To better understand the animated vertex shader, I'm going to go through a simple run-of-the-mill vertex shader first. This is what normally happens when you draw a 3D model to the screen. The vertices of the model are transformed by a couple transformation matrices to project each 3D vertex onto the 2D screen. Whenever a triangle defined by three of these vertices lands inside the viewport, the pixels it covers are processed by the fragment shader to color them. Anyway, here's the simple vertex shader:

varying float vUv;

void main() {
  // modelViewMatrix, position and projectionMatrix are magical
  // attributes that Three.js defines for us.

  // Transform current vertex by the modelViewMatrix
  // (bundled model world position & camera world position matrix).
  vec4 mvPosition = modelViewMatrix * position;

  // Project camera-space vertex to screen coordinates
  // using the camera's projection matrix.
  vec4 p = projectionMatrix * mvPosition;

  // uv is another magical attribute from Three.js.
  // We're passing it to the fragment shader unchanged.
  vUv = uv;

  gl_Position = p;
}

And now, the animated vertex shader. Basically, it does the same thing as the simple vertex shader, but with a small twist. Instead of transforming each vertex by just the transformation matrices, it applies a time-dependent animated transformation as well. To make each letter animate a bit differently, the animated vertex shader also modifies the animation based on the coordinates of the letter. It's going to look a good deal more complicated than the simple vertex shader because, well, it is more complicated.

uniform float uTime;
uniform float uEffectAmount;

varying float vZ;
varying vec2 vUv;

// rotateAngleAxisMatrix returns the mat3 rotation matrix
// for given angle and axis.
mat3 rotateAngleAxisMatrix(float angle, vec3 axis) {
  float c = cos(angle);
  float s = sin(angle);
  float t = 1.0 - c;
  axis = normalize(axis);
  float x = axis.x, y = axis.y, z = axis.z;
  return mat3(
    t*x*x + c,    t*x*y + s*z,  t*x*z - s*y,
    t*x*y - s*z,  t*y*y + c,    t*y*z + s*x,
    t*x*z + s*y,  t*y*z - s*x,  t*z*z + c
  );
}

// rotateAngleAxis rotates a vec3 over the given axis by the given angle and
// returns the rotated vector.
vec3 rotateAngleAxis(float angle, vec3 axis, vec3 v) {
  return rotateAngleAxisMatrix(angle, axis) * v;
}

void main() {
  // Compute the index of the letter (assuming 80-character max line length).
  float idx = floor(position.y/1.1)*80.0 + floor(position.x/1.1);

  // Round down the vertex coords to find the bottom-left corner point of the letter.
  vec3 corner = vec3(floor(position.x/1.1)*1.1, floor(position.y/1.1)*1.1, 0.0);

  // Find the midpoint of the letter.
  vec3 mid = corner + vec3(0.5, 0.5, 0.0);

  // Rotate the letter around its midpoint by an angle and axis dependent on
  // the letter's index and the current time.
  vec3 rpos = rotateAngleAxis(idx+uTime,
    vec3(mod(idx,16.0), -8.0+mod(idx,15.0), 1.0), position - mid) + mid;

  // uEffectAmount controls the amount of animation applied to the letter.
  // uEffectAmount ranges from 0.0 to 1.0.
  float effectAmount = uEffectAmount;

  vec4 fpos = vec4( mix(position,rpos,effectAmount), 1.0 );
  fpos.x += -35.0;

  // Apply spinning motion to individual letters.
  fpos.z += ((sin(idx+uTime*2.0)))*4.2*effectAmount;
  fpos.y += ((cos(idx+uTime*2.0)))*4.2*effectAmount;

  vec4 mvPosition = modelViewMatrix * fpos;

  // Apply wavy motion to the entire text.
  mvPosition.y += 10.0*sin(uTime*0.5+mvPosition.x/25.0)*effectAmount;
  mvPosition.x -= 10.0*cos(uTime*0.5+mvPosition.y/25.0)*effectAmount;

  vec4 p = projectionMatrix * mvPosition;

  // Pass texture coordinates and the vertex z-coordinate to the fragment shader.
  vUv = uv;
  vZ = p.z;

  // Send the final vertex position to WebGL.
  gl_Position = p;
}

To use the vertex shader, I use a THREE.ShaderMaterial, a material type that lets you use custom shaders and specify uniforms for them. Here's how I'm using THREE.ShaderMaterial in the demo:

// First, set up uniforms for the shader.
var uniforms = {

  // map contains the letter map texture.
  map: { type: "t", value: 1, texture: tex },

  // uTime is the urrent time.
  uTime: { type: "f", value: 1.0 },

  // uEffectAmount controls the amount of animation applied to the letters.
  uEffectAmount: { type: "f", value: 0.0 }
};

// Next, set up the THREE.ShaderMaterial.
var shaderMaterial = new THREE.ShaderMaterial({
  uniforms: uniforms,

  // I have my shaders inside HTML elements like
  // <script id="vertex" type="text/x-glsl-vert">... shaderSource ... <script>

  // The below gets the contents of the vertex shader script element.
  vertexShader: document.querySelector('#vertex').textContent,

  // The fragment shader is a bit special as well, drawing a rotating
  // rainbow gradient.
  fragmentShader: document.querySelector('#fragment').textContent
});

// I set depthTest to false so that the letters don't occlude each other.
shaderMaterial.depthTest = false;

On every animation frame, I update the shader uniforms:

// I'm controlling the uniforms through a proxy control object.
// The reason I'm using a proxy control object is to
// have different value ranges for the controls and the uniforms.
var controller = {
  effectAmount: 0
};

// I'm using <a href="http://code.google.com/p/dat-gui/">DAT.GUI</a> to do a quick & easy GUI for the demo.
var gui = new dat.GUI();
gui.add(controller, 'effectAmount', 0, 100);

var animate = function(t) {
  uniforms.uTime.value += 0.05;
  uniforms.uEffectAmount.value = controller.effectAmount/100;
  bookModel.position.y += 0.03;

  renderer.render(scene, camera);
  requestAnimationFrame(animate, renderer.domElement);
};
animate(Date.now());

And there you have it, shader-based animation. It looks pretty complex, but the only thing it really does is move the letters around in a way that depends on the current time and each letter's index. If performance wasn't a concern, you could have this logic running in JavaScript. However, at tens of thousands of animated objects, JavaScript stops being a viable solution.

Remaining concerns #

One problem now is that JavaScript doesn’t know about the particle positions. If you really need to know where your particles are, you could duplicate the vertex shader logic in JavaScript and recalculate the vertex positions using a web worker every time you need the positions. That way your rendering thread doesn’t have to wait for the math and you can continue animating at a smooth frame rate.

For more controllable animation, you could use render-to-texture functionality to animate between two sets of positions provided by JavaScript. First, render the current positions to a texture, then animate towards positions defined in a separate texture provided by JavaScript. The nice thing about this is that you can update a small fraction of the JavaScript-provided positions per frame and still continue animating all the letters every frame with the vertex shader tweening the positions.

Another concern is that 256 characters is far too few to do non-ASCII texts. If push the texture map size to 4096x4096 while decreasing the font size to 8px, you can fit the entire UCS-2 character set into the texture map. However, 8px font size is not very readable. To do larger font sizes, you can use multiple textures for your font. See this sprite atlas demo for an example. Another thing that would help is to create only the letters used in your text.

Summary #

In this article, I walked you through implementing a vertex shader -based animation demo using Three.js. The demo animates a million letters in real-time on a 2010 MacBook Air. The implementation bundled an entire book into a single geometry object for efficient drawing. The individual letters were animated by figuring out out which vertices belong to which letter and animating the vertices based on the index of the letter in the book text.

References #

Image filters with canvas

2011-05-25T00:00:00Z

Introduction #

The HTML5 canvas element can be used to write image filters. What you need to do is draw an image onto a canvas, read back the canvas pixels, and run your filter on them. You can then write the result onto a new canvas (or heck, just reuse the old one.)

Sounds simple? Good. Let's get cracking!

The original test image

Processing pixels #

First, retrieve the image pixels:

Filters = {};
Filters.getPixels = function(img) {
var c = this.getCanvas(img.width, img.height);
var ctx = c.getContext('2d');
ctx.drawImage(img);
return ctx.getImageData(0,0,c.width,c.height);
};

Filters.getCanvas = function(w,h) {
var c = document.createElement('canvas');
c.width = w;
c.height = h;
return c;
};

Next, we need a way to filter images. How about a filterImage method that takes a filter and an image and returns the filtered pixels?

Filters.filterImage = function(filter, image, var_args) {
var args = [this.getPixels(image)];
for (var i=2; i&lt;arguments.length; i++) {
args.push(arguments[i]);
}
return filter.apply(null, args);
};

Running simple filters #

Now that we have the pixel processing pipeline put together, it's time to write some simple filters. To start off, let's convert the image to grayscale.

Filters.grayscale = function(pixels, args) {
var d = pixels.data;
for (var i=0; i&lt;d.length; i+=4) {
var r = d[i];
var g = d[i+1];
var b = d[i+2];
// CIE luminance for the RGB
// The human eye is bad at seeing red and blue, so we de-emphasize them.
var v = 0.2126*r + 0.7152*g + 0.0722*b;
d[i] = d[i+1] = d[i+2] = v
}
return pixels;
};

Adjusting brightness can be done by adding a fixed value to the pixels:

Filters.brightness = function(pixels, adjustment) {
var d = pixels.data;
for (var i=0; i&lt;d.length; i+=4) {
d[i] += adjustment;
d[i+1] += adjustment;
d[i+2] += adjustment;
}
return pixels;
};

Thresholding an image is also quite simple. You just compare the grayscale value of a pixel to the threshold value and set the color based on that:

Filters.threshold = function(pixels, threshold) {
var d = pixels.data;
for (var i=0; i&lt;d.length; i+=4) {
var r = d[i];
var g = d[i+1];
var b = d[i+2];
var v = (0.2126*r + 0.7152*g + 0.0722*b >= threshold) ? 255 : 0;
d[i] = d[i+1] = d[i+2] = v
}
return pixels;
};

Convolving images #

Convolution filters are very useful generic filters for image processing. The basic idea is that you take the weighed sum of a rectangle of pixels from the source image and use that as the output value. Convolution filters can be used for blurring, sharpening, embossing, edge detection and a whole bunch of other things.

Filters.tmpCanvas = document.createElement('canvas');
Filters.tmpCtx = Filters.tmpCanvas.getContext('2d');

Filters.createImageData = function(w,h) {
return this.tmpCtx.createImageData(w,h);
};

Filters.convolute = function(pixels, weights, opaque) {
var side = Math.round(Math.sqrt(weights.length));
var halfSide = Math.floor(side/2);
var src = pixels.data;
var sw = pixels.width;
var sh = pixels.height;
// pad output by the convolution matrix
var w = sw;
var h = sh;
var output = Filters.createImageData(w, h);
var dst = output.data;
// go through the destination image pixels
var alphaFac = opaque ? 1 : 0;
for (var y=0; y&lt;h; y++) {
for (var x=0; x&lt;w; x++) {
  var sy = y;
  var sx = x;
  var dstOff = (y*w+x)*4;
  // calculate the weighed sum of the source image pixels that
  // fall under the convolution matrix
  var r=0, g=0, b=0, a=0;
  for (var cy=0; cy&lt;side; cy++) {
    for (var cx=0; cx&lt;side; cx++) {
      var scy = sy + cy - halfSide;
      var scx = sx + cx - halfSide;
      if (scy >= 0 && scy &lt; sh && scx >= 0 && scx &lt; sw) {
        var srcOff = (scy*sw+scx)*4;
        var wt = weights[cy*side+cx];
        r += src[srcOff] * wt;
        g += src[srcOff+1] * wt;
        b += src[srcOff+2] * wt;
        a += src[srcOff+3] * wt;
      }
    }
  }
  dst[dstOff] = r;
  dst[dstOff+1] = g;
  dst[dstOff+2] = b;
  dst[dstOff+3] = a + alphaFac*(255-a);
}
}
return output;
};

Here's a 3x3 sharpen filter. See how it focuses the weight on the center pixel. To maintain the brightness of the image, the sum of the matrix values should be one.

Filters.filterImage(Filters.convolute, image,
[  0, -1,  0,
-1,  5, -1,
  0, -1,  0 ]
);

Here's an another example of a convolution filter, the box blur. The box blur outputs the average of the pixel values inside the convolution matrix. The way to do that is to create a convolution matrix of size NxN where each of the weights is 1 / (NxN). That way each of the pixels inside the matrix contributes an equal amount to the output image and the sum of the weights is one.

Filters.filterImage(Filters.convolute, image,
[ 1/9, 1/9, 1/9,
1/9, 1/9, 1/9,
1/9, 1/9, 1/9 ]
);

We can make more complex image filters by combining existing filters. For example, let's write a Sobel filter. A Sobel filter computes the vertical and horizontal gradients of the image and combines the computed images to find edges in the image. The way we implement the Sobel filter here is by first grayscaling the image, then taking the horizontal and vertical gradients and finally combining the gradient images to make up the final image.

Regarding terminology, "gradient" here means the change in pixel value at an image position. If a pixel has a left neighbour with value 20 and a right neighbour with value 50, the horizontal gradient at the pixel would be 30. The vertical gradient has the same idea but uses the above and below neighbours.

var grayscale = Filters.filterImage(Filter.grayscale, image);
// Note that ImageData values are clamped between 0 and 255, so we need
// to use a Float32Array for the gradient values because they
// range between -255 and 255.
var vertical = Filters.convoluteFloat32(grayscale,
[ -1, 0, 1,
-2, 0, 2,
-1, 0, 1 ]);
var horizontal = Filters.convoluteFloat32(grayscale,
[ -1, -2, -1,
  0,  0,  0,
  1,  2,  1 ]);
var final_image = Filters.createImageData(vertical.width, vertical.height);
for (var i=0; i&lt;final_image.data.length; i+=4) {
// make the vertical gradient red
var v = Math.abs(vertical.data[i]);
final_image.data[i] = v;
// make the horizontal gradient green
var h = Math.abs(horizontal.data[i]);
final_image.data[i+1] = h;
// and mix in some blue for aesthetics
final_image.data[i+2] = (v+h)/4;
final_image.data[i+3] = 255; // opaque alpha
}

And there's a whole bunch of other cool convolution filters out there just waiting for you to discover them. For instance, try implementing a Laplace filter in the convolution toy above and see what it does.

Conclusion #

I hope this small article was useful in introducing the basic concepts of writing image filters in JavaScript using the HTML canvas tag. I encourage you to go and implement some more image filters, it's quite fun!

If you need better performance from your filters, you can usually port them to use WebGL fragment shaders to do the image processing. With shaders, you can run most simple filters in realtime, which allows you to use them for post-processing video and animations.