Gregg Tavares on web.dev

WebGL orthographic 3D

2012-06-20T00:00:00Z

WebGL Orthographic 3D #

This post is a continuation of a series of posts about WebGL. The first started with fundamentals and the previous was about 2d matrices about 2D matrices. If you haven't read those please view them first. In the last post we went over how 2d matrices worked. We talked about translation, rotation, scaling, and even projecting from pixels into clip space can all be done by 1 matrix and some magic matrix math. To do 3D is only a small step from there. In our previous 2D examples we had 2D points (x, y) that we multiplied by a 3x3 matrix. To do 3D we need 3D points (x, y, z) and a 4x4 matrix. Let's take our last example and change it to 3D. We'll use an F again but this time a 3D 'F'. The first thing we need to do is change the vertex shader to handle 3D. Here's the old shader.

<script id="2d-vertex-shader" type="x-shader/x-vertex">
attribute vec2 a_position;

uniform mat3 u_matrix;

void main() {
// Multiply the position by the matrix.
gl_Position = vec4((u_matrix * vec3(a_position, 1)).xy, 0, 1);
}
</script>

And here's the new one

<script id="3d-vertex-shader" type="x-shader/x-vertex">
attribute vec4 a_position;

uniform mat4 u_matrix;

void main() {
// Multiply the position by the matrix.
gl_Position = u_matrix * a_position;
}
</script>

It got even simpler! Then we need to provide 3D data.

...

gl.vertexAttribPointer(positionLocation, 3, gl.FLOAT, false, 0, 0);

...

// Fill the buffer with the values that define a letter 'F'.
function setGeometry(gl) {
gl.bufferData(
    gl.ARRAY_BUFFER,
    new Float32Array([
        // left column
        0,   0,  0,
        30,   0,  0,
        0, 150,  0,
        0, 150,  0,
        30,   0,  0,
        30, 150,  0,

        // top rung
        30,   0,  0,
        100,   0,  0,
        30,  30,  0,
        30,  30,  0,
        100,   0,  0,
        100,  30,  0,

        // middle rung
        30,  60,  0,
        67,  60,  0,
        30,  90,  0,
        30,  90,  0,
        67,  60,  0,
        67,  90,  0]),
    gl.STATIC_DRAW);
}

Next we need to change all the matrix functions from 2D to 3D Here's the 2D (before) versions of makeTranslation, makeRotation and makeScale

function makeTranslation(tx, ty) {
return [
1, 0, 0,
0, 1, 0,
tx, ty, 1
];
}

function makeRotation(angleInRadians) {
var c = Math.cos(angleInRadians);
var s = Math.sin(angleInRadians);
return [
c,-s, 0,
s, c, 0,
0, 0, 1
];
}

function makeScale(sx, sy) {
return [
sx, 0, 0,
0, sy, 0,
0, 0, 1
];
}

And here's the updated 3D versions.

function makeTranslation(tx, ty, tz) {
return [
    1,  0,  0,  0,
    0,  1,  0,  0,
    0,  0,  1,  0,
    tx, ty, tz, 1
];
}

function makeXRotation(angleInRadians) {
var c = Math.cos(angleInRadians);
var s = Math.sin(angleInRadians);

return [
1, 0, 0, 0,
0, c, s, 0,
0, -s, c, 0,
0, 0, 0, 1
];
};

function makeYRotation(angleInRadians) {
var c = Math.cos(angleInRadians);
var s = Math.sin(angleInRadians);

return [
c, 0, -s, 0,
0, 1, 0, 0,
s, 0, c, 0,
0, 0, 0, 1
];
};

function makeZRotation(angleInRadians) {
var c = Math.cos(angleInRadians);
var s = Math.sin(angleInRadians);
return [
    c, s, 0, 0,
-s, c, 0, 0,
    0, 0, 1, 0,
    0, 0, 0, 1,
];
}

function makeScale(sx, sy, sz) {
return [
sx, 0,  0,  0,
0, sy,  0,  0,
0,  0, sz,  0,
0,  0,  0,  1,
];
}

Notice we now have 3 rotations functions. We only needed one in 2D as we were effectively only rotating around the Z axis. Now though to do 3D we also want to be able to rotate around the x axis and y axis as well. You can see from looking at them they are all very similar. If we were to work them out you'd see them simplify just like before

Z rotation

newX = x * c + y * s;
newY = x * -s + y * c;

Y rotation


newX = x * c + z * s;
newZ = x * -s + z * c;

X rotation

newY = y * c + z * s;
newZ = y * -s + z * c;

We also need to update the projection function. Here's the old one

function make2DProjection(width, height) {
// Note: This matrix flips the Y axis so 0 is at the top.
return [
2 / width, 0, 0,
0, -2 / height, 0,
-1, 1, 1
];
}

which converted from pixels to clip space. For our first attempt at expanding it to 3D let's try

function make2DProjection(width, height, depth) {
// Note: This matrix flips the Y axis so 0 is at the top.
return [
    2 / width, 0, 0, 0,
    0, -2 / height, 0, 0,
    0, 0, 2 / depth, 0,
-1, 1, 0, 1,
];
}

Just like we needed to convert from pixels to clipspace for x and y, for z we need to do the same thing. In this case I'm making the Z space pixel units as well. I'll pass in some value similar to width for the depth so our space will be 0 to width pixels wide, 0 to height pixels tall, but for depth it will be -depth / 2 to +depth / 2. Finally we need to to update the code that computes the matrix.

// Compute the matrices
var projectionMatrix =
    make2DProjection(canvas.width, canvas.height, canvas.width);
var translationMatrix =
    makeTranslation(translation[0], translation[1], translation[2]);
var rotationXMatrix = makeXRotation(rotation[0]);
var rotationYMatrix = makeYRotation(rotation[1]);
var rotationZMatrix = makeZRotation(rotation[2]);
var scaleMatrix = makeScale(scale[0], scale[1], scale[2]);

// Multiply the matrices.
var matrix = matrixMultiply(scaleMatrix, rotationZMatrix);
matrix = matrixMultiply(matrix, rotationYMatrix);
matrix = matrixMultiply(matrix, rotationXMatrix);
matrix = matrixMultiply(matrix, translationMatrix);
matrix = matrixMultiply(matrix, projectionMatrix);

// Set the matrix.
gl.uniformMatrix4fv(matrixLocation, false, matrix);

The first problem we have is that our geometry is a flat F which makes it hard to see any 3D. To fix that let's expand the geometry to 3D. Our current F is made of 3 rectangles, 2 triangles each. To make it 3D will require a total of 16 rectangles. That's quite a few to list out here. 16 rectangles x 2 triangles per rectangle x 3 vertices per triangle is 96 vertices. If you want to see all of them view source on the sample. We have to draw more vertices so

// Draw the geometry.
gl.drawArrays(gl.TRIANGLES, 0, 16 * 6);

Moving the sliders it's pretty hard to tell that it's 3D. Let's try coloring each rectangle a different color. To do this we will add another attribute to our vertex shader and a varying to pass it from the vertex shader to the fragment shader. Here's the new vertex shader

<script id="3d-vertex-shader" type="x-shader/x-vertex">
attribute vec4 a_position;
attribute vec4 a_color;

uniform mat4 u_matrix;

varying vec4 v_color;

void main() {
// Multiply the position by the matrix.
gl_Position = u_matrix * a_position;

// Pass the color to the fragment shader.
v_color = a_color;
}
</script>

And we need to use that color in the fragment shader

<script id="3d-vertex-shader" type="x-shader/x-fragment">
precision mediump float;

// Passed in from the vertex shader.
varying vec4 v_color;

void main() {
gl_FragColor = v_color;
}
</script>

We need to lookup the location to supply the colors, then setup another buffer and attribute to give it the colors.

...
var colorLocation = gl.getAttribLocation(program, "a_color");

...
// Create a buffer for colors.
var buffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, buffer);
gl.enableVertexAttribArray(colorLocation);

// We'll supply RGB as bytes.
gl.vertexAttribPointer(colorLocation, 3, gl.UNSIGNED_BYTE, true, 0, 0);

// Set Colors.
setColors(gl);

...
// Fill the buffer with colors for the 'F'.

function setColors(gl) {
gl.bufferData(
    gl.ARRAY_BUFFER,
    new Uint8Array([
        // left column front
    200,  70, 120,
    200,  70, 120,
    200,  70, 120,
    200,  70, 120,
    200,  70, 120,
    200,  70, 120,

        // top rung front
    200,  70, 120,
    200,  70, 120,
    ...
    ...
    gl.STATIC_DRAW);
}

Uh oh, what's that's mess? Well, it turns out all the various parts of that 3D 'F', front, back, sides, etc get drawn in the order they appear in our geometry. That doesn't give us quite the desired results as sometimes the ones in the back get drawn after the ones in the front. Triangles in WebGL have the concept of front facing and back facing. A front facing triangle has its vertices go in a clockwise direction. A back facing triangle has its vertices go in a counter clockwise direction.

WebGL has the ability to draw only forward facing or back facing triangles. We can turn that feature on with

gl.enable(gl.CULL_FACE);

which we do just once, right at the start of our program. With that feature turned on, WebGL defaults to "culling" back facing triangles. "Culling" in this case is a fancy word for "not drawing". Note that as far as WebGL is conserned, whether or not a triangle is considered to be going clockwise or counter clockwise depends on the vertices of that triangle in clipspace. In other words, WebGL figures out whether a triangle is front or back AFTER you've applied math to the vertices in the vertex shader. That means for example a clockwise triangle that is scaled in X by -1 becomes a counter clockwise triangle or a clockwise triangle rotated 180 degrees around the X or Y axis becomes a counter clockwise triangle. Because we had CULL_FACE disabled we can see both clockwise(front) and counter clockwise(back) triangles. Now that we've turned it on, any time a front facing triangle flips around either because of scaling or rotation or for whatever reason, WebGL won't draw it. That's a good thing since as your turn something around in 3D you generally want which ever triangles are facing you to be considered front facing.

Hey! Where did all the triangles go? It turns out, many of them are facing the wrong way. Rotate it and you'll see them appear when you look at the other side. Fortunately it's easy to fix. We just look at which ones are backward and exchange 2 of their vertices. For example if one backward triangle is

1,   2,   3,
40,  50,  60,
700, 800, 900,

we just flip the last 2 vertices to make it forward.

1,   2,   3,
700, 800, 900,
40,  50,  60,

That's closer but there's still one more problem. Even with all the triangles facing in the correct direction and with the back facing ones being culled we still have places where triangles that should be in back are being drawn in over triangles that should be in front. Enter the DEPTH BUFFER. A Depth buffer, sometimes called a Z-Buffer, is a rectangle of depth pixels, one depth pixel for each color pixel used to make the image. As WebGL draws each color pixel it can also draw a depth pixel. It does this based on the values we return from the vertex shader for Z. Just like we had to convert to clip space for X and Y, so to Z is in clip space or (-1 to +1). That value is then converted into a depth space value (0 to +1). Before WebGL draws a color pixel it will check the corresponding depth pixel. If the depth value for the pixel it's about to draw is greater than the value of the corresponding depth pixel then WebGL does not draw the new color pixel. Otherwise it draws both the new color pixel with the color from your fragment shader AND it draws the depth pixel with the new depth value. This means, pixels that are behind other pixels won't get drawn. We can turn on this feature nearly as simply as we turned on culling with

gl.enable(gl.DEPTH_TEST);

We also need to clear the depth buffer back to 1.0 before we start drawing.

// Draw the scene.
function drawScene() {
// Clear the canvas AND the depth buffer.
gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
...

In the next post I'll go over how to make it have perspective.

Why is the attribute vec4 but gl.vertexAttribPointer size 3 #

For those of you who are detail oriented you might have noticed we defined our 2 attributes as

attribute vec4 a_position;
attribute vec4 a_color;

both of which are 'vec4' but when we tell WebGL how to take data out of our buffers we used

gl.vertexAttribPointer(positionLocation, 3, gl.FLOAT, false, 0, 0);
gl.vertexAttribPointer(colorLocation, 3, gl.UNSIGNED_BYTE, true, 0, 0);

That '3' in each of those says only to pull out 3 values per attribute. This works because in the vertex shader WebGL provides defaults for those you don't supply. The defaults are 0, 0, 0, 1 where x = 0, y = 0, z = 0 and w = 1. This is why in our old 2D vertex shader we had to explicitly supply the 1. We were passing in x and y and we needed a 1 for z but because the default for z is 0 we had to explicitly supply a 1. For 3D though, even though we don't supply a 'w' it defaults to 1 which is what we need for the matrix math to work.

WebGL transforms

2012-04-18T00:00:00Z

WebGL 2D Translation #

Before we move on to 3D let's stick with 2D for a little while longer. Bear with me please. This article might seem exceedingly obvious to some but I'll build up to a point in a few articles.

This article is a continuation of a series starting with WebGL Fundamentals. If you haven't read it I suggest you read at least the first chapter then come back here. Translation is some fancy math name that basically means "to move" something. I suppose moving a sentence from English to Japanese fits as well but in this case we're talking about moving geometry. Using the sample code we ended up with in the first post you could easily translate our rectangle just by changing the values passed to setRectangle right? Here's a sample based on our previous sample.

  // First lets make some variables 
  // to hold the translation of the rectangle
  var translation = [0, 0];
  // then let's make a function to
  // re-draw everything. We can call this
  // function after we update the translation.
  // Draw the scene.
  function drawScene() {
     // Clear the canvas.
    gl.clear(gl.COLOR_BUFFER_BIT);
    // Setup a rectangle
    setRectangle(gl, translation[0], translation[1], width, height);

    // Draw the rectangle.
    gl.drawArrays(gl.TRIANGLES, 0, 6);
  }

So far so good. But now imagine we wanted to do the same thing with a more complicated shape. Let's say we wanted to draw an 'F' that consists of 6 triangles like this.

Well, following are current code we'd have to change setRectangle to something more like this.

// Fill the buffer with the values that define a letter 'F'.
function setGeometry(gl, x, y) {
  var width = 100;
  var height = 150;
  var thickness = 30;
  gl.bufferData(
      gl.ARRAY_BUFFER,
      new Float32Array([
          // left column
          x, y,
          x + thickness, y,
          x, y + height,
          x, y + height,
          x + thickness, y,
          x + thickness, y + height,

          // top rung
          x + thickness, y,
          x + width, y,
          x + thickness, y + thickness,
          x + thickness, y + thickness,
          x + width, y,
          x + width, y + thickness,

          // middle rung
          x + thickness, y + thickness * 2,
          x + width * 2 / 3, y + thickness * 2,
          x + thickness, y + thickness * 3,
          x + thickness, y + thickness * 3,
          x + width * 2 / 3, y + thickness * 2,
          x + width * 2 / 3, y + thickness * 3]),
      gl.STATIC_DRAW);
}

You can hopefully see that's not going to scale well. If we want to draw some very complex geometry with hundreds or thousands of lines we'd have to write some pretty complex code. On top of that, every time we draw JavaScript has to update all the points. There's a simpler way. Just upload the geometry and do the translation in the shader. Here's the new shader

<script id="2d-vertex-shader" type="x-shader/x-vertex">
attribute vec2 a_position;

uniform vec2 u_resolution;
uniform vec2 u_translation;

void main() {
   // Add in the translation.
   vec2 position = a_position + u_translation;

   // convert the rectangle from pixels to 0.0 to 1.0
   vec2 zeroToOne = position / u_resolution;
   ...

and we'll restructure the code a little. For one we only need to set the geometry once.

// Fill the buffer with the values that define a letter 'F'.
function setGeometry(gl) {
  gl.bufferData(
      gl.ARRAY_BUFFER,
      new Float32Array([
          // left column
          0, 0,
          30, 0,
          0, 150,
          0, 150,
          30, 0,
          30, 150,

          // top rung
          30, 0,
          100, 0,
          30, 30,
          30, 30,
          100, 0,
          100, 30,

          // middle rung
          30, 60,
          67, 60,
          30, 90,
          30, 90,
          67, 60,
          67, 90]),
      gl.STATIC_DRAW);
}

Then we just need to update u_translation before we draw with the translation that we desire.

  ...
  var translationLocation = gl.getUniformLocation(
             program, "u_translation");
  ...
  // Set Geometry.
  setGeometry(gl);
  ..
  // Draw scene.
  function drawScene() {
    // Clear the canvas.
    gl.clear(gl.COLOR_BUFFER_BIT);

    // Set the translation.
    gl.uniform2fv(translationLocation, translation);

    // Draw the rectangle.
    gl.drawArrays(gl.TRIANGLES, 0, 18);
  }

Notice setGeometry is called only once. It is no longer inside drawScene.

Now when we draw WebGL is doing practically everything. All we are doing is setting a translation and asking it to draw. Even if our geometry had tens of thousands of points the main code would stay the same.

WebGL 2D Rotation #

I'm going to admit right up front I have no idea if how I explain this will make sense but what the heck, might as well try.

First I want to introduce you to what's called a "unit circle". If you remember your junior high school math (don't go to sleep on me!) a circle has a radius. The radius of a circle is the distance from the center of the circle to the edge. A unit circle is a circle with a radius of 1.0.

If you remember from basic 3rd grade math if you multiply something by 1 it stays the same. So 123 * 1 = 123. Pretty basic right? Well, a unit circle, a circle with a radius of 1.0 is also a form of 1. It's a rotating 1. So you can multiply something by this unit circle and in a way it's kind of like multiplying by 1 except magic happens and things rotate. We're going to take that X and Y value from any point on the unit circle and we'll multiply our geometry by them from our previous sample. Here's are the updates to our shader.

<script id="2d-vertex-shader" type="x-shader/x-vertex">
attribute vec2 a_position;

uniform vec2 u_resolution;
uniform vec2 u_translation;
uniform vec2 u_rotation;

void main() {
  // Rotate the position
  vec2 rotatedPosition = vec2(
     a_position.x * u_rotation.y + a_position.y * u_rotation.x,
     a_position.y * u_rotation.y - a_position.x * u_rotation.x);

  // Add in the translation.
  vec2 position = rotatedPosition + u_translation;

And we update the JavaScript so that we can pass those 2 values in.

  ...
  var rotationLocation = gl.getUniformLocation(program, "u_rotation");
  ...
  var rotation = [0, 1];
  ..
  // Draw the scene.
  function drawScene() {
    // Clear the canvas.
    gl.clear(gl.COLOR_BUFFER_BIT);

    // Set the translation.
    gl.uniform2fv(translationLocation, translation);

    // Set the rotation.
    gl.uniform2fv(rotationLocation, rotation);

    // Draw the rectangle.
    gl.drawArrays(gl.TRIANGLES, 0, 18);
  }

Why does it work? Well, look at the math.

rotatedX = a_position.x * u_rotation.y + a_position.y * u_rotation.x;
rotatedY = a_position.y * u_rotation.y - a_position.x * u_rotation.x;

Let's stay you have a rectangle and you want to rotate it. Before you start rotating it the top right corner is at 3.0, 9.0. Let's pick a point on the unit circle 30 degrees clockwise from 12 o'clock.

The position on the circle there is 0.50 and 0.87

3.0 * 0.87 + 9.0 * 0.50 = 7.1
9.0 * 0.87 - 3.0 * 0.50 = 6.3

That's exactly where we need it to be

The same for 60 degrees clockwise

The position on the circle there is 0.87 and 0.50

3.0 * 0.50 + 9.0 * 0.87 = 9.3
9.0 * 0.50 - 3.0 * 0.87 = 1.9

You can see that as we rotate that point clockwise to the right the X value gets bigger and the Y gets smaller. If kept going past 90 degrees X would start getting smaller again and Y would start getting bigger. That pattern gives us rotation. There's another name for the points on a unit circle. They're call the sine and cosine. So for any given angle we can just look up the sine and cosine like this.

function printSineAndCosineForAnAngle(angleInDegrees) {
  var angleInRadians = angleInDegrees * Math.PI / 180;
  var s = Math.sin(angleInRadians);
  var c = Math.cos(angleInRadians);
  console.log("s = " + s + " c = " + c);
}

If you copy and paste the code into your JavaScript console and type printSineAndCosignForAngle(30) you see it prints s = 0.49 c= 0.87 (note: I rounded off the numbers.) If you put it all together you can rotate your geometry to any angle you desire. Just set the rotation to the sine and cosine of the angle you want to rotate to.

  ...
  var angleInRadians = angleInDegrees * Math.PI / 180;
  rotation[0] = Math.sin(angleInRadians);
  rotation[1] = Math.cos(angleInRadians);

I hope that made some sense. Next up a simpler one. Scale.

What are radians? #

Radians are a unit of measurement used with circles, rotation and angles. Just like we can measure distance in inches, yards, meters, etc we can measure angles in degrees or radians.

You're probably aware that math with metric measurements is easier than math with imperial measurements. To go from inches to feet we divide by 12. To go from inches to yards we divide by 36. I don't know about you but I can't divide by 36 in my head. With metric it's much easier. To go from millimeters to centimeters we divide by 10. To from millimeters to meters we divide by 1000. I can divide by 1000 in my head.

Radians vs degrees are similar. Degrees make the math hard. Radians make the math easy. There are 360 degrees in a circle but there are only 2π radians. So a full turn is 2π radians. A half turn is π radians. A 1/4 turn, ie 90 degress is π/2 radian. So if you want to rotate something 90 degrees just use Math.PI * 0.5. If you want to rotate it 45 degrees use Math.PI * 0.25 etc.

Nearly all math involving angles, circles or rotation works very simply if you start thinking in radians. So give it try. Use radians, not degrees except in UI displays.

WebGL 2D Scale #

Scaling is just as easy as translation.

We multiply the position by our desired scale. Here are the changes from our previous sample.

<script id="2d-vertex-shader" type="x-shader/x-vertex">
attribute vec2 a_position;

uniform vec2 u_resolution;
uniform vec2 u_translation;
uniform vec2 u_rotation;
uniform vec2 u_scale;

void main() {
  // Scale the positon
  vec2 scaledPosition = a_position * u_scale;

  // Rotate the position
  vec2 rotatedPosition = vec2(
     scaledPosition.x * u_rotation.y +
        scaledPosition.y * u_rotation.x,
     scaledPosition.y * u_rotation.y -
        scaledPosition.x * u_rotation.x);

  // Add in the translation.
  vec2 position = rotatedPosition + u_translation;

and we add the JavaScript needed to set the scale when we draw.

  ...
  var scaleLocation = gl.getUniformLocation(program, "u_scale");
  ...
  var scale = [1, 1];
  ...
  // Draw the scene.
  function drawScene() {
    // Clear the canvas.
    gl.clear(gl.COLOR_BUFFER_BIT);

    // Set the translation.
    gl.uniform2fv(translationLocation, translation);

    // Set the rotation.
    gl.uniform2fv(rotationLocation, rotation);

    // Set the scale.
    gl.uniform2fv(scaleLocation, scale);

    // Draw the rectangle.
    gl.drawArrays(gl.TRIANGLES, 0, 18);
  }

One thing to notice is that scaling by a negative value flips our geometry. I hope these last 3 chapters were helpful in understanding translation, rotation and scale. Next we'll go over the magic that is matrices that combine all 3 of these into a much simpler and often more useful form.

Why an 'F'? #

The first time I saw someone use a 'F' was on a texture. The 'F' itself is not important. What is important is that you can tell its orientation from any direction. If we used a heart ♥ or a triangle △ for example we couldn't tell if it was flipped horizontally. A circle ○ would be even worse. A colored rectangle would arguably work with different colors on each corner but then you'd have to remember which corner was which. An F's orientation is instantly recognizable.

Any shape that you can tell the orientation of would work, I've just used 'F' ever since I was 'F'irst introduced to the idea.

WebGL 2D Matrices #

In the last 3 chapters we went over how to translate geometry, rotate geometry, and scale geometry. Translation, rotation and scale are each considered a type of 'transformation'. Each of these transformations required changes to the shader and each of the 3 transformations was order dependent.

For example here is a scale of 2, 1, rotation of 30%, and translation of 100, 0.

And here is a translation of 100,0, rotation of 30% and scale of 2, 1

The results are completely different. Even worse, if we needed the second example we'd have to write a different shader that applied the translation, rotation, and scale in our new desired order. Well, some people way smarter than me, figured out that you can do all the same stuff with matrix math. For 2d we use a 3x3 matrix. A 3x3 matrix is like grid with 9 boxes.

1.0	2.0	3.0
4.0	5.0	6.0
7.0	8.0	9.0

To do the math we multiply the position down the columns of the matrix and add up the results. Our positions only have 2 values, x and y but to do this math we need 3 values so we'll use 1 for the third value. in this case our result would be

newX = x * 1.0 + y * 4.0 + 1 * 7.0

newY = x * 2.0 + y * 5.0 + 1 * 8.0

extra = x * 3.0 + y * 6.0 + 1 * 9.0

You're probably looking at that and thinking "WHAT'S THE POINT". Well, Let's assume we have a translation. We'll call the amount we want to translate tx and ty. Let's make a matrix like this

1.0	0.0	0.0
0.0	1.0	0.0
tx	ty	1.0

And now check it out

newX = x * 1.0 + y * 0.0 + 1 * tx

newY = x * 0.0 + y * 1.0 + 1 * ty

extra = x * 0.0 + y * 0.0 + 1 * 1

If you remember your algebra, we can delete any place that multiplies by zero. Multiplying by 1 effectively does nothing so let's simplify to see what's happening

newX = x + tx;
newY = y + ty;

And extra we don't really care about. That looks surprisingly like the translation code from our translation example. Similarly let's do rotation. Like we pointed out in the rotation post we just need the sine and cosine of the angle at which we want to rotate so.

s = Math.sin(angleToRotateInRadians);
c = Math.cos(angleToRotateInRadians);

And we build a matrix like this

c	-s	0.0
s	c	0.0
0.0	0.0	1.0

Applying the matrix we get this

newX = x * c + y * s + 1 * 0

newY = x * -s + y * c + 1 * 0

extra = x * 0.0 + y * 0.0 + 1 * 1

Blacking out all multiply by 0s and 1s we get

newX = x *  c + y * s;
newY = x * -s + y * c;

Which is exactly what we had in our rotation sample. And lastly scale. We'll call our 2 scale factors sx and sy And we build a matrix like this

sx	0.0	0.0
0.0	sy	0.0
0.0	0.0	1.0

Applying the matrix we get this

newX = x * sx + y * 0 + 1 * 0

newY = x * 0 + y * sy + 1 * 0

extra = x * 0.0 + y * 0.0 + 1 * 1

which is really

newX = x * sx;
newY = y * sy;

Which is the same as our scaling sample. Now I'm sure you might still be thinking. So what? What's the point. That seems like a lot of work just to do the same thing we were already doing? This is where the magic comes in. It turns out we can multiply matrices together and apply all the transformations at once. Let's assume we have function, matrixMultiply, that takes two matrices, multiplies them and returns the result. To make things clearer let's make functions to build matrices for translation, rotation and scale.

function makeTranslation(tx, ty) {
  return [
    1, 0, 0,
    0, 1, 0,
    tx, ty, 1
  ];
}

function makeRotation(angleInRadians) {
  var c = Math.cos(angleInRadians);
  var s = Math.sin(angleInRadians);
  return [
    c,-s, 0,
    s, c, 0,
    0, 0, 1
  ];
}

function makeScale(sx, sy) {
  return [
    sx, 0, 0,
    0, sy, 0,
    0, 0, 1
  ];
}

Now let's change our shader. The old shader looked like this

<script id="2d-vertex-shader" type="x-shader/x-vertex">
attribute vec2 a_position;

uniform vec2 u_resolution;
uniform vec2 u_translation;
uniform vec2 u_rotation;
uniform vec2 u_scale;

void main() {
  // Scale the positon
  vec2 scaledPosition = a_position * u_scale;

  // Rotate the position
  vec2 rotatedPosition = vec2(
     scaledPosition.x * u_rotation.y + scaledPosition.y * u_rotation.x,
     scaledPosition.y * u_rotation.y - scaledPosition.x * u_rotation.x);

  // Add in the translation.
  vec2 position = rotatedPosition + u_translation;
  ...

Our new shader will be much simpler.

<script id="2d-vertex-shader" type="x-shader/x-vertex">
attribute vec2 a_position;

uniform vec2 u_resolution;
uniform mat3 u_matrix;

void main() {
  // Multiply the position by the matrix.
  vec2 position = (u_matrix * vec3(a_position, 1)).xy;
  ...

And here's how we use it

  // Draw the scene.
  function drawScene() {
    // Clear the canvas.
    gl.clear(gl.COLOR_BUFFER_BIT);

    // Compute the matrices
    var translationMatrix =
       makeTranslation(translation[0], translation[1]);
    var rotationMatrix = makeRotation(angleInRadians);
    var scaleMatrix = makeScale(scale[0], scale[1]);

    // Multiply the matrices.
    var matrix = matrixMultiply(scaleMatrix, rotationMatrix);
    matrix = matrixMultiply(matrix, translationMatrix);

    // Set the matrix.
    gl.uniformMatrix3fv(matrixLocation, false, matrix);

    // Draw the rectangle.
    gl.drawArrays(gl.TRIANGLES, 0, 18);
  }

Still, you might be asking, so what? That doesn't seem like much of a benefit . But, now if we want to change the order we don't have to write a new shader. We can just change the math.

    ...
    // Multiply the matrices.
    var matrix = matrixMultiply(translationMatrix, rotationMatrix);
    matrix = matrixMultiply(matrix, scaleMatrix);
    ...

Being able to apply matrices like this is especially important for hierarchical animation like arms on a body, moons on a planet around a sun, or branches on a tree. For a simple example of hierarchical animation lets draw draw our 'F' 5 times but each time lets start with the matrix from the previous 'F'.

  // Draw the scene.
  function drawScene() {
    // Clear the canvas.
    gl.clear(gl.COLOR_BUFFER_BIT);

    // Compute the matrices
    var translationMatrix = makeTranslation(translation[0], translation[1]);
    var rotationMatrix = makeRotation(angleInRadians);
    var scaleMatrix = makeScale(scale[0], scale[1]);

    // Starting Matrix.
    var matrix = makeIdentity();

    for (var i = 0; i < 5; ++i) {
      // Multiply the matrices.
      matrix = matrixMultiply(matrix, scaleMatrix);
      matrix = matrixMultiply(matrix, rotationMatrix);
      matrix = matrixMultiply(matrix, translationMatrix);

      // Set the matrix.
      gl.uniformMatrix3fv(matrixLocation, false, matrix);

      // Draw the geometry.
      gl.drawArrays(gl.TRIANGLES, 0, 18);
    }
  }

To do this we had introduce the function, makeIdentity, that makes an identity matrix. An identity matrix is a matrix that effectively represents 1.0 so that if you multiply by the identity nothing happens. Just like

X * 1 = X

so too

matrixX * identity = matrixX

Here's the code to make an identity matrix.

function makeIdentity() {
  return [
    1, 0, 0,
    0, 1, 0,
    0, 0, 1
  ];
}

One more example, In every sample so far our 'F' rotates around its top left corner. This is because the math we are using always rotates around the origin and the top left corner of our 'F' is at the origin, (0, 0) But now, because we can do matrix math and we can choose the order that transforms are applied we can move the origin before the rest of the transforms are applied.

    // make a matrix that will move the origin of the 'F' to
    // its center.
    var moveOriginMatrix = makeTranslation(-50, -75);
    ...

    // Multiply the matrices.
    var matrix = matrixMultiply(moveOriginMatrix, scaleMatrix);
    matrix = matrixMultiply(matrix, rotationMatrix);
    matrix = matrixMultiply(matrix, translationMatrix);

Using that technique you can rotate or scale from any point. Now you know how Photoshop or Flash let you move the rotation point. Let's go even more crazy. If you go back to the first article on WebGL fundamentals you might remember we have code in the shader to convert from pixels to clipspace that looks like this.

  ...
  // convert the rectangle from pixels to 0.0 to 1.0
  vec2 zeroToOne = position / u_resolution;

  // convert from 0->1 to 0->2
  vec2 zeroToTwo = zeroToOne * 2.0;

  // convert from 0->2 to -1->+1 (clipspace)
  vec2 clipSpace = zeroToTwo - 1.0;

  gl_Position = vec4(clipSpace * vec2(1, -1), 0, 1);

If you look at each of those steps in turn, the first step, "convert from pixels to 0.0 to 1.0", is really a scale operation. The second is also a scale operation. The next is a translation and the very last scales Y by -1. We can actually do that all in the matrix we pass into the shader. We could make 2 scale matrices, one to scale by 1.0/resolution, another to scale by 2.0, a 3rd to translate by -1.0,-1.0 and a 4th to scale Y by -1 then multiply them all together but instead, because the math is simple, we'll just make a function that makes a 'projection' matrix for a given resolution directly.

function make2DProjection(width, height) {
  // Note: This matrix flips the Y axis so that 0 is at the top.
  return [
    2 / width, 0, 0,
    0, -2 / height, 0,
    -1, 1, 1
  ];
}

Now we can simplify the shader even more. Here's the entire new vertex shader.

<script id="2d-vertex-shader" type="x-shader/x-vertex">
attribute vec2 a_position;

uniform mat3 u_matrix;

void main() {
  // Multiply the position by the matrix.
  gl_Position = vec4((u_matrix * vec3(a_position, 1)).xy, 0, 1);
}
</script>

And in JavaScript we need to multiply by the projection matrix

  // Draw the scene.
  function drawScene() {
    ...
    // Compute the matrices
    var projectionMatrix =
       make2DProjection(canvas.width, canvas.height);
    ...

    // Multiply the matrices.
    var matrix = matrixMultiply(scaleMatrix, rotationMatrix);
    matrix = matrixMultiply(matrix, translationMatrix);
    matrix = matrixMultiply(matrix, projectionMatrix);
    ...
  }

We also removed the code that set the resolution. With this last step we've gone from a rather complicated shader with 6-7 steps to a very simple shader with only 1 step all do to the magic of matrix math.

I hope this article has helped demystify matrix math. I'll move on to 3D next. In 3D matrix math follows the same principles and usage. I started with 2D to hopefully keep it simple to understand.

WebGL Fundamentals

2012-02-09T00:00:00Z

WebGL Fundamentals #

WebGL makes it possible to display amazing realtime 3D graphics in your browser but what many people don't know is that WebGL is actually a 2D API, not a 3D API. Let me explain.

WebGL only cares about 2 things. Clipspace coordinates in 2D and colors. Your job as a programmer using WebGL is to provide WebGL with those 2 things. You provide 2 "shaders" to do this. A Vertex shader which provides the clipspace coordinates and a fragment shader that provides the color. Clipspace coordinates always go from -1 to +1 no matter what size your canvas is. Here is a simple WebGL example that shows WebGL in its simplest form.

// Get A WebGL context
var canvas = document.getElementById("canvas");
var gl = canvas.getContext("experimental-webgl");

// setup a GLSL program
var vertexShader = createShaderFromScriptElement(gl, "2d-vertex-shader");
var fragmentShader = createShaderFromScriptElement(gl, "2d-fragment-shader");
var program = createProgram(gl, [vertexShader, fragmentShader]);
gl.useProgram(program);

// look up where the vertex data needs to go.
var positionLocation = gl.getAttribLocation(program, "a_position");

// Create a buffer and put a single clipspace rectangle in
// it (2 triangles)
var buffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, buffer);
gl.bufferData(
    gl.ARRAY_BUFFER,
    new Float32Array([
        -1.0, -1.0,
         1.0, -1.0,
        -1.0,  1.0,
        -1.0,  1.0,
         1.0, -1.0,
         1.0,  1.0]),
    gl.STATIC_DRAW);
gl.enableVertexAttribArray(positionLocation);
gl.vertexAttribPointer(positionLocation, 2, gl.FLOAT, false, 0, 0);

// draw
gl.drawArrays(gl.TRIANGLES, 0, 6);

Here's the 2 shaders

<script id="2d-vertex-shader" type="x-shader/x-vertex">
attribute vec2 a_position;

void main() {
  gl_Position = vec4(a_position, 0, 1);
}
</script>

<script id="2d-fragment-shader" type="x-shader/x-fragment">
void main() {
  gl_FragColor = vec4(0,1,0,1);  // green
}
</script>

Again, clipspace coordinates always go from -1 to +1 regardless of the size of the canvas. In the case above you can see we are doing nothing but passing on our position data directly. Since the position data is already in clipspace there is no work to do. If you want 3D it's up to you to supply shaders that convert from 3D to 2D because WebGL IS A 2D API! For 2D stuff you would probably rather work in pixels than clipspace so let's change the shader so we can supply rectangles in pixels and have it convert to clipspace for us. Here's the new vertex shader

<script id="2d-vertex-shader" type="x-shader/x-vertex">
attribute vec2 a_position;

uniform vec2 u_resolution;

void main() {
   // convert the rectangle from pixels to 0.0 to 1.0
   vec2 zeroToOne = a_position / u_resolution;

   // convert from 0->1 to 0->2
   vec2 zeroToTwo = zeroToOne * 2.0;

   // convert from 0->2 to -1->+1 (clipspace)
   vec2 clipSpace = zeroToTwo - 1.0;

   gl_Position = vec4(clipSpace, 0, 1);
}
</script>

Now we can change our data from clipspace to pixels

// set the resolution
var resolutionLocation = gl.getUniformLocation(program, "u_resolution");
gl.uniform2f(resolutionLocation, canvas.width, canvas.height);

// setup a rectangle from 10,20 to 80,30 in pixels
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([
    10, 20,
    80, 20,
    10, 30,
    10, 30,
    80, 20,
    80, 30]), gl.STATIC_DRAW);

You might notice the rectangle is near the bottom of that area. WebGL considers the bottom left corner to be 0,0. To get it to be the more traditional top left corner used for 2d graphics APIs we just flip the y coordinate.

gl_Position = vec4(clipSpace * vec2(1, -1), 0, 1);

Let's make the code that defines a rectangle into a function so we can call it for different sized rectangles. While we're at it we'll make the color settable. First we make the fragment shader take a color uniform input.

<script id="2d-fragment-shader" type="x-shader/x-fragment">
precision mediump float;

uniform vec4 u_color;

void main() {
   gl_FragColor = u_color;
}
</script>

And here's the new code that draws 50 rectangles in random places and random colors.

...

  var colorLocation = gl.getUniformLocation(program, "u_color");
  ...
  // Create a buffer
  var buffer = gl.createBuffer();
  gl.bindBuffer(gl.ARRAY_BUFFER, buffer);
  gl.enableVertexAttribArray(positionLocation);
  gl.vertexAttribPointer(positionLocation, 2, gl.FLOAT, false, 0, 0);

  // draw 50 random rectangles in random colors
  for (var ii = 0; ii < 50; ++ii) {
    // Setup a random rectangle
    setRectangle(
        gl, randomInt(300), randomInt(300), randomInt(300), randomInt(300));

    // Set a random color.
    gl.uniform4f(colorLocation, Math.random(), Math.random(), Math.random(), 1);

    // Draw the rectangle.
    gl.drawArrays(gl.TRIANGLES, 0, 6);
  }
}

// Returns a random integer from 0 to range - 1.
function randomInt(range) {
  return Math.floor(Math.random() * range);
}

// Fills the buffer with the values that define a rectangle.
function setRectangle(gl, x, y, width, height) {
  var x1 = x;
  var x2 = x + width;
  var y1 = y;
  var y2 = y + height;
  gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([
     x1, y1,
     x2, y1,
     x1, y2,
     x1, y2,
     x2, y1,
     x2, y2]), gl.STATIC_DRAW);
}

I hope you can see that WebGL is actually a pretty simple API. While it can get more complicated to do 3D that complication is added by you, the programmer, in the form of more complex shaders. The WebGL API itself is 2D and fairly simple.

What do type="x-shader/x-vertex" and type="x-shader/x-fragment" mean? #

<script> tags default to having JavaScript in them. You can put no type or you can put type="javascript" or type="text/javascript" and the browser will interpret the contents as JavaScript. If you put anything else the browser ignores the contents of the script tag.

We can use this feature to store shaders in script tags. Even better, we can make up our own type and in our javascript look for that to decide whether to compile the shader as a vertex shader or a fragment shader.

In this case the function createShaderFromScriptElement looks for a script with specified id and then looks at the type to decide what type of shader to create.

WebGL Image Processing #

Image processing is easy in WebGL. How easy? Read below.

To draw images in WebGL we need to use textures. Similarly to the way WebGL expects clipspace coordinates when rendering instead of pixels, WebGL expects texture coordinates when reading a texture. Texture coordinates go from 0.0 to 1.0 no matter the dimensions of the texture. Since we are only drawing a single rectangle (well, 2 triangles) we need to tell WebGL which place in the texture each point in the rectangle corresponds to. We'll pass this information from the vertex shader to the fragment shader using a special kind of variable called a 'varying'. It's called a varying because it varies. WebGL will interpolate the values we provide in the vertex shader as it draws each pixel using the fragment shader. Using the vertex shader from the end of previous section we need to add an attribute to pass in texture coordinates and then pass those on to the fragment shader.

attribute vec2 a_texCoord;
...
varying vec2 v_texCoord;

void main() {
   ...
   // pass the texCoord to the fragment shader
   // The GPU will interpolate this value between points
   v_texCoord = a_texCoord;
}

Then we supply a fragment shader to look up colors from the texture.

<script id="2d-fragment-shader" type="x-shader/x-fragment">
precision mediump float;

// our texture
uniform sampler2D u_image;

// the texCoords passed in from the vertex shader.
varying vec2 v_texCoord;

void main() {
   // Look up a color from the texture.
   gl_FragColor = texture2D(u_image, v_texCoord);
}
</script>

Finally we need to load an image, create a texture and copy the image into the texture. Because we are in a browser images load asynchronously so we have to re-arrange our code a little to wait for the texture to load. Once it loads we'll draw it.

function main() {
  var image = new Image();
  image.src = "http://someimage/on/our/server";  // MUST BE SAME DOMAIN!!!
  image.onload = function() {
    render(image);
  }
}

function render(image) {
  ...
  // all the code we had before.
  ...
  // look up where the texture coordinates need to go.
  var texCoordLocation = gl.getAttribLocation(program, "a_texCoord");

  // provide texture coordinates for the rectangle.
  var texCoordBuffer = gl.createBuffer();
  gl.bindBuffer(gl.ARRAY_BUFFER, texCoordBuffer);
  gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([
      0.0,  0.0,
      1.0,  0.0,
      0.0,  1.0,
      0.0,  1.0,
      1.0,  0.0,
      1.0,  1.0]), gl.STATIC_DRAW);
  gl.enableVertexAttribArray(texCoordLocation);
  gl.vertexAttribPointer(texCoordLocation, 2, gl.FLOAT, false, 0, 0);

  // Create a texture.
  var texture = gl.createTexture();
  gl.bindTexture(gl.TEXTURE_2D, texture);

  // Set the parameters so we can render any size image.
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);

  // Upload the image into the texture.
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, image);
  ...
}

Not too exciting so let's manipulate that image. How about just swapping red and blue?

...
gl_FragColor = texture2D(u_image, v_texCoord).bgra;
...

What if we want to do image processing that actually looks at other pixels? Since WebGL references textures in texture coordinates which go from 0.0 to 1.0 then we can calculate how much to move for 1 pixel with the simple math onePixel = 1.0 / textureSize. Here's a fragment shader that averages the left and right pixels of each pixel in the texture.

<script id="2d-fragment-shader" type="x-shader/x-fragment">
precision mediump float;

// our texture
uniform sampler2D u_image;
uniform vec2 u_textureSize;

// the texCoords passed in from the vertex shader.
varying vec2 v_texCoord;

void main() {
   // compute 1 pixel in texture coordinates.
   vec2 onePixel = vec2(1.0, 1.0) / u_textureSize;

   // average the left, middle, and right pixels.
   gl_FragColor = (
       texture2D(u_image, v_texCoord) +
       texture2D(u_image, v_texCoord + vec2(onePixel.x, 0.0)) +
       texture2D(u_image, v_texCoord + vec2(-onePixel.x, 0.0))) / 3.0;
}
</script>

We then need to pass in the size of the texture from JavaScript.

...
var textureSizeLocation = gl.getUniformLocation(program, "u_textureSize");
...
// set the size of the image
gl.uniform2f(textureSizeLocation, image.width, image.height);
...

Now that we know how to reference other pixels let's use a convolution kernel to do a bunch of common image processing. In this case we'll use a 3x3 kernel. A convolution kernel is just a 3x3 matrix where each entry in the matrix represents how much to multiply the 8 pixels around the pixel we are rendering. We then divide the result by the weight of the kernel (the sum of all values in the kernel) or 1.0, which ever is greater. Here's a pretty good article on it. And here's another article showing some actual code if you were to write this by hand in C++. In our case we're going to do that work in the shader so here's the new fragment shader.

<script id="2d-fragment-shader" type="x-shader/x-fragment">
precision mediump float;

// our texture
uniform sampler2D u_image;
uniform vec2 u_textureSize;
uniform float u_kernel[9];

// the texCoords passed in from the vertex shader.
varying vec2 v_texCoord;

void main() {
   vec2 onePixel = vec2(1.0, 1.0) / u_textureSize;
   vec4 colorSum =
     texture2D(u_image, v_texCoord + onePixel * vec2(-1, -1)) * u_kernel[0] +
     texture2D(u_image, v_texCoord + onePixel * vec2( 0, -1)) * u_kernel[1] +
     texture2D(u_image, v_texCoord + onePixel * vec2( 1, -1)) * u_kernel[2] +
     texture2D(u_image, v_texCoord + onePixel * vec2(-1,  0)) * u_kernel[3] +
     texture2D(u_image, v_texCoord + onePixel * vec2( 0,  0)) * u_kernel[4] +
     texture2D(u_image, v_texCoord + onePixel * vec2( 1,  0)) * u_kernel[5] +
     texture2D(u_image, v_texCoord + onePixel * vec2(-1,  1)) * u_kernel[6] +
     texture2D(u_image, v_texCoord + onePixel * vec2( 0,  1)) * u_kernel[7] +
     texture2D(u_image, v_texCoord + onePixel * vec2( 1,  1)) * u_kernel[8] ;
   float kernelWeight =
     u_kernel[0] +
     u_kernel[1] +
     u_kernel[2] +
     u_kernel[3] +
     u_kernel[4] +
     u_kernel[5] +
     u_kernel[6] +
     u_kernel[7] +
     u_kernel[8] ;

   if (kernelWeight <= 0.0) {
     kernelWeight = 1.0;
   }

   // Divide the sum by the weight but just use rgb
   // we'll set alpha to 1.0
   gl_FragColor = vec4((colorSum / kernelWeight).rgb, 1.0);
}
</script>

In JavaScript we need to supply a convolution kernel.

...
var kernelLocation = gl.getUniformLocation(program, "u_kernel[0]");
...
var edgeDetectKernel = [
    -1, -1, -1,
    -1,  8, -1,
    -1, -1, -1
];
gl.uniform1fv(kernelLocation, edgeDetectKernel);
...

I hope this has convinced you image processing in WebGL is pretty simple. Next up I'll go over how to apply more than one effect to the image.

What's with the a_, u_, and v_ prefixes in from of variables in GLSL? #

That's just a naming convention. a_ for attributes which is the data provided by buffers. u_ for uniforms which are inputs to the shaders, v_ for varyings which are values passed from a vertex shader to a fragment shader and interpolated (or varied) between the vertices for each pixel drawn.

Applying multiple effects #

The next most obvious question for image processing is how do apply multiple effects?

Well, you could try to generate shaders on the fly. Provide a UI that lets the user select the effects he wants to use then generate a shader that does all of the effects. That might not always be possible though that technique is often used to create effects for real time graphics. A more flexible way is to use 2 more textures and render to each texture in turn, ping ponging back and forth and applying the next effect each time.

Original Image -> [Blur]        -> Texture 1
Texture 1      -> [Sharpen]     -> Texture 2
Texture 2      -> [Edge Detect] -> Texture 1
Texture 1      -> [Blur]        -> Texture 2
Texture 2      -> [Normal]      -> Canvas

To do this we need to create framebuffers. In WebGL and OpenGL, a Framebuffer is actually a poor name. A WebGL/OpenGL Framebuffer is really just a collection of state and not actually a buffer of any kind. But, by attaching a texture to a framebuffer we can render into that texture. First let's turn the old texture creation code into a function

function createAndSetupTexture(gl) {
  var texture = gl.createTexture();
  gl.bindTexture(gl.TEXTURE_2D, texture);

  // Set up texture so we can render any size image and so we are
  // working with pixels.
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);

  return texture;
}

// Create a texture and put the image in it.
var originalImageTexture = createAndSetupTexture(gl);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, image);

And now let's use that function to make 2 more textures and attach them to 2 framebuffers.

// create 2 textures and attach them to framebuffers.
var textures = [];
var framebuffers = [];
for (var ii = 0; ii < 2; ++ii) {
  var texture = createAndSetupTexture(gl);
  textures.push(texture);

  // make the texture the same size as the image
  gl.texImage2D(
      gl.TEXTURE_2D, 0, gl.RGBA, image.width, image.height, 0,
      gl.RGBA, gl.UNSIGNED_BYTE, null);

  // Create a framebuffer
  var fbo = gl.createFramebuffer();
  framebuffers.push(fbo);
  gl.bindFramebuffer(gl.FRAMEBUFFER, fbo);

  // Attach a texture to it.
  gl.framebufferTexture2D(
      gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, texture, 0);
}

Now let's make a set of kernels and then a list of them to apply.

// Define several convolution kernels
var kernels = {
  normal: [
    0, 0, 0,
    0, 1, 0,
    0, 0, 0
  ],
  gaussianBlur: [
    0.045, 0.122, 0.045,
    0.122, 0.332, 0.122,
    0.045, 0.122, 0.045
  ],
  unsharpen: [
    -1, -1, -1,
    -1,  9, -1,
    -1, -1, -1
  ],
  emboss: [
     -2, -1,  0,
     -1,  1,  1,
      0,  1,  2
  ]
};

// List of effects to apply.
var effectsToApply = [
  "gaussianBlur",
  "emboss",
  "gaussianBlur",
  "unsharpen"
];

And finally let's apply each one, ping ponging which texture we are rendering too

// start with the original image
gl.bindTexture(gl.TEXTURE_2D, originalImageTexture);

// don't y flip images while drawing to the textures
gl.uniform1f(flipYLocation, 1);

// loop through each effect we want to apply.
for (var ii = 0; ii < effectsToApply.length; ++ii) {
  // Setup to draw into one of the framebuffers.
  setFramebuffer(framebuffers[ii % 2], image.width, image.height);

  drawWithKernel(effectsToApply[ii]);

  // for the next draw, use the texture we just rendered to.
  gl.bindTexture(gl.TEXTURE_2D, textures[ii % 2]);
}

// finally draw the result to the canvas.
gl.uniform1f(flipYLocation, -1);  // need to y flip for canvas
setFramebuffer(null, canvas.width, canvas.height);
drawWithKernel("normal");

function setFramebuffer(fbo, width, height) {
  // make this the framebuffer we are rendering to.
  gl.bindFramebuffer(gl.FRAMEBUFFER, fbo);

  // Tell the shader the resolution of the framebuffer.
  gl.uniform2f(resolutionLocation, width, height);

  // Tell webgl the viewport setting needed for framebuffer.
  gl.viewport(0, 0, width, height);
}

function drawWithKernel(name) {
  // set the kernel
  gl.uniform1fv(kernelLocation, kernels[name]);

  // Draw the rectangle.
  gl.drawArrays(gl.TRIANGLES, 0, 6);
}

Some things I should go over.

Calling gl.bindFramebuffer with null tells WebGL you want to render to the canvas instead of to one of your framebuffers. WebGL has to convert from clipspace back into pixels. It does this based on the settings of gl.viewport. The settings of gl.viewport default to the size of the canvas when we initialize WebGL. Since the framebuffers we are rendering into are a different size then the canvas we need to set the viewport appropriately. Finally in the WebGL fundamentals examples we flipped the Y coordinate when rendering because WebGL displays the canvas with 0,0 being the bottom left corner instead of the more traditional for 2D top left. That's not needed when rendering to a framebuffer. Because the framebuffer is never displayed, which part is top and bottom is irrelevant. All that matters is that pixel 0,0 in the framebuffer corresponds to 0,0 in our calculations. To deal with this I made it possible to set whether to flip or not by adding one more input into the shader.

<script id="2d-vertex-shader" type="x-shader/x-vertex">
...
uniform float u_flipY;
...

void main() {
   ...
   gl_Position = vec4(clipSpace * vec2(1, u_flipY), 0, 1);
   ...
}
</script>

And then we can set it when we render with

...
var flipYLocation = gl.getUniformLocation(program, "u_flipY");
...
// don't flip
gl.uniform1f(flipYLocation, 1);
...
// flip
gl.uniform1f(flipYLocation, -1);

I kept this example simple by using single GLSL program that can achieve multiple effects. If you wanted to do full on image processing you'd probably need many GLSL programs. A program for hue, saturation and luminance adjustment. Another for brightness and contrast. One for inverting, another for adjusting levels, etc. You'd need to change the code to switch GLSL programs and update the parameters for that particular program. I'd considered writing that example but it's an exercise best left to the reader because multiple GLSL programs each with their own parameter needs probably means some major refactoring to keep it all from becoming a big mess of spaghetti. I hope this and the preceding examples have made WebGL seem a little more approachable and I hope starting with 2D helps make WebGL a little easier to understand. If I find the time I'll try to write a few more articles about how to do 3D as well as more details on what WebGL is really doing under the hood.

WebGL and Alpha #

I've noticed some OpenGL developers having issues with how WebGL treats alpha in the backbuffer (ie, the canvas), so I thought it might be good to go over some of the differences between WebGL and OpenGL related to alpha.

The biggest difference between OpenGL and WebGL is that OpenGL renders to a backbuffer that is not composited with anything so, or effectively not composited with anything by the OS's window manager, so it doesn't matter what your alpha is. WebGL is composited by the browser with the web page and the default is to use pre-multiplied alpha the same as .png <img> tags with transparency and 2d canvas tags. WebGL has several ways to make this more like OpenGL.

#1) Tell WebGL you want it composited with non-premultiplied alpha #

gl = canvas.getContext("experimental-webgl", {premultipliedAlpha: false});

The default is true. Of course the result will still be composited over page with whatever background color ends up being under the canvas (the canvas's background color, the canvas's container background color, the page's background color, the stuff behind the canvas if the canvas has a z-index > 0, etc…) in other words, the color CSS defines for that area of the webpage. I really good way to find if you have any alpha problems is to set the canvas's background to a bright color like red. You'll immediately see what is happening.

<canvas style="background: red;"></canvas>

You could also set it to black which will hide any alpha issues you have.

#2) Tell WebGL you don't want alpha in the backbuffer #

gl = canvas.getContext("experimental-webgl", {alpha: false});

This will make it act more like OpenGL since the backbuffer will only have RGB. This is probably the best option because a good browser could see that you have no alpha and actually optimize the way WebGL is composited. Of course that also means it actually won't have alpha in the backbuffer so if you are using alpha in the backbuffer for some purpose that might not work for you. Few apps that I know of use alpha in the backbuffer. I think arguably this should have been the default.

#3) Clear alpha at the end of your rendering #

..
renderScene();
..
// Set the backbuffer's alpha to 1.0
gl.clearColor(1, 1, 1, 1);
gl.colorMask(false, false, false, true);
gl.clear(gl.COLOR_BUFFER_BIT);

Clearing is generally very fast as there is a special case for it in most hardware. I did this in most of my demos. If I was smart I'd switch to method #2 above. Maybe I'll do that right after I post this. It seems like most WebGL libraries should default to this method. Those few developers that are actually using alpha for compositing effects can ask for it. The rest will just get the best perf and the least surprises.

#4) Clear the alpha once then don't render to it anymore #

// At init time. Clear the back buffer.
gl.clearColor(1,1,1,1);
gl.clear(gl.COLOR_BUFFER_BIT);

// Turn off rendering to alpha
gl.colorMask(true, true, true, false);

Of course if you are rendering to your own framebuffers you may need to turn rendering to alpha back on and then turn it off again when you switch to rendering to the canvas.

#5) Handling Images #

Also, if you are loading PNG files with alpha into textures, the default is that their alpha is pre-multiplied which is generally NOT the way most games do things. If you want to prevent that behavior you need to tell WebGL with

gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, false);

#6) Using a blending equation that works with pre-multiplied alpha #

Almost all OpenGL apps I've writing or worked on use

gl.blendFunc(gl.SRC_ALPHA, gl_ONE_MINUS_SRC_ALPHA);

That works for non pre-multiplied alpha textures. If you actually want to work with pre-multiplied alpha textures then you probably want

gl.blendFunc(gl.ONE, gl_ONE_MINUS_SRC_ALPHA);

Those are the methods I'm aware of. If you know of more please post them below.