Mat Scales on web.dev

Recording Video from the User

2017-09-24T00:00:00Z

Start simple and progressively #

The easiest thing to do is simply ask the user for a pre-recorded file. Do this by creating a simple file input element and adding an accept filter that indicates we can only accept video files and a capture attribute that indicates we want to get it direct from the camera.

<input type="file" accept="video/*" capture />

This method works on all platforms. On desktop it will prompt the user to upload a file from the file system (ignoring the capture attribute). In Safari on iOS it will open up the camera app, allowing you to record video and then send it back to the web page; on Android it will give the user the choice of which app to use record the video in before sending it back to the web page.

Many mobile devices have more than one camera. If you have a preference, you can set the capture attribute to user, if you want the camera that faces the user, or environment if you want the camera that faces outward.

<input type="file" accept="video/*" capture="user" />
<input type="file" accept="video/*" capture="environment" />

Note that this is just a hint - if the browser doesn't support the option, or the camera type you ask for isn't available, the browser may choose another camera.

Once the user has finished recording and they are back in the website, you need to somehow get ahold of the file data. You can get quick access by attaching an onchange event to the input element and then reading the files property of the event object.

<input type="file" accept="video/*" capture="camera" id="recorder" />
<video id="player" controls></video>
<script>
  var recorder = document.getElementById('recorder');
  var player = document.getElementById('player');

  recorder.addEventListener('change', function (e) {
    var file = e.target.files[0];
    // Do something with the video file.
    player.src = URL.createObjectURL(file);
  });
</script>

Once you have access to the file you can do anything you want with it. For example, you can:

Attach it directly to a <video> element so that you can play it
Download it to the user's device
Upload it to a server by attaching to an XMLHttpRequest
Draw the frames into a canvas and apply filters to it

Whilst using the input element method of getting access to video data is ubiquitous, it is the least appealing option. We really want to get access to the camera and provide a nice experience directly in the page.

Access the camera interactively #

Modern browsers can have a direct line to the camera allowing us to build experiences that are fully integrated with the web page and the user will never leave the browser.

Acquire access to the camera #

We can directly access the camera by using an API in the WebRTC specification called getUserMedia(). getUserMedia() will prompt the user for access to their connected microphones and cameras.

If successful the API will return a Stream that will contain the data from either the camera or the microphone, and we can then either attach it to a <video> element, attach it to a WebRTC stream, or save it using the MediaRecorder API.

To get data from the camera we just set video: true in the constraints object that is passed to the getUserMedia() API

<video id="player" controls></video>
<script>
  var player = document.getElementById('player');

  var handleSuccess = function (stream) {
    player.srcObject = stream;
  };

  navigator.mediaDevices
    .getUserMedia({audio: true, video: true})
    .then(handleSuccess);
</script>

If you want to choose a particular camera you can first enumerate the available cameras.

navigator.mediaDevices.enumerateDevices().then((devices) => {
  devices = devices.filter((d) => d.kind === 'videoinput');
});

You can then pass the deviceId that you wish to use when you call getUserMedia.

navigator.mediaDevices.getUserMedia({
  audio: true,
  video: {
    deviceId: devices[0].deviceId,
  },
});

By itself, this isn't that useful. All we can do is take the video data and play it back.

Access the raw data from the camera #

To access the raw video data from the camera you can draw each frame into a <canvas> and manipulate the pixels directly.

For a 2D canvas you can use the drawImage method of the context to draw the current frame of a <video> element into the canvas.

context.drawImage(myVideoElement, 0, 0);

With a WebGL canvas you can use a <video> element as the source for a texture.

gl.texImage2D(
  gl.TEXTURE_2D,
  0,
  gl.RGBA,
  gl.RGBA,
  gl.UNSIGNED_BYTE,
  myVideoElement,
);

Note that in either case this will use the current frame of a playing video. To process multiple frames you need to redraw the video to the canvas each time.

You can learn more about this in our article about applying real-time effects to images and video.

Save the data from the camera #

The easiest way to save the data from the camera is to use the MediaRecorder API.

The MediaRecorder API will take the stream created by getUserMedia and then progressively save the data from the stream to you preferred destination.

<a id="download">Download</a>
<button id="stop">Stop</button>
<script>
  let shouldStop = false;
  let stopped = false;
  const downloadLink = document.getElementById('download');
  const stopButton = document.getElementById('stop');

  stopButton.addEventListener('click', function() {
    shouldStop = true;
  })

  var handleSuccess = function(stream) {
    const options = {mimeType: 'video/webm'};
    const recordedChunks = [];
    <strong>const mediaRecorder = new MediaRecorder(stream, options);

    mediaRecorder.addEventListener('dataavailable', function(e) {
      if (e.data.size > 0) {
        recordedChunks.push(e.data);
      }

      if(shouldStop === true && stopped === false) {
        mediaRecorder.stop();
        stopped = true;
      }
    });

    mediaRecorder.addEventListener('stop', function() {
      downloadLink.href = URL.createObjectURL(new Blob(recordedChunks));
      downloadLink.download = 'acetest.webm';
    });

    mediaRecorder.start();</strong>
  };

  navigator.mediaDevices.getUserMedia({ audio: true, video: true })
      .then(handleSuccess);
</script>

In our case we are saving the data directly into an array that we can later turn in to a Blob which can be then used to save to our Web Server or directly in storage on the user's device.

Ask permission to use camera responsibly #

If the user has not previously granted your site access to the camera then the instant that you call getUserMedia the browser will prompt the user to grant your site permission to the camera.

User's hate getting prompted for access to powerful devices on their machine and they will frequently block the request, or they will ignore it if they don't understand the context of which the prompt has been created. It is best practice to only ask to access the camera when first needed. Once the user has granted access they won't be asked again, however, if they reject access, you can't get access again to ask the user for permission.

Warning: Asking for access to the camera on page load will result in most of your users rejecting access.

Use the permissions API to check if you already have access #

The getUserMedia API provides you with no knowledge of if you already have access to the camera. This presents you with a problem, to provide a nice UI to get the user to grant you access to the camera, you have to ask for access to camera.

This can be solved in some browsers by using the Permission API. The navigator.permission API allows you to query the state of the ability to access specific API's without having to prompt again.

To query if you have access to the user's camera you can pass in {name: 'camera'} into the query method and it will return either:

granted — the user has previously given you access to the camera;
prompt — the user has not given you access and will be prompted when you call getUserMedia;
denied — the system or the user has explicitly blocked access to the camera and you won't be able to get access to it.

And you can now check quickly check to see if you need to alter your user interface to accommodate the actions that the user needs to take.

navigator.permissions.query({name: 'camera'}).then(function (result) {
  if (result.state == 'granted') {
  } else if (result.state == 'prompt') {
  } else if (result.state == 'denied') {
  }
  result.onchange = function () {};
});

Feedback #

Real-Time Effects For Images and Video

2017-08-08T00:00:00Z

Many of today's most popular apps let you apply filters and effects to images or video. This article shows how to implement these features on the open web.

The process is basically the same for videos and images, but I'll cover some important video considerations at the end. Throughout the article you can assume that 'image' means 'image or a single frame of a video'

How to get at the pixel data for an image #

There are 3 basic categories of image manipulation that are common:

Pixel effects like contrast, brightness, warmth, sepia tone, saturation.
Multi-pixel effects, called convolution filters, like sharpening, edge detection, blur.
Whole image distortion, like cropping, skewing, stretching, lens effects, ripples.

All of these involve getting at the actual pixel data of the source image and then creating a new image from it, and the only interface for doing that is a canvas.

The really important choice, then, is whether to do the processing on the CPU, with a 2D canvas, or on the GPU, with WebGL.

Let's take a quick look at the differences between the two approaches.

2D canvas #

This is definitely, by a long way, the simplest of the two options. First you draw the image on the canvas.

const source = document.getElementById('source-image');

// Create the canvas and get a context
const canvas = document.createElement('canvas');
const context = canvas.getContext('2d');

// Set the canvas to be the same size as the original image
canvas.width = source.naturalWidth;
canvas.height = source.naturalHeight;

// Draw the image onto the top-left corner of the canvas
context.drawImage(theOriginalImage, 0, 0);

Then you get an array of pixel values for the whole canvas.

const imageData = context.getImageData(0, 0, canvas.width, canvas.height);
const pixels = imageData.data;

At this point the pixels variable is a Uint8ClampedArray with a length of width * height * 4. Every array element is one byte and every four elements in the array represents the color of one pixel. Each of the four elements represents amount of red, green, blue, and alpha (transparency) in that order. The pixels are ordered starting from the top-left corner and working left to right and top to bottom.

pixels[0] = red value for pixel 0
pixels[1] = green value for pixel 0
pixels[2] = blue value for pixel 0
pixels[3] = alpha value for pixel 0
pixels[4] = red value for pixel 1
pixels[5] = green value for pixel 1
pixels[6] = blue value for pixel 1
pixels[7] = alpha value for pixel 1
pixels[8] = red value for pixel 2
pixels[9] = green value for pixel 2
pixels[10] = blue value for pixel 2
pixels[11] = alpha value for pixel 2
pixels[12] = red value for pixel 3
...

To find the index for any given pixel from its coordinates, there is a simple formula.

const index = (x + y * imageWidth) * 4;
const red = pixels[index];
const green = pixels[index + 1];
const blue = pixels[index + 2];
const alpha = pixels[index + 3];

You can now read and write this data however you want, allowing you to apply any effects you can think of. However, this array is a copy of the actual pixel data for the canvas. To write the edited version back you need to use the putImageData method to write this back to the top-left corner of the canvas

context.putImageData(imageData, 0, 0);

WebGL #

WebGL is a big topic, certainly too big to do it justice in a single article. If you'd like to learn more about WebGL, check out the recommended reading at the end of this article.

However, here is a very brief introduction to what needs to be done in the case of manipulating a single image.

One of the most important things to remember about WebGL is that it is not a 3D graphics API. In fact, WebGL (and OpenGL) is good at exactly one thing - drawing triangles. In your application you must describe what you actually want to draw in terms of triangles. In the case of a 2D image, that's very simple, because a rectangle is two similar right-angled triangles, arranged so that their hypotenuses are in the same place.

The basic process is:

Send data to the GPU that describes the vertices (points) of the triangles.
Send your source image to the GPU as a texture (image).
Create a 'vertex shader'.
Create a 'fragment shader'.
Set some shader variables, called 'uniforms'.
Run the shaders.

Let's go into detail. Start by allocating some memory on the graphics card called a vertex buffer. You store data in it that describes each point of each triangle. You can also set some variables, called uniforms, that are global values through both shaders.

A vertex shader uses data from the vertex buffer to calculate where on the screen to draw the three points of each triangle.

Now the GPU knows which pixels within the canvas need to be drawn. The fragment shader is called once per pixel, and needs to return the color that should be drawn to the screen. The fragment shader can read information from one or more textures to determine the color.

When reading a texture in a fragment shader you specify which part of the image you want to read using two floating-point coordinates between 0 (left or bottom) and 1 (right or top).

If you want to read the texture based on pixel coordinates then you need to pass the size of the texture in pixels as a uniform vector so that you can do the conversion for each pixel.

varying vec2 pixelCoords;

uniform vec2 textureSize;
uniform sampler2D textureSampler;

main() {
  vec2 textureCoords = pixelCoords / textureSize;
  vec4 textureColor = texture2D(textureSampler, textureCoords);
  gl_FragColor = textureColor;
 }
Pretty much every kind of 2D image manipulation that you might want to do can be done in the
fragment shader, and all of the other WebGL parts can be abstracted away. You can see [the
abstraction layer](https://github.com/GoogleChromeLabs/snapshot/blob/master/src/filters/image-shader.ts) (in
TypeScript) that is being in used in one of our sample applications if you'd like to see an example.

### Which should I use?

For pretty much any professional quality image manipulation, you should use WebGL. There is no
getting away from the fact that this kind of work is the whole reason GPUs were invented. You can
process images an order of magnitude faster on the GPU, which is essential for any real-time
effects.

The way that graphics cards work means that every pixel can be calculated in it's own thread. Even
if you parallelize your code CPU-based code with `Worker`s, your GPU may have 100s of times as many
specialized cores as your CPU has general cores.

2D canvas is much simpler, so is great for prototyping and may be fine for one-off transformations.
However, there are plenty of abstractions around for WebGL that mean you can get the performance
boost without needing to learn the details.

Examples in this article are mostly for 2D canvas to make explanations easier, but the principles
should translate pretty easily to fragment shader code.

## Effect types

### Pixel effects

This is the simplest category to both understand and implement. All of these transformations take
the color value of a single pixel and pass it into a function that returns another color value.

There are many variations on these operations that are more or less complicated. Some will take into
account how the human brain processes visual information based on decades of research, and some will
be dead simple ideas that give an effect that's mostly reasonable.

For example, a brightness control can be implemented by simply taking the red, green and blue values
of the pixel and multiplying them by a brightness value. A brightness of 0 will make the image
entirely black. A value of 1 will leave the image unchanged. A value greater than 1 will make it
brighter.

For 2D canvas:

```js
const brightness = 1.1; // Make the image 10% brighter
for (let i = 0; i < imageData.data.length; i += 4) {
  imageData.data[i] = imageData.data[i] * brightness;
  imageData.data[i + 1] = imageData.data[i + 1] * brightness;
  imageData.data[i + 2] = imageData.data[i + 2] * brightness;
}

Note that the loop moves 4 bytes along at a time, but only changes three values - this is because this particular transformation doesn't change the alpha value. Also remember that a Uint8ClampedArray will round all values to integers, and clamp values to be between 0 and 255.

WebGL fragment shader:

    float brightness = 1.1;
    gl_FragColor = textureColor;
    gl_FragColor.rgb *= brightness;

Similarly, only the RGB part of the output color is multiplied for this particular transformation.

Some of these filters take extra information, such as the average luminance of the whole image, but these are things that can be calculated once for the whole image.

One way of changing contrast, for example, can be to move each pixel towards or away from some 'gray' value, for lower or higher contrast respectively. The gray value is commonly chosen to be a gray color whose luminance is the median luminance of all pixels in the image.

You can calculate this value once when the image is loaded and then use it each time you need to adjust the image effect.

Multi-pixel #

Some effects use the color of neighboring pixels when deciding the color of the current pixel.

This slightly changes how you do things in the 2D canvas case because you want to be able to read the original colors of the image, and the previous example was updating the pixels in place.

This is easy enough, though. When you initially create your image data object you can make a copy of the data.

const originalPixels = new Uint8Array(imageData.data);

For the WebGL case you don't need to make any changes, since the shader does not write to the input texture.

The most common category of multi-pixel effect is called a convolution filter. A convolution filter uses several pixels from the input image to calculate the color of each pixel in the input image. The level of influence that each input pixel has on the output is called a weight.

The weights can be represented by a matrix, called a kernel, with the central value corresponding to the current pixel. For example, this is the kernel for a 3x3 Gaussian blur.

    | 0  1  0 |
    | 1  4  1 |
    | 0  1  0 |

So let's say that you want to calculate the output color of the pixel at (23, 19). Take the 8 pixels surrounding (23, 19) as well as the pixel itself, and multiply the color values for each of them by the corresponding weight.

    (22, 18) x 0    (23, 18) x 1    (24, 18) x 0
    (22, 19) x 1    (23, 19) x 4    (24, 19) x 1
    (22, 20) x 0    (23, 20) x 1    (24, 20) x 0

Sum them all together then divide the result by 8, which is the sum of the weights. You can see how the result will be a pixel that is mostly the original, but with the nearby pixels bleeding in.

const kernel = [
  [0, 1, 0],
  [1, 4, 1],
  [0, 1, 0],
];

for (let y = 0; y < imageHeight; y++) {
  for (let x = 0; x < imageWidth; x++) {
    let redTotal = 0;
    let greenTotal = 0;
    let blueTotal = 0;
    let weightTotal = 0;
    for (let i = -1; i <= 1; i++) {
      for (let j = -1; j <= 1; j++) {
        // Filter out pixels that are off the edge of the image
        if (
          x + i > 0 &&
          x + i < imageWidth &&
          y + j > 0 &&
          y + j < imageHeight
        ) {
          const index = (x + i + (y + j) * imageWidth) * 4;
          const weight = kernel[i + 1][j + 1];
          redTotal += weight * originalPixels[index];
          greenTotal += weight * originalPixels[index + 1];
          blueTotal += weight * originalPixels[index + 2];
          weightTotal += weight;
        }
      }
    }

    const outputIndex = (x + y * imageWidth) * 4;
    imageData.data[outputIndex] = redTotal / weightTotal;
    imageData.data[outputIndex + 1] = greenTotal / weightTotal;
    imageData.data[outputIndex + 2] = blueTotal / weightTotal;
  }
}

This gives the basic idea, but there are guides out there that will go into much more detail, and list many other useful kernels.

Whole image #

Some whole image transformations are simple. In a 2D canvas, cropping and scaling is a simple case of only drawing part of the source image onto the canvas.

// Set the canvas to be a little smaller than the original image
canvas.width = source.naturalWidth - 100;
canvas.height = source.naturalHeight - 100;

// Draw only part of the image onto the canvas
const sx = 50; // The left-most part of the source image to copy
const sy = 50; // The top-most part of the source image to copy
const sw = source.naturalWidth - 100; // How wide the rectangle to copy is
const sh = source.naturalHeight - 100; // How tall the rectangle to copy is

const dx = 0; // The left-most part of the canvas to draw over
const dy = 0; // The top-most part of the canvas to draw over
const dw = canvas.width; // How wide the rectangle to draw over is
const dh = canvas.height; // How tall the rectangle to draw over is

context.drawImage(theOriginalImage, sx, sy, sw, sh, dx, dy, dw, dh);

Rotation and reflection are directly available in the 2D context. Before you draw the image into the canvas, change the various transforms.

// Move the canvas so that the center of the canvas is on the Y-axis
context.translate(-canvas.width / 2, 0);

// An X scale of -1 reflects the canvas in the Y-axis
context.scale(-1, 1);

// Rotate the canvas by 90°
context.rotate(Math.PI / 2);

But more powerfully, many 2D transformations can be written as 2x3 matrices and applied to the canvas with setTransform(). This example uses a matrix that combines a rotation and a translation.

const matrix = [
  Math.cos(rot) * x1,
  -Math.sin(rot) * x2,
  tx,
  Math.sin(rot) * y1,
  Math.cos(rot) * y2,
  ty,
];

context.setTransform(
  matrix[0],
  matrix[1],
  matrix[2],
  matrix[3],
  matrix[4],
  matrix[5],
);

More complicated effects like lens distortion or ripples involve applying some offset to each destination coordinate to calculate the source pixel coordinate. For example, to have a horizontal wave effect you could offset the source pixel x coordinate by some value based on the y coordinate.

for (let y = 0; y < canvas.height; y++) {
  const xOffset = 20 * Math.sin((y * Math.PI) / 20);
  for (let x = 0; x < canvas.width; x++) {
    // Clamp the source x between 0 and width
    const sx = Math.min(Math.max(0, x + xOffset), canvas.width);

    const destIndex = (y * canvas.width + x) * 4;
    const sourceIndex = (y * canvas.width + sx) * 4;

    imageData.data[destIndex] = originalPixels.data[sourceIndex];
    imageData.data[destIndex + 1] = originalPixels.data[sourceIndex + 1];
    imageData.data[destIndex + 2] = originalPixels.data[sourceIndex + 2];
  }
}

Video #

Everything else in the article already works for video if you use a video element as the source image.

Canvas 2D:

context.drawImage(<strong>video</strong>, 0, 0);

WebGL:

gl.texImage2D(
  gl.TEXTURE_2D,
  0,
  gl.RGBA,
  gl.RGBA,
  gl.UNSIGNED_BYTE,
  <strong>video</strong>,
);

However, this will only use the current video frame. So if you wish to apply an effect to a playing video you need to use drawImage/texImage2D on each frame to grab a new video frame and process it on each browser animation frame.

const draw = () => {
  requestAnimationFrame(draw);

  context.drawImage(video, 0, 0);

  // ...image processing goes here
};

When working with video it becomes particularly important that your processing is fast. With a still image a user might not notice a delay of 100ms between clicking a button and an effect being applied. When animated, however, delays of only 16ms can cause visible jerkiness.

Feedback #

Capturing an image from the user

2016-08-23T00:00:00Z

Many browsers now have the ability to access video and audio input from the user. However, depending on the browser it might be a full dynamic and inline experience, or it could be delegated to another app on the user's device. On top of that, not every device even has a camera. So how can you create an experience that uses a user generated image that works well everywhere?

Start simple and progressively #

If you want to progressively enhance your experience, you need to start with something that works everywhere. The easiest thing to do is simply ask the user for a pre-recorded file.

Ask for a URL #

This is the best supported but least satisfying option. Get the user to give you a URL, and then use that. For just displaying the image this works everywhere. Create an img element, set the src and you're done.

Though, if you want to manipulate the image in any way, things are a bit more complicated. CORS prevents you from accessing the actual pixels unless the server sets the appropriate headers and you mark the image as crossorigin ; the only practical way around that is to run a proxy server.

File input #

You can also use a simple file input element, including an accept filter that indicates you only want image files.

<input type="file" accept="image/*" />

This method works on all platforms. On desktop it will prompt the user to upload an image file from the file system. In Chrome and Safari on iOS and Android this method will give the user a choice of which app to use to capture the image, including the option of taking a photo directly with the camera or choosing an existing image file.

The data can then be attached to a <form> or manipulated with JavaScript by listening for an onchange event on the input element and then reading the files property of the event target.

<input type="file" accept="image/*" id="file-input" />
<script>
  const fileInput = document.getElementById('file-input');

  fileInput.addEventListener('change', (e) =>
    doSomethingWithFiles(e.target.files),
  );
</script>

The files property is a FileList object, which I'll talk more about later.

You can also optionally add the capture attribute to the element, which indicates to the browser that you prefer getting an image from the camera.

<input type="file" accept="image/*" capture />
<input type="file" accept="image/*" capture="user" />
<input type="file" accept="image/*" capture="environment" />

Adding the capture attribute without a value let's the browser decide which camera to use, while the "user" and "environment" values tell the browser to prefer the front and rear cameras, respectively.

The capture attribute works on Android and iOS, but is ignored on desktop. Be aware, however, that on Android this means that the user will no longer have the option of choosing an existing picture. The system camera app will be started directly, instead.

Drag and drop #

If you are already adding the ability to upload a file, there are a couple of easy ways that you can make the user experience a little richer.

The first is to add a drop target to your page that allows the user to drag in a file from the desktop or another application.

<div id="target">You can drag an image file here</div>
<script>
  const target = document.getElementById('target');

  target.addEventListener('drop', (e) => {
    e.stopPropagation();
    e.preventDefault();

    doSomethingWithFiles(e.dataTransfer.files);
  });

  target.addEventListener('dragover', (e) => {
    e.stopPropagation();
    e.preventDefault();

    e.dataTransfer.dropEffect = 'copy';
  });
</script>

Similar to the file input, you can get a FileList object from the dataTransfer.files property of the drop event;

The dragover event handler lets you signal to the user what will happen when they drop the file by using the dropEffect property.

Drag and drop has been around for a long time and is well-supported by the major browsers.

Paste from clipboard #

The final way to get an existing image file is from the clipboard. The code for this is very simple, but the user experience is a little harder to get right.

<textarea id="target">Paste an image here</textarea>
<script>
  const target = document.getElementById('target');

  target.addEventListener('paste', (e) => {
    e.preventDefault();
    doSomethingWithFiles(e.clipboardData.files);
  });
</script>

(e.clipboardData.files is yet another FileList object.)

The tricky part with the clipboard API is that, for full cross-browser support, the target element needs to be both selectable and editable. Both <textarea> and <input type="text"> fit the bill here, as do elements with the contenteditable attribute. But these are also obviously designed for editing text.

It can be difficult to make this work smoothly if you don't want the user to be able to input text. Tricks like having a hidden input that gets selected when you click on some other element may make maintaining accessibility harder.

Handling a FileList object #

Since most of the above methods produce a FileList, I should talk a little about what that is.

A FileList is similar to an Array. It has numeric keys and a length property, but it isn't actually an array. There are no array methods, like forEach() or pop(), and it isn't iterable. Of course, you can get a real Array by using Array.from(fileList).

The entries of the FileList are File objects. These are exactly the same as Blob objects except that they have additional name and lastModified read-only properties.

<img id="output" />
<script>
  const output = document.getElementById('output');

  function doSomethingWithFiles(fileList) {
    let file = null;

    for (let i = 0; i < fileList.length; i++) {
      if (fileList[i].type.match(/^image\//)) {
        file = fileList[i];
        break;
      }
    }

    if (file !== null) {
      output.src = URL.createObjectURL(file);
    }
  }
</script>

This example finds the first file that has an image MIME type, but it could also handle multiple images being selected/pasted/dropped at once.

Once you have access to the file you can do anything you want with it. For example, you can:

Draw it into a <canvas> element so that you can manipulate it
Download it to the user's device
Upload it to a server with fetch()

Access the camera interactively #

Now that you've covered your bases, it's time to progressively enhance!

Modern browsers can get direct access to cameras, allowing you to build experiences that are fully integrated with the web page, so the user need never leave the browser.

Acquire access to the camera #

You can directly access a camera and microphone by using an API in the WebRTC specification called getUserMedia(). This will prompt the user for access to their connected microphones and cameras.

Support for getUserMedia() is pretty good, but it isn't yet everywhere. In particular, it is not available in Safari 10 or lower, which at the time of writing is still the latest stable version. However, Apple have announced that it will be available in Safari 11.

It's very simple to detect support, however.

const supported = 'mediaDevices' in navigator;

When you call getUserMedia(), you need to pass in an object that describes what kind of media you want. These choices are called constraints. There are a several possible constraints, covering things like whether you prefer a front- or rear-facing camera, whether you want audio, and your preferred resolution for the stream.

To get data from the camera, however, you need just one constraint, and that is video: true.

If successful the API will return a MediaStream that contains data from the camera, and you can then either attach it to a <video> element and play it to show a real time preview, or attach it to a <canvas> to get a snapshot.

<video id="player" controls autoplay></video>
<script>
  const player = document.getElementById('player');

  const constraints = {
    video: true,
  };

  navigator.mediaDevices.getUserMedia(constraints).then((stream) => {
    player.srcObject = stream;
  });
</script>

By itself, this isn't that useful. All you can do is take the video data and play it back. If you want to get an image, you have to do a little extra work.

Grab a snapshot #

Your best supported option for getting an image is to draw a frame from the video to a canvas.

Unlike the Web Audio API, there isn't a dedicated stream processing API for video on the web so you have to resort to a tiny bit of hackery to capture a snapshot from the user's camera.

The process is as follows:

Create a canvas object that will hold the frame from the camera
Get access to the camera stream
Attach it to a video element
When you want to capture a precise frame, add the data from the video element to a canvas object using drawImage().

<video id="player" controls autoplay></video>
<button id="capture">Capture</button>
<canvas id="canvas" width="320" height="240"></canvas>
<script>
  const player = document.getElementById('player');
  const canvas = document.getElementById('canvas');
  const context = canvas.getContext('2d');
  const captureButton = document.getElementById('capture');

  const constraints = {
    video: true,
  };

  captureButton.addEventListener('click', () => {
    // Draw the video frame to the canvas.
    context.drawImage(player, 0, 0, canvas.width, canvas.height);
  });

  // Attach the video stream to the video element and autoplay.
  navigator.mediaDevices.getUserMedia(constraints).then((stream) => {
    player.srcObject = stream;
  });
</script>

Once you have data from the camera stored in the canvas you can do many things with it. You could:

Upload it straight to the server
Store it locally
Apply funky effects to the image

Tips #

Stop streaming from the camera when not needed #

It is good practice to stop using the camera when you no longer need it. Not only will this save battery and processing power, it will also give users confidence in your application.

To stop access to the camera you can simply call stop() on each video track for the stream returned by getUserMedia().

<video id="player" controls autoplay></video>
<button id="capture">Capture</button>
<canvas id="canvas" width="320" height="240"></canvas>
<script>
  const player = document.getElementById('player');
  const canvas = document.getElementById('canvas');
  const context = canvas.getContext('2d');
  const captureButton = document.getElementById('capture');

  const constraints = {
    video: true,
  };

  captureButton.addEventListener('click', () => {
    context.drawImage(player, 0, 0, canvas.width, canvas.height);

    // Stop all video streams.
    player.srcObject.getVideoTracks().forEach(track => track.stop());
  });

  navigator.mediaDevices.getUserMedia(constraints).then((stream) => {
    // Attach the video stream to the video element and autoplay.
    player.srcObject = stream;
  });
</script>

Ask permission to use camera responsibly #

If the user has not previously granted your site access to the camera then the instant that you call getUserMedia() the browser will prompt the user to grant your site permission to the camera.

Users hate getting prompted for access to powerful devices on their machine and they will frequently block the request, or they will ignore it if they don't understand the context for which the prompt has been created. It is best practice to only ask to access the camera when first needed. Once the user has granted access they won't be asked again. However, if the user rejects access, you can't get access again, unless they manually change camera permission settings.

Compatibility #

More information about mobile and desktop browser implementation:

We also recommend using the adapter.js shim to protect apps from WebRTC spec changes and prefix differences.

Mat Scales on web.dev

Recording Video from the User

Start simple and progressively #

Access the camera interactively #

Acquire access to the camera #

Access the raw data from the camera #

Save the data from the camera #

Ask permission to use camera responsibly #

Use the permissions API to check if you already have access #

Feedback #

Real-Time Effects For Images and Video

How to get at the pixel data for an image #

2D canvas #

WebGL #

Multi-pixel #

Whole image #

Video #

Recommended reading #

Feedback #

Capturing an image from the user

Start simple and progressively #

Ask for a URL #

File input #

Drag and drop #

Paste from clipboard #

Handling a FileList object #

Access the camera interactively #

Acquire access to the camera #

Grab a snapshot #

Tips #

Stop streaming from the camera when not needed #

Ask permission to use camera responsibly #

Compatibility #

Feedback #