Ingvar Stepanyan on web.dev

Drawing to canvas in Emscripten

2022-02-07T00:00:00Z

Different operating systems have different APIs for drawing graphics. The differences become even more confusing when writing a cross-platform code, or porting graphics from one system to another, including when porting native code to WebAssembly.

In this post you will learn a couple of methods for drawing 2D graphics to the canvas element on the web from C or C++ code compiled with Emscripten.

Canvas via Embind #

If you're starting a new project rather than trying to port an existing one, it might be easiest to use the HTML Canvas API via Emscripten's binding system Embind. Embind allows you to operate directly on arbitrary JavaScript values.

To understand how to use Embind, first take a look at the following example from MDN that finds a <canvas> element, and draws some shapes on it

const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');

ctx.fillStyle = 'green';
ctx.fillRect(10, 10, 150, 100);

Here's how it can be transliterated to C++ with Embind:

#include <emscripten/val.h>

using emscripten::val;

// Use thread_local when you want to retrieve & cache a global JS variable once per thread.
thread_local const val document = val::global("document");

// …

int main() {
  val canvas = document.call<val>("getElementById", "canvas");
  val ctx = canvas.call<val>("getContext", "2d");
  ctx.set("fillStyle", "green");
  ctx.call<void>("fillRect", 10, 10, 150, 100);
}

When linking this code, make sure to pass --bind to enable Embind:

emcc --bind example.cpp -o example.html

Then you can serve the compiled assets with a static server and load the example in a browser:

Choosing the canvas element #

When using the Emscripten-generated HTML shell with the preceding shell command, the canvas is included and set up for you. It makes it easier to build simple demos and examples, but in larger applications you'd want to include the Emscripten-generated JavaScript and WebAssembly in an HTML page of your own design.

The generated JavaScript code expects to find the canvas element stored in the Module.canvas property. Like other Module properties, it can be set during initialization.

If you're using ES6 mode (setting output to a path with an extension .mjs or using the -s EXPORT_ES6 setting), you can pass the canvas like this:

import initModule from './emscripten-generated.mjs';

const Module = await initModule({
  canvas: document.getElementById('my-canvas')
});

If you're using regular script output, you need to declare the Module object before loading the Emscripten-generated JavaScript file:

<script>
var Module = {
  canvas: document.getElementById('my-canvas')
};
</script>
<script src="emscripten-generated.js"></script>

OpenGL and SDL2 #

OpenGL is a popular cross-platform API for computer graphics. When used in Emscripten, it will take care of converting the supported subset of OpenGL operations to WebGL. If your application relies on features supported in OpenGL ES 2.0 or 3.0, but not in WebGL, Emscripten can take care of emulating those too, but you need to opt-in via the corresponding settings.

You can use OpenGL either directly or via higher-level 2D and 3D graphics libraries. A couple of those have been ported to the web with Emscripten. In this post, I'm focusing on 2D graphics, and for that SDL2 is currently the preferred library because it's been well-tested and supports the Emscripten backend officially upstream.

Drawing a rectangle #

"About SDL" section on the official website says:

Simple DirectMedia Layer is a cross-platform development library designed to provide low level access to audio, keyboard, mouse, joystick, and graphics hardware via OpenGL and Direct3D.

All those features - controlling audio, keyboard, mouse and graphics - have been ported and work with Emscripten on the web too so you can port entire games built with SDL2 without much hassle. If you're porting an existing project, check out the "Integrating with a build system" section of Emscripten docs.

For simplicity, in this post I'll focus on a single-file case and translate the earlier rectangle example to SDL2:

#include <SDL2/SDL.h>

int main() {
  // Initialize SDL graphics subsystem.
  SDL_Init(SDL_INIT_VIDEO);

  // Initialize a 300x300 window and a renderer.
  SDL_Window *window;
  SDL_Renderer *renderer;
  SDL_CreateWindowAndRenderer(300, 300, 0, &window, &renderer);

  // Set a color for drawing matching the earlier `ctx.fillStyle = "green"`.
  SDL_SetRenderDrawColor(renderer, /* RGBA: green */ 0x00, 0x80, 0x00, 0xFF);
  // Create and draw a rectangle like in the earlier `ctx.fillRect()`.
  SDL_Rect rect = {.x = 10, .y = 10, .w = 150, .h = 100};
  SDL_RenderFillRect(renderer, &rect);

  // Render everything from a buffer to the actual screen.
  SDL_RenderPresent(renderer);

  // TODO: cleanup
}

When linking with Emscripten, you need to use -s USE_SDL=2. This will tell Emscripten to fetch the SDL2 library, already precompiled to WebAssembly, and link it with your main application.

emcc example.cpp -o example.html -s USE_SDL=2

When the example is loaded in the browser, you'll see the familiar green rectangle:

This code has a couple of problems though. First, it lacks proper cleanup of allocated resources. Second, on the web, pages don't get closed automatically when an application has finished its execution, so the image on the canvas gets preserved. However, when the same code is recompiled natively with

clang example.cpp -o example -lSDL2

and executed, the created window will only blink briefly and immediately close upon exit, so the user doesn't have a chance to see the image.

Integrating an event loop #

A more complete and idiomatic example would look need to wait in an event loop until the user chooses to quit the application:

#include <SDL2/SDL.h>

int main() {
  SDL_Init(SDL_INIT_VIDEO);

  SDL_Window *window;
  SDL_Renderer *renderer;
  SDL_CreateWindowAndRenderer(300, 300, 0, &window, &renderer);

  SDL_SetRenderDrawColor(renderer, /* RGBA: green */ 0x00, 0x80, 0x00, 0xFF);
  SDL_Rect rect = {.x = 10, .y = 10, .w = 150, .h = 100};
  SDL_RenderFillRect(renderer, &rect);
  SDL_RenderPresent(renderer);

  while (1) {
    SDL_Event event;
    SDL_PollEvent(&event);
    if (event.type == SDL_QUIT) {
      break;
    }
  }

  SDL_DestroyRenderer(renderer);
  SDL_DestroyWindow(window);

  SDL_Quit();
}

After the image has been drawn to a window, the application now waits in a loop, where it can process keyboard, mouse and other user events. When the user closes the window, they'll trigger an SDL_QUIT event, which will be intercepted to exit the loop. After the loop is exited, the application will do the cleanup and then exit itself.

Now compiling this example on Linux works as expected and shows a 300 by 300 window with a green rectangle:

However, the example no longer works on the web. The Emscripten-generated page freezes immediately during the load and never shows the rendered image:

What happened? I'll quote the answer from the article "Using asynchronous web APIs from WebAssembly":

The short version is that the browser runs all the pieces of code in sort of an infinite loop, by taking them from the queue one by one. When some event is triggered, the browser queues the corresponding handler, and on the next loop iteration it's taken out from the queue and executed. This mechanism allows simulating concurrency and running lots of parallel operations while using only a single thread.

The important thing to remember about this mechanism is that, while your custom JavaScript (or WebAssembly) code executes, the event loop is blocked […]

The preceding example executes an infinite event loop, while the code itself runs inside another infinite event loop, implicitly provided by the browser. The inner loop never relinquishes control to the outer one, so the browser doesn't get a chance to process external events or draw things onto the page.

There are two ways to fix this problem.

Unblocking event loop with Asyncify #

First, as described in the linked article, you can use Asyncify. It's an Emscripten feature that allows to "pause" the C or C++ program, give control back to the event loop, and wake up the program when some asynchronous operation has finished.

Such asynchronous operation can be even "sleep for the minimum possible time", expressed via emscripten_sleep(0) API. By embedding it in the middle of the loop, I can ensure that the control is returned to browser's event loop on each iteration, and the page remains responsive and can handle any events:

#include <SDL2/SDL.h>
#ifdef __EMSCRIPTEN__
#include <emscripten.h>
#endif

int main() {
  SDL_Init(SDL_INIT_VIDEO);

  SDL_Window *window;
  SDL_Renderer *renderer;
  SDL_CreateWindowAndRenderer(300, 300, 0, &window, &renderer);

  SDL_SetRenderDrawColor(renderer, /* RGBA: green */ 0x00, 0x80, 0x00, 0xFF);
  SDL_Rect rect = {.x = 10, .y = 10, .w = 150, .h = 100};
  SDL_RenderFillRect(renderer, &rect);
  SDL_RenderPresent(renderer);

  while (1) {
    SDL_Event event;
    SDL_PollEvent(&event);
    if (event.type == SDL_QUIT) {
      break;
    }
#ifdef __EMSCRIPTEN__
    emscripten_sleep(0);
#endif
  }

  SDL_DestroyRenderer(renderer);
  SDL_DestroyWindow(window);

  SDL_Quit();
}

This code now needs to be compiled with Asyncify enabled:

emcc example.cpp -o example.html -s USE_SDL=2 -s ASYNCIFY

And the application works as expected on the web again:

However, Asyncify can have non-trivial code size overhead. If it's only used for a top-level event loop in the application, a better option can be to use the emscripten_set_main_loop function.

Unblocking event loop with "main loop" APIs #

emscripten_set_main_loop doesn't require any compiler transformations for unwinding and rewinding the call stack, and that way avoids the code size overhead. However, in exchange, it requires a lot more manual modifications to the code.

First, the body of the event loop needs to be extracted into a separate function. Then, emscripten_set_main_loop needs to be called with that function as a callback in the first argument, an FPS in the second argument (0 for the native refresh interval), and a boolean indicating whether to simulate infinite loop (true) in the third:

emscripten_set_main_loop(callback, 0, true);

The newly created callback won't have any access to the stack variables in the main function, so variables like window and renderer need to be either extracted into a heap-allocated struct and its pointer passed via emscripten_set_main_loop_arg variant of the API, or extracted into global static variables (I went with the latter for simplicity). The result is slightly harder to follow, but it draws the same rectangle as the last example:

#include <SDL2/SDL.h>
#include <stdio.h>
#ifdef __EMSCRIPTEN__
#include <emscripten.h>
#endif

SDL_Window *window;
SDL_Renderer *renderer;

bool handle_events() {
  SDL_Event event;
  SDL_PollEvent(&event);
  if (event.type == SDL_QUIT) {
    return false;
  }
  return true;
}

void run_main_loop() {
#ifdef __EMSCRIPTEN__
  emscripten_set_main_loop([]() { handle_events(); }, 0, true);
#else
  while (handle_events())
    ;
#endif
}

int main() {
  SDL_Init(SDL_INIT_VIDEO);

  SDL_CreateWindowAndRenderer(300, 300, 0, &window, &renderer);

  SDL_SetRenderDrawColor(renderer, /* RGBA: green */ 0x00, 0x80, 0x00, 0xFF);
  SDL_Rect rect = {.x = 10, .y = 10, .w = 150, .h = 100};
  SDL_RenderFillRect(renderer, &rect);
  SDL_RenderPresent(renderer);

  run_main_loop();

  SDL_DestroyRenderer(renderer);
  SDL_DestroyWindow(window);

  SDL_Quit();
}

Since all the control flow changes are manual and reflected in the source code, it can be compiled without the Asyncify feature again:

emcc example.cpp -o example.html -s USE_SDL=2

This example might seem useless, because it works no differently from the first version, where the rectangle was drawn on canvas successfully despite the code being a lot simpler, and the SDL_QUIT event—the only one handled in the handle_events function—is ignored on the web anyway.

However, proper event loop integration - either via Asyncify or via emscripten_set_main_loop - pays off if you decide to add any kind of animation or interactivity.

Handling user interactions #

For example, with a few changes to the last example you can make the rectangle move in response to keyboard events:

#include <SDL2/SDL.h>
#ifdef __EMSCRIPTEN__
#include <emscripten.h>
#endif

SDL_Window *window;
SDL_Renderer *renderer;

SDL_Rect rect = {.x = 10, .y = 10, .w = 150, .h = 100};

void redraw() {
  SDL_SetRenderDrawColor(renderer, /* RGBA: black */ 0x00, 0x00, 0x00, 0xFF);
  SDL_RenderClear(renderer);
  SDL_SetRenderDrawColor(renderer, /* RGBA: green */ 0x00, 0x80, 0x00, 0xFF);
  SDL_RenderFillRect(renderer, &rect);
  SDL_RenderPresent(renderer);
}

uint32_t ticksForNextKeyDown = 0;

bool handle_events() {
  SDL_Event event;
  SDL_PollEvent(&event);
  if (event.type == SDL_QUIT) {
    return false;
  }
  if (event.type == SDL_KEYDOWN) {
    uint32_t ticksNow = SDL_GetTicks();
    if (SDL_TICKS_PASSED(ticksNow, ticksForNextKeyDown)) {
      // Throttle keydown events for 10ms.
      ticksForNextKeyDown = ticksNow + 10;
      switch (event.key.keysym.sym) {
        case SDLK_UP:
          rect.y -= 1;
          break;
        case SDLK_DOWN:
          rect.y += 1;
          break;
        case SDLK_RIGHT:
          rect.x += 1;
          break;
        case SDLK_LEFT:
          rect.x -= 1;
          break;
      }
      redraw();
    }
  }
  return true;
}

void run_main_loop() {
#ifdef __EMSCRIPTEN__
  emscripten_set_main_loop([]() { handle_events(); }, 0, true);
#else
  while (handle_events())
    ;
#endif
}

int main() {
  SDL_Init(SDL_INIT_VIDEO);

  SDL_CreateWindowAndRenderer(300, 300, 0, &window, &renderer);

  redraw();
  run_main_loop();

  SDL_DestroyRenderer(renderer);
  SDL_DestroyWindow(window);

  SDL_Quit();
}

Drawing other shapes with SDL2_gfx #

SDL2 abstracts away cross-platform differences and various types of media devices in a single API, but it's still a pretty low-level library. In particular for graphics, while it provides APIs for drawing points, lines and rectangles, implementation of any more complex shapes and transformations is left to the user.

SDL2_gfx is a separate library that fills that gap. For example, it can be used to replace a rectangle in the example above with a circle:

#include <SDL2/SDL.h>
#include <SDL2/SDL2_gfxPrimitives.h>
#ifdef __EMSCRIPTEN__
#include <emscripten.h>
#endif

SDL_Window *window;
SDL_Renderer *renderer;

SDL_Point center = {.x = 100, .y = 100};
const int radius = 100;

void redraw() {
  SDL_SetRenderDrawColor(renderer, /* RGBA: black */ 0x00, 0x00, 0x00, 0xFF);
  SDL_RenderClear(renderer);
  filledCircleRGBA(renderer, center.x, center.y, radius,
                   /* RGBA: green */ 0x00, 0x80, 0x00, 0xFF);
  SDL_RenderPresent(renderer);
}

uint32_t ticksForNextKeyDown = 0;

bool handle_events() {
  SDL_Event event;
  SDL_PollEvent(&event);
  if (event.type == SDL_QUIT) {
    return false;
  }
  if (event.type == SDL_KEYDOWN) {
    uint32_t ticksNow = SDL_GetTicks();
    if (SDL_TICKS_PASSED(ticksNow, ticksForNextKeyDown)) {
      // Throttle keydown events for 10ms.
      ticksForNextKeyDown = ticksNow + 10;
      switch (event.key.keysym.sym) {
        case SDLK_UP:
          center.y -= 1;
          break;
        case SDLK_DOWN:
          center.y += 1;
          break;
        case SDLK_RIGHT:
          center.x += 1;
          break;
        case SDLK_LEFT:
          center.x -= 1;
          break;
      }
      redraw();
    }
  }
  return true;
}

void run_main_loop() {
#ifdef __EMSCRIPTEN__
  emscripten_set_main_loop([]() { handle_events(); }, 0, true);
#else
  while (handle_events())
    ;
#endif
}

int main() {
  SDL_Init(SDL_INIT_VIDEO);

  SDL_CreateWindowAndRenderer(300, 300, 0, &window, &renderer);

  redraw();
  run_main_loop();

  SDL_DestroyRenderer(renderer);
  SDL_DestroyWindow(window);

  SDL_Quit();
}

Now the SDL2_gfx library also needs to be linked into the application. It's done similarly to SDL2:

# Native version
$ clang example.cpp -o example -lSDL2 -lSDL2_gfx
# Web version
$ emcc --bind foo.cpp -o foo.html -s USE_SDL=2 -s USE_SDL_GFX=2

And here are the results running on Linux:

And on the web:

For more graphics primitives, check out the auto-generated docs.

Porting USB applications to the web. Part 2: gPhoto2

2022-02-01T00:00:00Z

In the previous post I showed how the libusb library was ported to run on the web with WebAssembly / Emscripten, Asyncify, and WebUSB.

I also showed a demo built with gPhoto2 that can control DSLR and mirrorless cameras over USB from a web application. In this post I'll go deeper into the technical details behind the gPhoto2 port.

Pointing build systems to custom forks #

Since I was targeting WebAssembly, I couldn't use the libusb and libgphoto2 provided by the system distributions. Instead, I needed my application to use my custom fork of libgphoto2, while that fork of libgphoto2 had to use my custom fork of libusb.

Additionally, libgphoto2 uses libtool for loading dynamic plugins, and even though I didn't have to fork libtool like the other two libraries, I still had to build it to WebAssembly, and point libgphoto2 to that custom build instead of the system package.

Here's an approximate dependency diagram (dashed lines denote dynamic linking):

Most configure-based build systems, including the ones used in these libraries, allow overriding paths for dependencies via various flags, so that's what I tried to do first. However, when the dependency graph becomes complex, the list of path overrides for each library's dependencies becomes verbose and error-prone. I also found some bugs where build systems weren't actually prepared for their dependencies to live in non-standard paths.

Instead, an easier approach is to create a separate folder as a custom system root (often shortened to "sysroot") and point all the involved build systems to it. That way, each library will both search for its dependencies in the specified sysroot during build, and it will also install itself in the same sysroot so that others can find it more easily.

Emscripten already has its own sysroot under (path to emscripten cache)/sysroot, which it uses for its system libraries, Emscripten ports, and tools like CMake and pkg-config. I chose to reuse the same sysroot for my dependencies too.

# This is the default path, but you can override it
# to store the cache elsewhere if you want.
#
# For example, it might be useful for Docker builds
# if you want to preserve the deps between reruns.
EM_CACHE = $(EMSCRIPTEN)/cache

# Sysroot is always under the `sysroot` subfolder.
SYSROOT = $(EM_CACHE)/sysroot

# …

# For all dependencies I've used the same ./configure command with the
# earlier defined SYSROOT path as the --prefix.
deps/%/Makefile: deps/%/configure
        cd $(@D) && ./configure --prefix=$(SYSROOT) # …

With such configuration, I only needed to run make install in each dependency, which installed it under the sysroot, and then the libraries found each other automatically.

Dealing with dynamic loading #

As mentioned above, libgphoto2 uses libtool to enumerate and dynamically load I/O port adapters and camera libraries. For example, the code for loading I/O libraries looks like this:

lt_dlinit ();
lt_dladdsearchdir (iolibs);
result = lt_dlforeachfile (iolibs, foreach_func, list);
lt_dlexit ();

There are a few problems with this approach on the web:

There is no standard support for dynamic linking of WebAssembly modules. Emscripten has its custom implementation that can simulate the dlopen() API used by libtool, but it requires you to build "main'' and "side" modules with different flags, and, specifically for dlopen(), also to preload the side modules into the emulated filesystem during the application start-up. It can be difficult to integrate those flags and tweaks into an existing autoconf build system with lots of dynamic libraries.
Even if the dlopen() itself is implemented, there's no way to enumerate all dynamic libraries in a certain folder on the web, because most HTTP servers don't expose directory listings for security reasons.
Linking dynamic libraries on the command line instead of enumerating in runtime can also lead to problems, such as the duplicate symbols issue, that are caused by differences between representation of shared libraries in Emscripten and on other platforms.

It's possible to adapt the build system to those differences and hardcode the list of dynamic plugins somewhere during the build, but an even easier way to solve all those issues is to avoid dynamic linking to begin with.

Turns out, libtool abstracts away various dynamic linking methods on different platforms, and even supports writing custom loaders for others. One of the built-in loaders it supports is called "Dlpreopening":

“Libtool provides special support for dlopening libtool object and libtool library files, so that their symbols can be resolved even on platforms without any dlopen and dlsym functions.
…
Libtool emulates -dlopen on static platforms by linking objects into the program at compile time, and creating data structures that represent the program’s symbol table. In order to use this feature, you must declare the objects you want your application to dlopen by using the -dlopen or -dlpreopen flags when you link your program (see Link mode).”

This mechanism allows emulating dynamic loading at libtool level instead of Emscripten, while linking everything statically into a single library.

The only problem this doesn't solve is enumeration of dynamic libraries. The list of those still needs to be hardcoded somewhere. Luckily, the set of plugins I needed for the app is minimal:

On the ports side, I only care about the libusb-based camera connection and not about PTP/IP, serial access, or USB drive modes.
On the camlibs side, there are various vendor-specific plugins which might provide some specialized functions, but for general settings control and capture it's enough to use the Picture Transfer Protocol, which is represented by the ptp2 camlib and supported by virtually every camera on the market.

Here's what the updated dependency diagram looks like with everything linked statically together:

So that's what I hardcoded for Emscripten builds:

LTDL_SET_PRELOADED_SYMBOLS();
lt_dlinit ();
#ifdef __EMSCRIPTEN__
  result = foreach_func("libusb1", list);
#else
  lt_dladdsearchdir (iolibs);
  result = lt_dlforeachfile (iolibs, foreach_func, list);
#endif
lt_dlexit ();

and

LTDL_SET_PRELOADED_SYMBOLS();
lt_dlinit ();
#ifdef __EMSCRIPTEN__
  ret = foreach_func("libptp2", &foreach_data);
#else
  lt_dladdsearchdir (dir);
  ret = lt_dlforeachfile (dir, foreach_func, &foreach_data);
#endif
lt_dlexit ();

In the autoconf build system, I now had to add -dlpreopen with both of those files as link flags for all executables (examples, tests and my own demo app), like this:

if HAVE_EMSCRIPTEN
LDADD += -dlpreopen $(top_builddir)/libgphoto2_port/usb1.la \
         -dlpreopen $(top_builddir)/camlibs/ptp2.la
endif

Finally, now that all the symbols are linked statically in a single library, libtool needs a way to determine which symbol belongs to which library. To achieve this, it requires developers to rename all exposed symbols like {function name} to {library name}_LTX_{function name}. The easiest way to do this is by using #define to redefine symbol names at the top of the implementation file:

// …
#include "config.h"

/* Define _LTX_ names - required to prevent clashes when using libtool preloading. */
#define gp_port_library_type libusb1_LTX_gp_port_library_type
#define gp_port_library_list libusb1_LTX_gp_port_library_list
#define gp_port_library_operations libusb1_LTX_gp_port_library_operations

#include <gphoto2/gphoto2-port-library.h>
// …

This naming scheme also prevents name clashes in case I decide to link camera-specific plugins in the same app in the future.

After all these changes were implemented, I could build the test application and load the plugins successfully.

Generating the settings UI #

gPhoto2 allows camera libraries to define their own settings in a form of widget tree. The hierarchy of widget types consists of:

Window - top-level configuration container
- Sections - named groups of other widgets
  - Button fields
  - Text fields
  - Numeric fields
  - Date fields
  - Toggles
  - Radio buttons

The name, type, children, and all the other relevant properties of each widget can be queried (and, in case of values, also modified) via the exposed C API. Together, they provide a foundation for automatically generating settings UI in any language that can interact with C.

Settings can be changed either via gPhoto2, or on the camera itself at any point in time. Additionally, some widgets can be readonly, and even the readonly state itself depends on the camera mode and other settings. For example, shutter speed is a writable numeric field in M (manual mode), but becomes an informational readonly field in P (program mode). In P mode, the value of shutter speed will also be dynamic and continuously changing depending on the brightness of the scene the camera is looking at.

All in all, it's important to always show up-to-date information from the connected camera in the UI, while at the same time allowing the user to edit those settings from the same UI. Such bidirectional data flow is more complex to handle.

gPhoto2 does not have a mechanism to retrieve only changed settings, only the entire tree or individual widgets. In order to keep the UI up-to-date without flickering and losing input focus or scroll position, I needed a way to diff the widget trees between the invocations and update only the changed UI properties. Luckily, this is a solved problem on the web, and is the core functionality of frameworks like React or Preact. I went with Preact for this project, as it's much more lightweight and does everything I need.

On the C++ side I now needed to retrieve and recursively walk the settings tree via the earlier linked C API, and convert each widget to a JavaScript object:

static std::pair<val, val> walk_config(CameraWidget *widget) {
  val result = val::object();

  val name(GPP_CALL(const char *, gp_widget_get_name(widget, _)));
  result.set("name", name);
  result.set("info", /* … */);
  result.set("label", /* … */);
  result.set("readonly", /* … */);

  auto type = GPP_CALL(CameraWidgetType, gp_widget_get_type(widget, _));

  switch (type) {
    case GP_WIDGET_RANGE: {
      result.set("type", "range");
      result.set("value", GPP_CALL(float, gp_widget_get_value(widget, _)));

      float min, max, step;
      gpp_try(gp_widget_get_range(widget, &min, &max, &step));
      result.set("min", min);
      result.set("max", max);
      result.set("step", step);

      break;
    }
    case GP_WIDGET_TEXT: {
      result.set("type", "text");
      result.set("value",
                  GPP_CALL(const char *, gp_widget_get_value(widget, _)));

      break;
    }
    // …

On the JavaScript side, I could now call configToJS, walk over the returned JavaScript representation of the settings tree, and build the UI via Preact function h:

let inputElem;
switch (config.type) {
  case 'range': {
    let { min, max, step } = config;
    inputElem = h(EditableInput, {
      type: 'number',
      min,
      max,
      step,
      …attrs
    });
    break;
  }
  case 'text':
    inputElem = h(EditableInput, attrs);
    break;
  case 'toggle': {
    inputElem = h('input', {
      type: 'checkbox',
      …attrs
    });
    break;
  }
  // …

By running this function repeatedly in an infinite event loop, I could get the settings UI to always show the latest information, while also sending commands to the camera whenever one of the fields is edited by the user.

Preact can take care of diffing the results and updating the DOM only for the changed bits of the UI, without disrupting the page focus or edit states. One problem that remains is the bidirectional data flow. Frameworks like React and Preact were designed around unidirectional data flow, because it makes it a lot easier to reason about the data and compare it between reruns, but I'm breaking that expectation by allowing an external source - the camera - to update the settings UI at any time.

I worked around this problem by opting out from UI updates for any input fields that are currently being edited by the user:

/**
 * Wrapper around <input /> that doesn't update it while it's in focus to allow editing.
 */
class EditableInput extends Component {
  ref = createRef();

  shouldComponentUpdate() {
    return this.props.readonly || document.activeElement !== this.ref.current;
  }

  render(props) {
    return h('input', Object.assign(props, {ref: this.ref}));
  }
}

This way, there is always only one owner of any given field. Either the user is currently editing it, and won't be disrupted by the updated values from the camera, or the camera is updating the field value while it's out of focus.

Building a live "video" feed #

During the pandemic, a lot of people moved to online meetings. Among other things, this led to shortages on the webcam market. To get a better video quality compared to built-in cameras in laptops, and in response to said shortages, many DSLR and mirrorless camera owners started looking for ways to use their photography cameras as webcams. Several camera vendors even shipped official utilities for this very purpose.

Like the official tools, gPhoto2 supports streaming video from the camera to a locally stored file or directly to a virtual webcam too. I wanted to use that feature to provide a live view in my demo. However, while it's available in the console utility, I couldn't find it anywhere in the libgphoto2 library APIs.

Looking at the source code of the corresponding function in the console utility, I found that it's not actually getting a video at all, but instead keeps retrieving the camera's preview as individual JPEG images in an endless loop, and writing them out one by one to form an M-JPEG stream:

while (1) {
  const char *mime;
  r = gp_camera_capture_preview (p->camera, file, p->context);
  // …

I was astonished that this approach works efficiently enough to get an impression of smooth realtime video. I was even more skeptical about being able to match the same performance in the web application too, with all the extra abstractions and Asyncify in the way. However, I decided to try anyway.

On the C++ side I exposed a method called capturePreviewAsBlob() that invokes the same gp_camera_capture_preview() function, and converts the resulting in-memory file to a Blob that can be passed to other web APIs more easily:

val capturePreviewAsBlob() {
  return gpp_rethrow([=]() {
    auto &file = get_file();

    gpp_try(gp_camera_capture_preview(camera.get(), &file, context.get()));

    auto params = blob_chunks_and_opts(file);
    return Blob.new_(std::move(params.first), std::move(params.second));
  });
}

On the JavaScript side, I have a loop, similar to the one in gPhoto2, that keeps retrieving preview images as Blobs, decodes them in the background with createImageBitmap, and transfers them to the canvas on the next animation frame:

while (this.canvasRef.current) {
  try {
    let blob = await this.props.getPreview();

    let img = await createImageBitmap(blob, { /* … */ });
    await new Promise(resolve => requestAnimationFrame(resolve));
    canvasCtx.transferFromImageBitmap(img);
  } catch (err) {
    // …
  }
}

Using those modern APIs ensures that all of the decoding work is done in the background, and the canvas gets updated only when both the image and the browser are fully prepared for drawing. This achieved a consistent 30+ FPS on my laptop, which matched the native performance of both gPhoto2 and the official Sony software.

Synchronizing the USB access #

When a USB data transfer is requested while another operation is already in progress, it will commonly result in a "device is busy" error. Since the preview and the settings UI update regularly, and the user might be trying to capture an image or modify settings at the same time, such conflicts between different operations turned out to be very frequent.

To avoid them, I needed to synchronize all the accesses within the application. For that, I've built a promise-based async queue:

let context = await new Module.Context();

let queue = Promise.resolve();

function schedule(op) {
  let res = queue.then(() => op(context));
  queue = res.catch(rethrowIfCritical);
  return res;
}

By chaining each operation in a then() callback of the existing queue promise, and storing the chained result as the new value of queue, I can make sure that all operations are executed one by one, in order and without overlaps.

Any operation errors are returned to the caller, while critical (unexpected) errors mark the entire chain as a rejected promise, and ensure that no new operation will be scheduled afterwards.

By keeping the module context in a private (non-exported) variable, I'm minimizing the risks of accessing the context by accident somewhere else in the app without going through the schedule() call.

To tie things together, now each access to the device context has to be wrapped in a schedule() call like this:

let config = await this.connection.schedule((context) => context.configToJS());

and

this.connection.schedule((context) => context.captureImageAsFile());

After that, all the operations were executing successfully without conflicts.

Conclusion #

Feel free to browse the codebase on Github for more implementation insights. I also want to thank Marcus Meissner for maintenance of gPhoto2 and for his reviews of my upstream PRs.

As shows in these posts, WebAssembly, Asyncify and Fugu APIs provide a capable compilation target for even the most complex applications. They allow you to take a library or an application previously built for a single platform, and port it to the web, making it available to a vastly larger number of users across desktop and mobile devices alike.

WebAssembly feature detection

2022-01-27T00:00:00Z

WebAssembly 1.0 was released four years ago, but development didn't stop there. New features are added through the proposal standardization process. As is generally the case with new features on the web, their implementation order and timelines can differ significantly between different engines. If you want to use those new features, you need to ensure that none of your users are left out. In this article you’ll learn an approach for achieving this.

Some new features improve code size by adding new instructions for common operations, some add powerful performance primitives, and others improve developer experience and integration with the rest of the web.

You can find the complete list of proposals and their respective stages in the official repo or track their implementation status in engines on the official feature roadmap page.

To ensure that users of all browsers can use your application, you need to figure out which features you want to use. Then, split them up into groups based on browser support. Then, compile your codebase separately for each of those groups. Finally, on the browser side you need to detect supported features and load the corresponding JavaScript and Wasm bundle.

Picking and grouping features #

Let's walk through those steps by picking some arbitrary feature set as an example. Let's say I've identified that I want to use SIMD, threads, and exception handling in my library for size and performance reasons. Their browser support is as follows:

View this feature table on webassembly.org/roadmap.

You can split browsers into the following cohorts to make sure that each user gets the most optimized experience:

Chrome-based browsers: Threads, SIMD, and exception handling are all supported.
Firefox: Thread and SIMD are supported, exception handling is not.
Safari: Threads are supported, SIMD and exception handling are not.
Other browsers: assume only baseline WebAssembly support.

This breakdown splits by feature support in the latest version of each browser. Modern browsers are evergreen and auto-update, so in the majority of cases you only need to worry about the latest release. However, as long as you include baseline WebAssembly as a fallback cohort, you can still provide a working application even for users with outdated browsers.

Compiling for different feature sets #

WebAssembly doesn't have a built-in way to detect supported features in runtime, therefore all instructions in the module must be supported on the target. Because of that, you need to compile the source code into Wasm separately for each of those different feature sets.

Each toolchain and build system is different, and you'll need to consult the documentation of your own compiler for how to tweak those features. For the sake of simplicity, I'll use a single-file C++ library in the following example and show how to compile it with Emscripten.

I'll use SIMD via SSE2 emulation, threads via Pthreads library support, and choose between Wasm exception handling and the fallback JavaScript implementation:

# First bundle: threads + SIMD + Wasm exceptions
$ emcc main.cpp -o main.threads-simd-exceptions.mjs -pthread -msimd128 -msse2 -fwasm-exceptions
# Second bundle: threads + SIMD + JS exceptions fallback
$ emcc main.cpp -o main.threads-simd.mjs -pthread -msimd128 -msse2 -fexceptions
# Third bundle: threads + JS exception fallback
$ emcc main.cpp -o main.threads.mjs -pthread -fexceptions
# Fourth bundle: basic Wasm with JS exceptions fallback
$ emcc main.cpp -o main.basic.mjs -fexceptions

The C++ code itself can use #ifdef __EMSCRIPTEN_PTHREADS__ and #ifdef __SSE2__ to conditionally choose between parallel (threads and SIMD) implementations of the same functions and the serial implementations at compile-time. It would look like this:

void process_data(std::vector<int>& some_input) {
#ifdef __EMSCRIPTEN_PTHREADS__
#ifdef __SSE2__
  // …implementation using threads and SIMD for max speed
#else
  // …implementation using threads but not SIMD
#endif
#else
  // …fallback implementation for browsers without those features
#endif
}

The exception handling doesn't need #ifdef directives, because it can be used in the same way from C++ regardless of the underlying implementation chosen via the compilation flags.

Loading the correct bundle #

Once you have built bundles for all feature cohorts, you need to load the correct one from the main JavaScript application. To do that, first, detect which features are supported in the current browser. You can do that with the wasm-feature-detect library. By combining it with dynamic import, you can load the most optimized bundle in any browser:

import { simd, threads, exceptions } from 'https://unpkg.com/wasm-feature-detect?module';

let initModule;
if (await threads()) {
  if (await simd()) {
    if (await exceptions()) {
      initModule = import('./main.threads-simd-exceptions.mjs');
    } else {
      initModule = import('./main.threads-simd.mjs');
    }
  } else {
    initModule = import('./main.threads.mjs');
  }
} else {
  initModule = import('./main.basic.mjs');
}

const Module = await initModule();
// now you can use `Module` Emscripten object like you normally would

Final words #

In this post, I've shown how to choose, build and switch between bundles for different feature sets.

As the number of features grows,the number of feature cohorts may become unmaintainable. To alleviate this problem, you can choose feature cohorts based on your real-world user data, skip the less popular browsers and let them fall back to slightly less optimal cohorts. As long as your application still works for all users, this approach can provide a reasonable balance between progressive enhancement and runtime performance.

In the future, WebAssembly might get a built-in way to detect supported features and switch between different implementations of the same function within the module. However, such a mechanism would itself be a post-MVP feature that you would need to detect and load conditionally using the approach above. Until then, this approach remains the only way to build and load code using new WebAssembly features across all browsers.

Porting USB applications to the web. Part 1: libusb

2022-01-20T00:00:00Z

In a previous post, I showed how to port apps using filesystem APIs to the web with File System Access API, WebAssembly and Asyncify. Now I want to continue the same topic of integrating Fugu APIs with WebAssembly and porting apps to the web without losing important features.

I'll show how apps that communicate with USB devices can be ported to the web by porting libusb—a popular USB library written in C—to WebAssembly (via Emscripten), Asyncify and WebUSB.

First things first: a demo #

The most important thing to do when porting a library is choosing the right demo—something that would showcase the capabilities of the ported library, allowing you to test it in a variety of ways, and be visually compelling at the same time.

The idea I chose was DSLR remote control. In particular, an open source project gPhoto2 has been in this space long enough to reverse-engineer and implement support for a wide variety of digital cameras. It supports several protocols, but the one I was most interested in was USB support, which it performs via libusb.

I'll describe the steps for building this demo in two parts. In this blog post, I'll describe how I ported libusb itself, and what tricks might be necessary to port other popular libraries to Fugu APIs. In the second post, I'll go into details on porting and integrating gPhoto2 itself.

In the end, I got a working web application that previews live feed from a DSLR and can control its settings over USB. Feel free to check out the live or the pre-recorded demo before reading up on technical details:

The demo running on a laptop connected to a Sony camera.

Note on camera-specific quirks #

You might have noticed that changing settings takes a while in the video. Like with most other issues you might see, this is not caused by the performance of WebAssembly or WebUSB, but by how gPhoto2 interacts with the specific camera chosen for the demo.

Sony a6600 doesn't expose an API to set values like ISO, aperture or shutter speed directly, and instead only provides commands to increase or decrease them by the specified number of steps. To make matters more complicated, it doesn't return a list of the actually supported values, either—the returned list seems hardcoded across many Sony camera models.

When setting one of those values, gPhoto2 has no other choice but to:

Make a step (or a few) in the direction of the chosen value.
Wait a bit for the camera to update the settings.
Read back the value the camera actually landed on.
Check that the last step didn't jump over the desired value nor wrapped around the end or the beginning of the list.
Repeat.

It can take some time, but if the value is actually supported by the camera, it will get there, and, if not, it will stop on the nearest supported value.

Other cameras will likely have different sets of settings, underlying APIs, and quirks. Keep in mind that gPhoto2 is an open-source project, and either automated or manual testing of all the camera models out there is simply not feasible, so detailed issue reports and PRs are always welcome (but make sure to reproduce the issues with the official gPhoto2 client first).

Important cross-platform compatibility notes #

Unfortunately, on Windows any "well-known" devices, including DSLR cameras, are assigned a system driver, which is not compatible with WebUSB. If you want to try the demo on Windows, you'll have to use a tool like Zadig to override the driver for the connected DSLR to either WinUSB or libusb. This approach works fine for me and many other users, but you should use it at your own risk.

On Linux, you will likely need to set custom permissions to allow access to your DSLR via WebUSB, although this depends on your distribution.

On macOS and Android, the demo should work out of the box. If you're trying it on an Android phone, make sure to switch to the landscape mode as I didn't put much effort into making it responsive (PRs are welcome!):

The same demo running on an Android phone. Picture by Surma.

For a more in-depth guide on cross-platform usage of WebUSB, see the "Platform-specific considerations" section of "Building a device for WebUSB".

Adding a new backend to libusb #

Now onto the technical details. While it's possible to provide a shim API similar to libusb (this has been done by others before) and link other applications against it, this approach is error-prone and makes any further extension or maintenance harder. I wanted to do things right, in a way that could be potentially contributed back upstream and merged into libusb in the future.

Luckily, the libusb README says:

“libusb is abstracted internally in such a way that it can hopefully be ported to other operating systems. Please see the PORTING file for more information.”

libusb is structured in a way where the public API is separate from "backends". Those backends are responsible for listing, opening, closing and actually communicating to the devices via the operating system's low-level APIs. This is how libusb already abstracts away differences between Linux, macOS, Windows, Android, OpenBSD/NetBSD, Haiku and Solaris and works on all these platforms.

What I had to do was add another backend for the Emscripten+WebUSB "operating system". The implementations for those backends live in the libusb/os folder:

~/w/d/libusb $ ls libusb/os
darwin_usb.c           haiku_usb_raw.h  threads_posix.lo
darwin_usb.h           linux_netlink.c  threads_posix.o
events_posix.c         linux_udev.c     threads_windows.c
events_posix.h         linux_usbfs.c    threads_windows.h
events_posix.lo        linux_usbfs.h    windows_common.c
events_posix.o         netbsd_usb.c     windows_common.h
events_windows.c       null_usb.c       windows_usbdk.c
events_windows.h       openbsd_usb.c    windows_usbdk.h
haiku_pollfs.cpp       sunos_usb.c      windows_winusb.c
haiku_usb_backend.cpp  sunos_usb.h      windows_winusb.h
haiku_usb.h            threads_posix.c
haiku_usb_raw.cpp      threads_posix.h

Each backend includes the libusbi.h header with common types and helpers, and needs to expose a usbi_backend variable of type usbi_os_backend. For example, this is what the Windows backend looks like:

const struct usbi_os_backend usbi_backend = {
  "Windows",
  USBI_CAP_HAS_HID_ACCESS,
  windows_init,
  windows_exit,
  windows_set_option,
  windows_get_device_list,
  NULL,   /* hotplug_poll */
  NULL,   /* wrap_sys_device */
  windows_open,
  windows_close,
  windows_get_active_config_descriptor,
  windows_get_config_descriptor,
  windows_get_config_descriptor_by_value,
  windows_get_configuration,
  windows_set_configuration,
  windows_claim_interface,
  windows_release_interface,
  windows_set_interface_altsetting,
  windows_clear_halt,
  windows_reset_device,
  NULL,   /* alloc_streams */
  NULL,   /* free_streams */
  NULL,   /* dev_mem_alloc */
  NULL,   /* dev_mem_free */
  NULL,   /* kernel_driver_active */
  NULL,   /* detach_kernel_driver */
  NULL,   /* attach_kernel_driver */
  windows_destroy_device,
  windows_submit_transfer,
  windows_cancel_transfer,
  NULL,   /* clear_transfer_priv */
  NULL,   /* handle_events */
  windows_handle_transfer_completion,
  sizeof(struct windows_context_priv),
  sizeof(union windows_device_priv),
  sizeof(struct windows_device_handle_priv),
  sizeof(struct windows_transfer_priv),
};

Looking through the properties, we can see that the struct includes the backend name, a set of its capabilities, handlers for various low-level USB operations in form of function pointers, and, finally, sizes to allocate for storing private device-/context-/transfer-level data.

The private data fields are useful at least for storing OS handles to all those things, as without handles we don't know which item any given operation applies to. In the web implementation, the OS handles would be the underlying WebUSB JavaScript objects. The natural way to represent and store them in Emscripten is via the emscripten::val class, which is provided as part of Embind (Emscripten's bindings system).

Most of the backends in the folder are implemented in C, but a few are implemented in C++. Embind only works with C++, so the choice was made for me and I've added libusb/libusb/os/emscripten_webusb.cpp with the required structure and with sizeof(val) for the private data fields:

#include <emscripten.h>
#include <emscripten/val.h>

#include "libusbi.h"

using namespace emscripten;

// …function implementations

const usbi_os_backend usbi_backend = {
  .name = "Emscripten + WebUSB backend",
  .caps = LIBUSB_CAP_HAS_CAPABILITY,
  // …handlers—function pointers to implementations above
  .device_priv_size = sizeof(val),
  .transfer_priv_size = sizeof(val),
};

Storing WebUSB objects as device handles #

libusb provides ready-to-use pointers to the allocated area for private data. To work with those pointers as val instances, I've added small helpers that construct them in-place, retrieve them as references, and move values out:

// We store an Embind handle to WebUSB USBDevice in "priv" metadata of
// libusb device, this helper returns a pointer to it.
struct ValPtr {
 public:
  void init_to(val &&value) { new (ptr) val(std::move(value)); }

  val &get() { return *ptr; }
  val take() { return std::move(get()); }

 protected:
  ValPtr(val *ptr) : ptr(ptr) {}

 private:
  val *ptr;
};

struct WebUsbDevicePtr : ValPtr {
 public:
  WebUsbDevicePtr(libusb_device *dev)
      : ValPtr(static_cast<val *>(usbi_get_device_priv(dev))) {}
};

val &get_web_usb_device(libusb_device *dev) {
  return WebUsbDevicePtr(dev).get();
}

struct WebUsbTransferPtr : ValPtr {
 public:
  WebUsbTransferPtr(usbi_transfer *itransfer)
      : ValPtr(static_cast<val *>(usbi_get_transfer_priv(itransfer))) {}
};

Async web APIs in synchronous C contexts #

Now needed a way to handle async WebUSB APIs where libusb expects synchronous operations. For this, I could use Asyncify, or, more specifically, its Embind integration via val::await().

I also wanted to correctly handle WebUSB errors and convert them into libusb error codes, but Embind currently doesn't have any way to handle JavaScript exceptions or Promise rejections from the C++ side. This problem can be worked around by catching a rejection on the JavaScript side and converting the result into an { error, value } object that can be now safely parsed from the C++ side. I did this with a combination of the EM_JS macro and Emval.to{Handle, Value} APIs:

EM_JS(EM_VAL, em_promise_catch_impl, (EM_VAL handle), {
  let promise = Emval.toValue(handle);
  promise = promise.then(
    value => ({error : 0, value}),
    error => {
      const ERROR_CODES = {
        // LIBUSB_ERROR_IO
        NetworkError : -1,
        // LIBUSB_ERROR_INVALID_PARAM
        DataError : -2,
        TypeMismatchError : -2,
        IndexSizeError : -2,
        // LIBUSB_ERROR_ACCESS
        SecurityError : -3,
        …
      };
      console.error(error);
      let errorCode = -99; // LIBUSB_ERROR_OTHER
      if (error instanceof DOMException)
      {
        errorCode = ERROR_CODES[error.name] ?? errorCode;
      }
      else if (error instanceof RangeError || error instanceof TypeError)
      {
        errorCode = -2; // LIBUSB_ERROR_INVALID_PARAM
      }
      return {error: errorCode, value: undefined};
    }
  );
  return Emval.toHandle(promise);
});

val em_promise_catch(val &&promise) {
  EM_VAL handle = promise.as_handle();
  handle = em_promise_catch_impl(handle);
  return val::take_ownership(handle);
}

// C++ struct representation for {value, error} object from above
// (performs conversion in the constructor).
struct promise_result {
  libusb_error error;
  val value;

  promise_result(val &&result)
      : error(static_cast<libusb_error>(result["error"].as<int>())),
        value(result["value"]) {}

  // C++ counterpart of the promise helper above that takes a promise, catches
  // its error, converts to a libusb status and returns the whole thing as
  // `promise_result` struct for easier handling.
  static promise_result await(val &&promise) {
    promise = em_promise_catch(std::move(promise));
    return {promise.await()};
  }
};

Now I could use promise_result::await() on any Promise returned from WebUSB operations and inspect its error and value fields separately.

For example, retrieving a val representing a USBDevice from libusb_device_handle, calling its open() method, awaiting its result, and returning an error code as a libusb status code looks like this:

int em_open(libusb_device_handle *handle) {
  auto web_usb_device = get_web_usb_device(handle->dev);
  return promise_result::await(web_usb_device.call<val>("open")).error;
}

Device enumeration #

Of course, before I can open any device, libusb needs to retrieve a list of available devices. The backend must implement this operation via a get_device_list handler.

The difficulty is that, unlike on other platforms, there is no way to enumerate all the connected USB devices on the web for security reasons. Instead, the flow is split into two parts. First, the web application requests devices with specific properties via navigator.usb.requestDevice() and the user manually chooses which device they want to expose or rejects the permission prompt. Afterwards, the application lists the already approved and connected devices via navigator.usb.getDevices().

At first I tried to use requestDevice() directly in the implementation of the get_device_list handler. However, showing a permission prompt with a list of connected devices is considered a sensitive operation, and it must be triggered by user interaction (like a button click on a page), otherwise it always returns a rejected promise. libusb applications might often want to list the connected devices upon application start-up, so using requestDevice() was not an option.

Instead, I had to leave invocation of navigator.usb.requestDevice() to the end developer, and only expose the already approved devices from navigator.usb.getDevices():

// Store the global `navigator.usb` once upon initialisation.
thread_local const val web_usb = val::global("navigator")["usb"];

int em_get_device_list(libusb_context *ctx, discovered_devs **devs) {
  // C++ equivalent of `await navigator.usb.getDevices()`.
  // Note: at this point we must already have some devices exposed -
  // caller must have called `await navigator.usb.requestDevice(...)`
  // in response to user interaction before going to LibUSB.
  // Otherwise this list will be empty.
  auto result = promise_result::await(web_usb.call<val>("getDevices"));
  if (result.error) {
    return result.error;
  }
  auto &web_usb_devices = result.value;
  // Iterate over the exposed devices.
  uint8_t devices_num = web_usb_devices["length"].as<uint8_t>();
  for (uint8_t i = 0; i < devices_num; i++) {
    auto web_usb_device = web_usb_devices[i];
    // …
    *devs = discovered_devs_append(*devs, dev);
  }
  return LIBUSB_SUCCESS;
}

Most of the backend code uses val and promise_result in a similar way as already shown above. There are few more interesting hacks in the data transfer handling code, but those implementation details are less important for the purposes of this article. Make sure to check the code and comments on Github if you're interested.

Porting event loops to the web #

One more piece of the libusb port that I want to discuss is event handling. As described in the previous article, most APIs in system languages like C are synchronous, and event handling is no exception. It's usually implemented via an infinite loop that "polls" (tries to read data or blocks execution until some data is available) from a set of external I/O sources, and, when at least one of those responds, passes that as an event to the corresponding handler. Once the handler is finished, the control returns to the loop, and it pauses for another poll.

There are a couple of problems with this approach on the web.

First, WebUSB doesn't and cannot expose raw handles of the underlying devices, so polling those directly is not an option. Second, libusb uses eventfd and pipe APIs for other events as well as for handling transfers on operating systems without raw device handles, but eventfd is not currently supported in Emscripten, and pipe, while supported, currently doesn't conform to the spec and can't wait for events.

Finally, the biggest problem is that the web has its own event loop. This global event loop is used for any external I/O operations (including fetch(), timers, or, in this case, WebUSB), and it invokes event or Promise handlers whenever corresponding operations finish. Executing another, nested, infinite event loop will block the browser's event loop from ever progressing, which means that not only will the UI become unresponsive, but also that the code will never get notifications for the very same I/O events it's waiting for. This usually results in a deadlock, and that's what happened when I tried to use libusb in a demo, too. The page froze.

Like with other blocking I/O, to port such event loops to the web, developers need to find a way to run those loops without blocking the main thread. One way is to refactor the application to handle I/O events in a separate thread and pass the results back to the main one. The other is to use Asyncify to pause the loop and wait for events in a non-blocking fashion.

I didn't want to do significant changes to either libusb or gPhoto2, and I've already used Asyncify for Promise integration, so that's the path I've chosen. To simulate a blocking variant of poll(), for the initial proof of concept I've used a loop as shown below:

#ifdef __EMSCRIPTEN__
  // TODO: optimize this. Right now it will keep unwinding-rewinding the stack
  // on each short sleep until an event comes or the timeout expires.
  // We should probably create an actual separate thread that does signaling
  // or come up with a custom event mechanism to report events from
  // `usbi_signal_event` and process them here.
  double until_time = emscripten_get_now() + timeout_ms;
  do {
    // Emscripten `poll` ignores timeout param, but pass 0 explicitly just
    // in case.
    num_ready = poll(fds, nfds, 0);
    if (num_ready != 0) break;
    // Yield to the browser event loop to handle events.
    emscripten_sleep(0);
  } while (emscripten_get_now() < until_time);
#else
  num_ready = poll(fds, nfds, timeout_ms);
#endif

What it does is:

Calls poll() to check if any events were reported by the backend yet. If there are some, the loop stops. Otherwise Emscripten's implementation of poll() will immediately return with 0.
Calls emscripten_sleep(0). This function uses Asyncify and setTimeout() under the hood and is used here to yield control back to the main browser event loop. This allows the browser to handle any user interactions and I/O events, including WebUSB.
Check if the specified timeout has expired yet, and, if not, continue the loop.

As the comment mentions, this approach was not optimal, because it kept saving-restoring the entire call stack with Asyncify even when there were no USB events to handle yet (which is most of the time), and because setTimeout() itself has a minimal duration of 4ms in modern browsers. Still, it worked well enough to produce 13-14 FPS livestream from DSLR in the proof-of-concept.

Later, I decided to improve it by leveraging the browser event system. There are several ways in which this implementation could be improved further, but for now I've chosen to emit custom events directly on the global object, without associating them with a particular libusb data structure. I've done so via the following wait and notify mechanism based on the EM_ASYNC_JS macro:

EM_JS(void, em_libusb_notify, (void), {
  dispatchEvent(new Event("em-libusb"));
});

EM_ASYNC_JS(int, em_libusb_wait, (int timeout), {
  let onEvent, timeoutId;

  try {
    return await new Promise(resolve => {
      onEvent = () => resolve(0);
      addEventListener('em-libusb', onEvent);

      timeoutId = setTimeout(resolve, timeout, -1);
    });
  } finally {
    removeEventListener('em-libusb', onEvent);
    clearTimeout(timeoutId);
  }
});

The em_libusb_notify() function is used whenever libusb tries to report an event, such as data transfer completion:

void usbi_signal_event(usbi_event_t *event)
{
  uint64_t dummy = 1;
  ssize_t r;

  r = write(EVENT_WRITE_FD(event), &dummy, sizeof(dummy));
  if (r != sizeof(dummy))
    usbi_warn(NULL, "event write failed");
#ifdef __EMSCRIPTEN__
  em_libusb_notify();
#endif
}

Meanwhile, the em_libusb_wait() part is used to "wake up" from Asyncify sleep when either an em-libusb event is received, or the timeout has expired:

double until_time = emscripten_get_now() + timeout_ms;
for (;;) {
  // Emscripten `poll` ignores timeout param, but pass 0 explicitly just
  // in case.
  num_ready = poll(fds, nfds, 0);
  if (num_ready != 0) break;
  int timeout = until_time - emscripten_get_now();
  if (timeout <= 0) break;
  int result = em_libusb_wait(timeout);
  if (result != 0) break;
}

Due to significant reduction in sleeps and wake-ups, this mechanism fixed the efficiency problems of the earlier emscripten_sleep()-based implementation, and increased the DSLR demo throughput from 13-14 FPS to consistent 30+ FPS, which is enough for a smooth live feed.

Build system and the first test #

After the backend was done, I had to add it to Makefile.am and configure.ac. The only interesting bit here is Emscripten-specific flags modification:

emscripten)
  AC_SUBST(EXEEXT, [.html])
  # Note: LT_LDFLAGS is not enough here because we need link flags for executable.
  AM_LDFLAGS="${AM_LDFLAGS} --bind -s ASYNCIFY -s ASSERTIONS -s ALLOW_MEMORY_GROWTH -s INVOKE_RUN=0 -s EXPORTED_RUNTIME_METHODS=['callMain']"
  ;;

First, executables on Unix platforms normally don't have file extensions. Emscripten, however, produces different output depending on which extension you request. I'm using AC_SUBST(EXEEXT, …) to change the executable extension to .html so that any executable within a package—tests and examples—becomes an HTML with Emscripten's default shell that takes care of loading and instantiating JavaScript and WebAssembly.

Second, because I'm using Embind and Asyncify, I need to enable those features (--bind -s ASYNCIFY) as well as allow dynamic memory growth (-s ALLOW_MEMORY_GROWTH) via linker parameters. Unfortunately, there is no way for a library to report those flags to the linker, so every application that uses this libusb port will have to add the same linker flags into their build configuration as well.

Finally, as mentioned earlier, WebUSB requires device enumeration to be done via a user gesture. libusb examples and tests assume that they can enumerate devices at start-up, and fail with an error without changes. Instead, I had to disable automatic execution (-s INVOKE_RUN=0) and expose the manual callMain() method (-s EXPORTED_RUNTIME_METHODS=...).

Once all of this was done, I could serve the generated files with a static web server, initialize WebUSB, and run those HTML executables manually with the help of DevTools.

It doesn't look like much, but, when porting libraries to a new platform, getting to the stage where it produces a valid output for the first time is pretty exciting!

Using the port #

As mentioned above, the port depends on a few Emscripten features that currently need to be enabled at the linking stage of the application. If you want to use this libusb port in your own application, here's what you'll need to do:

Download the latest libusb either as an archive as part of your build or add it as a git submodule in your project.
Run autoreconf -fiv in the libusb folder.
Run emconfigure ./configure –host=wasm32 –prefix=/some/installation/path to initialize the project for cross-compilation and to set a path where you want to put the built artifacts.
Run emmake make install.
Point your application or higher-level library to search for the libusb under the earlier chosen path.
Add the following flags to your application's link arguments: --bind -s ASYNCIFY -s ALLOW_MEMORY_GROWTH.

The library currently has a few limitations:

No transfer cancellation support. This is a limitation of WebUSB, which, in turn, stems from lack of cross-platform transfer cancellation in libusb itself.
No isochronous transfer support. It shouldn't be hard to add it by following the implementation of existing transfer modes as examples, but it's also a somewhat rare mode and I didn't have any devices to test it on, so for now I left it as unsupported. If you do have such devices, and want to contribute to the library, PRs are welcome!
The earlier mentioned cross-platform limitations. Those limitations are imposed by operating systems, so not much we can do here, except ask users to override the driver or permissions. However, if you're porting HID or serial devices, you can follow the libusb example and port some other library to another Fugu API. For example, you could port a C library hidapi to WebHID and side-step those issues, associated with low-level USB access, altogether.

Conclusion #

In this post I've shown how, with the help of Emscripten, Asyncify and Fugu APIs even low-level libraries like libusb can be ported to the web with a few integration tricks.

Porting such essential and widely used low-level libraries is particularly rewarding, because, in turn, it allows bringing higher-level libraries or even whole applications to the web, too. This opens experiences that were previously limited to users of one or two platforms, to all kinds of devices and operating systems, making those experiences available just a link click away.

In the next post I'll walk through the steps involved in building the web gPhoto2 demo which not only retrieves device information, but extensively uses the transfer feature of libusb too. Meanwhile, I hope you found the libusb example inspiring and will try out the demo, play with the library itself, or perhaps even go ahead and port another widely used library to one of the Fugu APIs too.

Embedding JavaScript snippets in C++ with Emscripten

2022-01-18T00:00:00Z

When working on WebAssembly integration with the web, you need a way to call out to external APIs such as web APIs and third-party libraries. You then need a way to store the values and object instances those APIs return, and a way to pass those stored values to other APIs later. For asynchronous APIs, you might also need to await promises in your synchronous C/C++ code with Asyncify and read the result once the operation is finished.

Emscripten provides several tools for such interactions:

emscripten::val for storing and operating on JavaScript values in C++.
EM_JS for embedding JavaScript snippets and binding them as C/C++ functions.
EM_ASYNC_JS that's similar to EM_JS, but makes it easier to embed asynchronous JavaScript snippets.
EM_ASM for embedding short snippets and executing them inline, without declaring a function.
--js-library for advanced scenarios where you want to declare lots of JavaScript functions together as a single library.

In this post you’ll learn how to use all of them for similar tasks.

emscripten::val class #

The emcripten::val class is provided by Embind. It can invoke global APIs, bind JavaScript values to C++ instances, and convert values between C++ and JavaScript types.

Here's how to use it with Asyncify's .await() to fetch and parse some JSON:

#include <emscripten/val.h>

using namespace emscripten;

val fetch_json(const char *url) {
  // Get and cache a binding to the global `fetch` API in each thread.
  thread_local const val fetch = val::global("fetch");
  // Invoke fetch and await the returned `Promise<Response>`.
  val response = fetch(url).await();
  // Ask to read the response body as JSON and await the returned `Promise<any>`.
  val json = response.call<val>("json").await();
  // Return the JSON object.
  return json;
}

// Example URL.
val example_json = fetch_json("https://httpbin.org/json");

// Now we can extract fields, e.g.
std::string author = json["slideshow"]["author"].as<std::string>();

This code works well, but it performs lots of intermediate steps. Each operation on val needs to perform the following steps:

Convert C++ values passed as arguments into some intermediate format.
Go to JavaScript, read and convert arguments into JavaScript values.
Execute the function
Convert the result from JavaScript to intermediate format.
Return the converted result to C++, and C++ finally reads it back.

Each await() also has to pause the C++ side by unwinding the entire call stack of the WebAssembly module, returning to JavaScript, waiting, and restoring the WebAssembly stack when the operation is complete.

Such code doesn't need anything from C++. C++ code is acting only as a driver for a series of JavaScript operations. What if you could move fetch_json to JavaScript and reduce the overhead of intermediate steps at the same time?

EM_JS macro #

The EM_JS macro lets you move fetch_json to JavaScript. EM_JS in Emscripten lets you declare a C/C++ function that is implemented by a JavaScript snippet.

Like WebAssembly itself, it has a limitation of supporting only numeric arguments and return values. In order to pass any other values, you need to convert them manually via corresponding APIs. Here are some examples.

Passing numbers doesn't need any conversion:

// Passing numbers, doesn't need any conversion.
EM_JS(int, add_one, (int x), {
  return x + 1;
});

int x = add_one(41);

When passing strings to and from JavaScript you need to use the corresponding conversion and allocation functions from preamble.js:

EM_JS(void, log_string, (const char *msg), {
  console.log(UTF8ToString(msg));
});

EM_JS(const char *, get_input, (), {
  let str = document.getElementById('myinput').value;
  // Returns heap-allocated string.
  // C/C++ code is responsible for calling `free` once unused.
  return allocate(intArrayFromString(str), 'i8', ALLOC_NORMAL);
});

Finally, for more complex, arbitrary, value types, you can use the JavaScript API for the earlier mentioned val class. Using it, you can convert JavaScript values and C++ classes into intermediate handles and back:

EM_JS(void, log_value, (EM_VAL val_handle), {
  let value = Emval.toValue(val_handle);
  console.log(value);
});

EM_JS(EM_VAL, find_myinput, (), {
  let input = document.getElementById('myinput');
  return Emval.toHandle(input);
});

val obj = val::object();
obj.set("x", 1);
obj.set("y", 2);
log_value(obj.as_handle()); // logs { x: 1, y: 2 }

val myinput = val::take_ownership(find_input());
// Now you can store the `find_myinput` DOM element for as long as you like, and access it later like:
std::string value = input["value"].as<std::string>();

With those APIs in mind, the fetch_json example could be rewritten to do most work without leaving JavaScript:

EM_JS(EM_VAL, fetch_json, (const char *url), {
  return Asyncify.handleAsync(async () => {
    url = UTF8ToString(url);
    // Invoke fetch and await the returned `Promise<Response>`.
    let response = await fetch(url);
    // Ask to read the response body as JSON and await the returned `Promise<any>`.
    let json = await response.json();
    // Convert JSON into a handle and return it.
    return Emval.toHandle(json);
  });
});

// Example URL.
val example_json = val::take_ownership(fetch_json("https://httpbin.org/json"));

// Now we can extract fields, e.g.
std::string author = json["slideshow"]["author"].as<std::string>();

We still have a couple of explicit conversions at the entry and exit points of the function, but the rest is now regular JavaScript code. Unlike val equivalent, it can now be optimized by the JavaScript engine and only requires pausing the C++ side once for all async operations.

EM_ASYNC_JS macro #

The only bit left that does not look pretty is the Asyncify.handleAsync wrapper—its only purpose is to allow executing async JavaScript functions with Asyncify. In fact, this use case is so common that there is now a specialized EM_ASYNC_JS macro that combines them together.

Here's how you could use it to produce the final version of the fetch example:

EM_ASYNC_JS(EM_VAL, fetch_json, (const char *url), {
  url = UTF8ToString(url);
  // Invoke fetch and await the returned `Promise<Response>`.
  let response = await fetch(url);
  // Ask to read the response body as JSON and await the returned `Promise<any>`.
  let json = await response.json();
  // Convert JSON into a handle and return it.
  return Emval.toHandle(json);
});

// Example URL.
val example_json = val::take_ownership(fetch_json("https://httpbin.org/json"));

// Now we can extract fields, e.g.
std::string author = json["slideshow"]["author"].as<std::string>();

EM_ASM #

EM_JS is the recommended way to declare JavaScript snippets. It's efficient because it binds the declared snippets directly like any other JavaScript function imports. It also provides good ergonomics by enabling you to explicitly declare all parameter types and names.

In some cases, however, you want to insert a quick snippet for console.log call, a debugger; statement or something similar and don't want to bother with declaring a whole separate function. In those rare cases, an EM_ASM macros family (EM_ASM, EM_ASM_INT and EM_ASM_DOUBLE) might be a simpler choice. Those macros are similar to the EM_JS macro, but they execute code inline where they're inserted, instead of defining a function.

Since they don't declare a function prototype, they need a different way of specifying the return type and accessing arguments.

You need to use the right macro name to choose the return type. EM_ASM blocks are expected to act like void functions, EM_ASM_INT blocks can return an integer value, and EM_ASM_DOUBLE blocks return floating-point numbers correspondingly.

Any passed arguments will be available under names $0, $1, and so on in the JavaScript body. As with EM_JS or WebAssembly in general, the arguments are limited only to numeric values—integers, floating-point numbers, pointers and handles.

Here's an example of how you could use an EM_ASM macro to log an arbitrary JS value to the console:

val obj = val::object();
obj.set("x", 1);
obj.set("y", 2);
// executes inline immediately
EM_ASM({
  // convert handle passed under $0 into a JavaScript value
  let obj = Emval.fromHandle($0);
  console.log(obj); // logs { x: 1, y: 2 }
}, obj.as_handle());

--js-library #

Finally, Emscripten supports declaring JavaScript code in a separate file in a customits own library format:

mergeInto(LibraryManager.library, {
  log_value: function (val_handle) {
    let value = Emval.toValue(val_handle);
    console.log(value);
  }
});

Then you need to declare corresponding prototypes manually on the C++ side:

extern "C" void log_value(EM_VAL val_handle);

Once declared on both sides, the JavaScript library can be linked together with the main code via the --js-library option, connecting prototypes with corresponding JavaScript implementations.

However, this module format is non-standard and requires careful dependency annotations. As such, it's mostly reserved for advanced scenarios.

Conclusion #

In this post we've looked at various ways to integrate JavaScript code into C++ when working with WebAssembly.

Including such snippets allows you to express long sequences of operations in a cleaner and more efficient way, and to tap into third-party libraries, new JavaScript APIs, and even JavaScript syntax features that are not yet expressible via C++ or Embind.

Bundling non-JavaScript resources

2021-09-08T00:00:00Z

Suppose you're working on a web app. In that case, it's likely that you have to deal not only with JavaScript modules, but also with all sorts of other resources—Web Workers (which are also JavaScript, but not part of the regular module graph), images, stylesheets, fonts, WebAssembly modules and others.

It's possible to include references to some of those resources directly in the HTML, but often they're logically coupled to reusable components. For example, a stylesheet for a custom dropdown tied to its JavaScript part, icon images tied to a toolbar component, or WebAssembly module tied to its JavaScript glue. In those cases, it's more convenient to reference the resources directly from their JavaScript modules and load them dynamically when (or if) the corresponding component is loaded.

However, most large projects have build systems that perform additional optimizations and reorganization of content—for example, bundling and minification. They can't execute the code and predict what the result of execution will be, nor can they traverse every possible string literal in JavaScript and make guesses about whether it's a resource URL or not. So how can you make them "see" those dynamic assets loaded by JavaScript components, and include them in the build?

Custom imports in bundlers #

One common approach is to reuse the static import syntax. In some bundlers it might auto-detect format by the file extension, while others allow plugins to use a custom URL scheme like in the following example:

// regular JavaScript import
import { loadImg } from './utils.js';

// special "URL imports" for assets
import imageUrl from 'asset-url:./image.png';
import wasmUrl from 'asset-url:./module.wasm';
import workerUrl from 'js-url:./worker.js';

loadImg(imageUrl);
WebAssembly.instantiateStreaming(fetch(wasmUrl));
new Worker(workerUrl);

When a bundler plugin finds an import with either an extension it recognizes or such an explicit custom scheme (asset-url: and js-url: in the example above), it adds the referenced asset to the build graph, copies it to the final destination, performs optimizations applicable for the asset's type and returns the final URL to be used during runtime.

The benefits of this approach: reusing the JavaScript import syntax guarantees that all URLs are static and relative to the current file, which makes locating such dependencies easy for the build system.

However, it has one significant drawback: such code can't work directly in the browser, as the browser doesn't know how to handle those custom import schemes or extensions. This might be fine if you control all the code and rely on a bundler for development anyway, but it's increasingly common to use JavaScript modules directly in the browser, at least during development, to reduce the friction. Someone working on a small demo might not even need a bundler at all, even in production.

Universal pattern for browsers and bundlers #

If you're working on a reusable component, you'd want it to function in either environment, whether it's used directly in the browser or pre-built as part of a larger app. Most modern bundlers allow for this by accepting the following pattern in JavaScript modules:

new URL('./relative-path', import.meta.url)

This pattern can be detected statically by tools, almost as if it was a special syntax, yet it's a valid JavaScript expression that works directly in the browser, too.

When using this pattern, the example above can be rewritten as:

// regular JavaScript import
import { loadImg } from './utils.js';

loadImg(new URL('./image.png', import.meta.url));
WebAssembly.instantiateStreaming(
  fetch(new URL('./module.wasm', import.meta.url)),
  { /* … */ }
);
new Worker(new URL('./worker.js', import.meta.url));

How does it work? Let's break it up. The new URL(...) constructor takes a relative URL as the first argument and resolves it against an absolute URL provided as the second argument. In our case, the second argument is import.meta.url which gives the URL of the current JavaScript module, so the first argument can be any path relative to it.

It has similar tradeoffs to the dynamic import. While it's possible to use import(...) with arbitrary expressions like import(someUrl), the bundlers give special treatment to a pattern with static URL import('./some-static-url.js') as a way to preprocess a dependency known at compile-time, yet split it out into its own chunk that's loaded dynamically.

Similarly, you can use new URL(...) with arbitrary expressions like new URL(relativeUrl, customAbsoluteBase), yet the new URL('...', import.meta.url) pattern is a clear signal for bundlers to preprocess and include a dependency alongside the main JavaScript.

Ambiguous relative URLs #

You might be wondering, why can't bundlers detect other common patterns—for example, fetch('./module.wasm') without the new URL wrappers?

The reason is that, unlike import statements, any dynamic requests are resolved relatively to the document itself, and not to the current JavaScript file. Let's say you have the following structure:

index.html:

<script src="src/main.js" type="module"></script>

src/
- main.js
- module.wasm

If you want to load module.wasm from main.js, it might be tempting to use a relative path like fetch('./module.wasm').

However, fetch does not know the URL of the JavaScript file it's executed in, instead, it resolves URLs relatively to the document. As a result, fetch('./module.wasm') would end up trying to load http://example.com/module.wasm instead of the intended http://example.com/src/module.wasm and fail (or, worse, silently load a different resource than you intended).

By wrapping the relative URL into new URL('...', import.meta.url) you can avoid this problem and guarantee that any provided URL is resolved relative to the URL of the current JavaScript module (import.meta.url) before it's passed on to any loaders.

Replace fetch('./module.wasm') with fetch(new URL('./module.wasm', import.meta.url)) and it will successfully load the expected WebAssembly module, as well as give bundlers a way to find those relative paths during the build time too.

Tooling support #

Bundlers #

Following bundlers support the new URL scheme already:

Webpack v5
Rollup (Achieved via plugins—@web/rollup-plugin-import-meta-assets for generic assets and @surma/rollup-plugin-off-main-thread for Workers specifically.)
Parcel v2 (beta)
Vite

WebAssembly #

When working with WebAssembly, you will commonly not load the Wasm module by hand, but instead import the JavaScript glue emitted by the toolchain. The following toolchains can emit the described new URL(...) pattern under the hood for you.

C/C++ via Emscripten #

When using Emscripten, you can ask it to emit JavaScript glue as an ES6 module instead of a regular script via one of the following options:

$ emcc input.cpp -o output.mjs
## or, if you don't want to use .mjs extension
$ emcc input.cpp -o output.js -s EXPORT_ES6

When using this option, the output will use the new URL(..., import.meta.url) pattern under the hood, so that bundlers can find the associated Wasm file automatically.

You can also use this option with WebAssembly threads by adding a -pthread flag:

$ emcc input.cpp -o output.mjs -pthread
## or, if you don't want to use .mjs extension
$ emcc input.cpp -o output.js -s EXPORT_ES6 -pthread

In this case, the generated Web Worker will be included in the same fashion and will also be discoverable by bundlers and browsers alike.

Rust via wasm-pack / wasm-bindgen #

wasm-pack—the primary Rust toolchain for WebAssembly—also has several output modes.

By default, it will emit a JavaScript module that relies on the WebAssembly ESM integration proposal. At the moment of writing, this proposal is still experimental, and the output will work only when bundled with Webpack.

Instead, you can ask wasm-pack to emit a browser-compatible ES6 module via --target web:

$ wasm-pack build --target web

The output will use the described new URL(..., import.meta.url) pattern, and the Wasm file will be automatically discovered by bundlers as well.

If you want to use WebAssembly threads with Rust, the story is a bit more complicated. Check out the corresponding section of the guide to learn more.

Short version is that you can't use arbitrary thread APIs, but if you use Rayon, you can combine it with the wasm-bindgen-rayon adapter so that it can spawn Workers on the Web. The JavaScript glue used by wasm-bindgen-rayon also includes the new URL(...) pattern under the hood, and so the Workers will be discoverable and included by bundlers as well.

Future features #

import.meta.resolve #

A dedicated import.meta.resolve(...) call is a potential future improvement. It would allow resolving specifiers relatively to the current module in a more straightforward fashion, without extra params:

new URL('...', import.meta.url)
await import.meta.resolve('...')

It would also integrate better with import maps and custom resolvers as it would go through the same module resolution system as import. It would be a stronger signal for bundlers too as it's a static syntax that doesn't depend on runtime APIs like URL.

import.meta.resolve is already implemented as an experiment in Node.js but there are still some unresolved questions about how it should work on the web.

Import assertions #

Import assertions are a new feature that allows to import types other than ECMAScript modules. For now they're limited to JSON:

foo.json:

{ "answer": 42 }

main.mjs:

import json from './foo.json' assert { type: 'json' };
console.log(json.answer); // 42

They might also be used by bundlers and replace the use-cases currently covered by the new URL pattern, but types in import assertions are added on a per-case basis. For now they only cover JSON, with CSS modules coming up soon, but other kinds of assets will still require a more generic solution.

Check out the v8.dev feature explainer to learn more about this feature.

Conclusion #

As you can see, there are various ways to include non-JavaScript resources on the web, but they have various drawbacks and don't work across various toolchains. Future proposals might let us import such assets with specialized syntax, but we're not quite there yet.

Until then, the new URL(..., import.meta.url) pattern is the most promising solution that already works in browsers, various bundlers and WebAssembly toolchains today.

Using WebAssembly threads from C, C++ and Rust

2021-07-12T00:00:00Z

WebAssembly threads support is one of the most important performance additions to WebAssembly. It allows you to either run parts of your code in parallel on separate cores, or the same code over independent parts of the input data, scaling it to as many cores as the user has and significantly reducing the overall execution time.

In this article you will learn how to use WebAssembly threads to bring multithreaded applications written in languages like C, C++, and Rust to the web.

How WebAssembly threads work #

WebAssembly threads is not a separate feature, but a combination of several components that allows WebAssembly apps to use traditional multithreading paradigms on the web.

Web Workers #

First component is the regular Workers you know and love from JavaScript. WebAssembly threads use the new Worker constructor to create new underlying threads. Each thread loads a JavaScript glue, and then the main thread uses Worker#postMessage method to share the compiled WebAssembly.Module as well as a shared WebAssembly.Memory (see below) with those other threads. This establishes communication and allows all those threads to run the same WebAssembly code on the same shared memory without going through JavaScript again.

Web Workers have been around for over a decade now, are widely supported, and don't require any special flags.

`SharedArrayBuffer` #

WebAssembly memory is represented by a WebAssembly.Memory object in the JavaScript API. By default WebAssembly.Memory is a wrapper around an ArrayBuffer—a raw byte buffer that can be accessed only by a single thread.

> new WebAssembly.Memory({ initial:1, maximum:10 }).buffer
ArrayBuffer { … }

To support multithreading, WebAssembly.Memory gained a shared variant too. When created with a shared flag via the JavaScript API, or by the WebAssembly binary itself, it becomes a wrapper around a SharedArrayBuffer instead. It's a variation of ArrayBuffer that can be shared with other threads and read or modified simultaneously from either side.

> new WebAssembly.Memory({ initial:1, maximum:10, shared:true }).buffer
SharedArrayBuffer { … }

Unlike postMessage, normally used for communication between main thread and Web Workers, SharedArrayBuffer doesn't require copying data or even waiting for the event loop to send and receive messages. Instead, any changes are seen by all threads nearly instantly, which makes it a much better compilation target for traditional synchronisation primitives.

SharedArrayBuffer has a complicated history. It was initially shipped in several browsers mid-2017, but had to be disabled in the beginning of 2018 due to discovery of Spectre vulnerabilities. The particular reason was that data extraction in Spectre relies on timing attacks—measuring execution time of a particular piece of code. To make this kind of attack harder, browsers reduced precision of standard timing APIs like Date.now and performance.now. However, shared memory, combined with a simple counter loop running in a separate thread is also a very reliable way to get high-precision timing, and it's much harder to mitigate without significantly throttling runtime performance.

Instead, Chrome 68 (mid-2018) re-enabled SharedArrayBuffer again by leveraging Site Isolation—a feature that puts different websites into different processes and makes it much more difficult to use side-channel attacks like Spectre. However, this mitigation was still limited only to Chrome desktop, as Site Isolation is a fairly expensive feature, and couldn't be enabled by default for all sites on low-memory mobile devices nor was it yet implemented by other vendors.

Fast-forward to 2020, Chrome and Firefox both have implementations of Site Isolation, and a standard way for websites to opt-in to the feature with COOP and COEP headers. An opt-in mechanism allows to use Site Isolation even on low-powered devices where enabling it for all the websites would be too expensive. To opt-in, add the following headers to the main document in your server configuration:

Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-origin

Once you opt-in, you get access to SharedArrayBuffer (including WebAssembly.Memory backed by a SharedArrayBuffer), precise timers, memory measurement and other APIs that require an isolated origin for security reasons. Check out the Making your website "cross-origin isolated" using COOP and COEP for more details.

WebAssembly atomics #

While SharedArrayBuffer allows each thread to read and write to the same memory, for correct communication you want to make sure they don't perform conflicting operations at the same time. For example, it's possible for one thread to start reading data from a shared address, while another thread is writing to it, so the first thread will now get a corrupted result. This category of bugs is known as race conditions. In order to prevent race conditions, you need to somehow synchronize those accesses. This is where atomic operations come in.

WebAssembly atomics is an extension to the WebAssembly instruction set that allow to read and write small cells of data (usually 32- and 64-bit integers) "atomically". That is, in a way that guarantees that no two threads are reading or writing to the same cell at the same time, preventing such conflicts at a low level. Additionally, WebAssembly atomics contain two more instruction kinds—"wait" and "notify"—that allow one thread to sleep ("wait") on a given address in a shared memory until another thread wakes it up via "notify".

All the higher-level synchronisation primitives, including channels, mutexes, and read-write locks build upon those instructions.

How to use WebAssembly threads #

Feature detection #

WebAssembly atomics and SharedArrayBuffer are relatively new features and aren't yet available in all browsers with WebAssembly support. You can find which browsers support new WebAssembly features on the webassembly.org roadmap.

To ensure that all users can load your application, you'll need to implement progressive enhancement by building two different versions of Wasm—one with multithreading support and one without it. Then load the supported version depending on feature detection results. To detect WebAssembly threads support at runtime, use wasm-feature-detect library and load the module like this:

import { threads } from 'wasm-feature-detect';

const hasThreads = await threads();

const module = await (
  hasThreads
    ? import('./module-with-threads.js')
    : import('./module-without-threads.js')
);

// …now use `module` as you normally would

Now let's take a look at how to build a multithreaded version of the WebAssembly module.

C #

In C, particularly on Unix-like systems, the common way to use threads is via POSIX Threads provided by the pthread library. Emscripten provides an API-compatible implementation of the pthread library built atop Web Workers, shared memory and atomics, so that the same code can work on the web without changes.

Let's take a look at an example:

example.c:

#include <stdio.h>
#include <unistd.h>
#include <pthread.h>

void *thread_callback(void *arg)
{
    sleep(1);
    printf("Inside the thread: %d\n", *(int *)arg);
    return NULL;
}

int main()
{
    puts("Before the thread");

    pthread_t thread_id;
    int arg = 42;
    pthread_create(&thread_id, NULL, thread_callback, &arg);

    pthread_join(thread_id, NULL);

    puts("After the thread");

    return 0;
}

Here the headers for the pthread library are included via pthread.h. You can also see a couple of crucial functions for dealing with threads.

pthread_create will create a background thread. It takes a destination to store a thread handle in, some thread creation attributes (here not passing any, so it's just NULL), the callback to be executed in the new thread (here thread_callback), and an optional argument pointer to pass to that callback in case you want to share some data from the main thread—in this example we're sharing a pointer to a variable arg.

pthread_join can be called later at any time to wait for the thread to finish the execution, and get the result returned from the callback. It accepts the earlier assigned thread handle as well as a pointer to store the result. In this case, there aren't any results so the function takes a NULL as an argument.

To compile code using threads with Emscripten, you need to invoke emcc and pass a -pthread parameter, as when compiling the same code with Clang or GCC on other platforms:

emcc -pthread example.c -o example.js

However, when you try to run it in a browser or Node.js, you'll see a warning and then the program will hang:

Before the thread
Tried to spawn a new thread, but the thread pool is exhausted.
This might result in a deadlock unless some threads eventually exit or the code
explicitly breaks out to the event loop.
If you want to increase the pool size, use setting `-s PTHREAD_POOL_SIZE=...`.
If you want to throw an explicit error instead of the risk of deadlocking in those
cases, use setting `-s PTHREAD_POOL_SIZE_STRICT=2`.
[…hangs here…]

What happened? The problem is, most of the time-consuming APIs on the web are asynchronous and rely on the event loop to execute. This limitation is an important distinction compared to traditional environments, where applications normally run I/O in synchronous, blocking manner. Check out the blog post about Using asynchronous web APIs from WebAssembly if you'd like to learn more.

In this case, the code synchronously invokes pthread_create to create a background thread, and follows up by another synchronous call to pthread_join that waits for the background thread to finish execution. However, Web Workers, that are used behind the scenes when this code is compiled with Emscripten, are asynchronous. So what happens is, pthread_create only schedules a new Worker thread to be created on the next event loop run, but then pthread_join immediately blocks the event loop to wait for that Worker, and by doing so prevents it from ever being created. It's a classic example of a deadlock.

One way to solve this problem is to create a pool of Workers ahead of time, before the program has even started. When pthread_create is invoked, it can take a ready-to-use Worker from the pool, run the provided callback on its background thread, and return the Worker back to the pool. All of this can be done synchronously, so there won't be any deadlocks as long as the pool is sufficiently large.

This is exactly what Emscripten allows with the -s PTHREAD_POOL_SIZE=... option. It allows to specify a number of threads—either a fixed number, or a JavaScript expression like navigator.hardwareConcurrency to create as many threads as there are cores on the CPU. The latter option is helpful when your code can scale to an arbitrary number of threads.

In the example above, there is only one thread being created, so instead of reserving all cores it's sufficient to use -s PTHREAD_POOL_SIZE=1:

emcc -pthread -s PTHREAD_POOL_SIZE=1 example.c -o example.js

This time, when you execute it, things work successfully:

Before the thread
Inside the thread: 42
After the thread
Pthread 0x701510 exited.

There is another problem though: see that sleep(1) in the code example? It executes in the thread callback, meaning off the main thread, so it should be fine, right? Well, it isn't.

When pthread_join is called, it has to wait for the thread execution to finish, meaning that if the created thread is performing long-running tasks—in this case, sleeping 1 second—then the main thread will also have to block for the same amount of time till the results are back. When this JS is executed in the browser, it will block the UI thread for 1 second until the thread callback returns. This leads to poor user experience.

There are few solutions to this:

pthread_detach
-s PROXY_TO_PTHREAD
Custom Worker and Comlink

pthread_detach #

First, if you only need to run some tasks off the main thread, but don't need to wait for the results, you can use pthread_detach instead of pthread_join. This will leave the thread callback running in the background. If you're using this option, you can switch off the warning with -s PTHREAD_POOL_SIZE_STRICT=0.

PROXY_TO_PTHREAD #

Second, if you're compiling a C application rather than a library, you can use -s PROXY_TO_PTHREAD option, which will offload the main application code to a separate thread in addition to any nested threads created by the application itself. This way, main code can block safely at any time without freezing the UI. Incidentally, when using this option, you don't have to precreate the thread pool either—instead, Emscripten can leverage the main thread for creating new underlying Workers, and then block the helper thread in pthread_join without deadlocking.

Comlink #

Third, if you're working on a library and still need to block, you can create your own Worker, import the Emscripten-generated code and expose it with Comlink to the main thread. Main thread will be able to invoke any exported methods as asynchronous functions, and that way will also avoid blocking the UI.

In a simple application such as the previous example -s PROXY_TO_PTHREAD is the best option:

emcc -pthread -s PROXY_TO_PTHREAD example.c -o example.js

C++ #

All the same caveats and logic apply in the same way to C++. The only new thing you gain is access to higher-level APIs like std::thread and std::async, which use the previously discussed pthread library under the hood.

So the example above can be rewritten in more idiomatic C++ like this:

example.cpp:

#include <iostream>
#include <thread>
#include <chrono>

int main()
{
    puts("Before the thread");

    int arg = 42;
    std::thread thread([&]() {
        std::this_thread::sleep_for(std::chrono::seconds(1));
        std::cout << "Inside the thread: " << arg << std::endl;
    });

    thread.join();

    std::cout << "After the thread" << std::endl;

    return 0;
}

When compiled and executed with similar parameters, it will behave in the same way as the C example:

emcc -std=c++11 -pthread -s PROXY_TO_PTHREAD example.cpp -o example.js

Output:

Before the thread
Inside the thread: 42
Pthread 0xc06190 exited.
After the thread
Proxied main thread 0xa05c18 finished with return code 0. EXIT_RUNTIME=0 set, so
keeping main thread alive for asynchronous event operations.
Pthread 0xa05c18 exited.

Rust #

Unlike Emscripten, Rust doesn't have a specialized end-to-end web target, but instead provides a generic wasm32-unknown-unknown target for generic WebAssembly output.

If Wasm is intended to be used in a web environment, any interaction with JavaScript APIs is left to external libraries and tooling like wasm-bindgen and wasm-pack. Unfortunately, this means that the standard library is not aware of Web Workers and standard APIs such as std::thread won't work when compiled to WebAssembly.

Luckily, the majority of the ecosystem depends on higher-level libraries to take care of multithreading. At that level it's much easier to abstract away all the platform differences.

In particular, Rayon is the most popular choice for data-parallelism in Rust. It allows you to take method chains on regular iterators and, usually with a single line change, convert them in a way where they'd run in parallel on all available threads instead of sequentially. For example:

pub fn sum_of_squares(numbers: &[i32]) -> i32 {
  numbers
  .iter()
  .par_iter()
  .map(|x| x * x)
  .sum()
}

With this small change, the code will split up the input data, calculate x * x and partial sums in parallel threads, and in the end add up those partial results together.

To accommodate for platforms without working std::thread, Rayon provides hooks that allow to define custom logic for spawning and exiting threads.

wasm-bindgen-rayon taps into those hooks to spawn WebAssembly threads as Web Workers. To use it, you need to add it as a dependency and follow the configuration steps described in the docs. The example above will end up looking like this:

pub use wasm_bindgen_rayon::init_thread_pool;

#[wasm_bindgen]
pub fn sum_of_squares(numbers: &[i32]) -> i32 {
  numbers
  .par_iter()
  .map(|x| x * x)
  .sum()
}

Once done, the generated JavaScript will export an extra initThreadPool function. This function will create a pool of Workers and reuse them throughout the lifetime of the program for any multithreaded operations done by Rayon.

This pool mechanism is similar to the -s PTHREAD_POOL_SIZE=... option in Emscripten explained earlier, and also needs to be initialized before the main code to avoid deadlocks:

import init, { initThreadPool, sum_of_squares } from './pkg/index.js';

// Regular wasm-bindgen initialization.
await init();

// Thread pool initialization with the given number of threads
// (pass `navigator.hardwareConcurrency` if you want to use all cores).
await initThreadPool(navigator.hardwareConcurrency);

// ...now you can invoke any exported functions as you normally would
console.log(sum_of_squares(new Int32Array([1, 2, 3]))); // 14

Note that the same caveats about blocking the main thread apply here too. Even the sum_of_squares example still needs to block the main thread to wait for the partial results from other threads.

It might be a very short wait or a long one, depending on the complexity of iterators and number of available threads, but, to be on the safe side, browser engines actively prevent blocking the main thread altogether and such code will throw an error. Instead, you should create a Worker, import the wasm-bindgen-generated code there, and expose its API with a library like Comlink to the main thread.

Check out the wasm-bindgen-rayon example for an end-to-end demo showing:

Real-world use cases #

We actively use WebAssembly threads in Squoosh.app for client-side image compression—in particular, for formats like AVIF (C++), JPEG-XL (C++), OxiPNG (Rust) and WebP v2 (C++). Thanks to the multithreading alone, we've seen consistent 1.5x-3x speed-ups (exact ratio differs per codec), and were able to push those numbers even further by combining WebAssembly threads with WebAssembly SIMD!

Google Earth is another notable service that's using WebAssembly threads for its web version.

FFMPEG.WASM is a WebAssembly version of a popular FFmpeg multimedia toolchain that uses WebAssembly threads to efficiently encode videos directly in the browser.

There are many more exciting examples using WebAssembly threads out there. Be sure to check out the demos and bring your own multithreaded applications and libraries to the web!

Using asynchronous web APIs from WebAssembly

2021-04-26T00:00:00Z

The I/O APIs on the web are asynchronous, but they're synchronous in most system languages. When compiling code to WebAssembly, you need to bridge one kind of APIs to another—and this bridge is Asyncify. In this post, you'll learn when and how to use Asyncify and how it works under the hood.

I/O in system languages #

I'll start with a simple example in C. Say, you want to read the user's name from a file, and greet them with a "Hello, (username)!" message:

#include <stdio.h>

int main() {
    FILE *stream = fopen("name.txt", "r");
    char name[20+1];
    size_t len = fread(&name, 1, 20, stream);
    name[len] = '\0';
    fclose(stream);
    printf("Hello, %s!\n", name);
    return 0;
}

While the example doesn't do much, it already demonstrates something you'll find in an application of any size: it reads some inputs from the external world, processes them internally and writes outputs back to the external world. All such interaction with the outside world happens via a few functions commonly called input-output functions, also shortened to I/O.

To read the name from C, you need at least two crucial I/O calls: fopen, to open the file, and fread to read data from it. Once you retrieve the data, you can use another I/O function printf to print the result to the console.

Those functions look quite simple at first glance and you don't have to think twice about the machinery involved to read or write data. However, depending on the environment, there can be quite a lot going on inside:

If the input file is located on a local drive, the application needs to perform a series of memory and disk accesses to locate the file, check permissions, open it for reading, and then read block by block until the requested number of bytes is retrieved. This can be pretty slow, depending on the speed of your disk and the requested size.
Or, the input file might be located on a mounted network location, in which case, the network stack will now be involved too, increasing the complexity, latency and number of potential retries for each operation.
Finally, even printf is not guaranteed to print things to the console and might be redirected to a file or a network location, in which case it would have to go via the same steps above.

Long story short, I/O can be slow and you can't predict how long a particular call will take by a quick glance at the code. While that operation is running, your whole application will appear frozen and unresponsive to the user.

This is not limited to C or C++ either. Most system languages present all the I/O in a form of synchronous APIs. For example, if you translate the example to Rust, the API might look simpler, but the same principles apply. You just make a call and synchronously wait for it to return the result, while it performs all the expensive operations and eventually returns the result in a single invocation:

fn main() {
    let s = std::fs::read_to_string("name.txt");
    println!("Hello, {}!", s);
}

But what happens when you try to compile any of those samples to WebAssembly and translate them to the web? Or, to provide a specific example, what could "file read" operation translate to? It would need to read data from some storage.

Asynchronous model of the web #

The web has a variety of different storage options you could map to, such as in-memory storage (JS objects), localStorage, IndexedDB, server-side storage, and a new File System Access API.

However, only two of those APIs—the in-memory storage and the localStorage—can be used synchronously, and both are the most limiting options in what you can store and for how long. All the other options provide only asynchronous APIs.

This is one of the core properties of executing code on the web: any time-consuming operation, which includes any I/O, has to be asynchronous.

The reason is that the web is historically single-threaded, and any user code that touches the UI has to run on the same thread as the UI. It has to compete with the other important tasks like layout, rendering and event handling for the CPU time. You wouldn't want a piece of JavaScript or WebAssembly to be able to start a "file read" operation and block everything else—the entire tab, or, in the past, the entire browser—for a range from milliseconds to a few seconds, until it's over.

Instead, code is only allowed to schedule an I/O operation together with a callback to be executed once it's finished. Such callbacks are executed as part of the browser's event loop. I won't be going into details here, but if you're interested in learning how the event loop works under the hood, check out Tasks, microtasks, queues and schedules which explains this topic in-depth.

The short version is that the browser runs all the pieces of code in sort of an infinite loop, by taking them from the queue one by one. When some event is triggered, the browser queues the corresponding handler, and on the next loop iteration it's taken out from the queue and executed. This mechanism allows simulating concurrency and running lots of parallel operations while using only a single thread.

The important thing to remember about this mechanism is that, while your custom JavaScript (or WebAssembly) code executes, the event loop is blocked and, while it is, there is no way to react to any external handlers, events, I/O, etc. The only way to get the I/O results back is to register a callback, finish executing your code, and give the control back to the browser so that it can keep processing any pending tasks. Once I/O is finished, your handler will become one of those tasks and will get executed.

For example, if you wanted to rewrite the samples above in modern JavaScript and decided to read a name from a remote URL, you would use Fetch API and async-await syntax:

async function main() {
  let response = await fetch("name.txt");
  let name = await response.text();
  console.log("Hello, %s!", name);
}

Even though it looks synchronous, under the hood each await is essentially syntax sugar for callbacks:

function main() {
  return fetch("name.txt")
    .then(response => response.text())
    .then(name => console.log("Hello, %s!", name));
}

In this de-sugared example, which is a bit clearer, a request is started and responses are subscribed to with the first callback. Once the browser receives the initial response—just the HTTP headers—it asynchronously invokes this callback. The callback starts reading the body as text using response.text(), and subscribes to the result with another callback. Finally, once fetch has retrieved all the contents, it invokes the last callback, which prints "Hello, (username)!" to the console.

Thanks to the asynchronous nature of those steps, the original function can return control to the browser as soon as the I/O has been scheduled, and leave the entire UI responsive and available for other tasks, including rendering, scrolling and so on, while the I/O is executing in background.

As a final example, even simple APIs like "sleep", which makes an application wait a specified number of seconds, are also a form of an I/O operation:

#include <stdio.h>
#include <unistd.h>
// ...
printf("A\n");
sleep(1);
printf("B\n");

Sure, you could translate it in a very straightforward manner that would block the current thread until the time expires:

console.log("A");
for (let start = Date.now(); Date.now() - start < 1000;);
console.log("B");

In fact, that's exactly what Emscripten does in its default implementation of "sleep", but that's very inefficient, will block the entire UI and won't allow any other events to be handled meanwhile. Generally, don't do that in production code.

Instead, a more idiomatic version of "sleep" in JavaScript would involve calling setTimeout(), and subscribing with a handler:

console.log("A");
setTimeout(() => {
    console.log("B");
}, 1000);

What's common to all these examples and APIs? In each case, the idiomatic code in the original systems language uses a blocking API for the I/O, whereas an equivalent example for the web uses an asynchronous API instead. When compiling to the web, you need to somehow transform between those two execution models, and WebAssembly has no built-in ability to do so just yet.

Bridging the gap with Asyncify #

This is where Asyncify comes in. Asyncify is a compile-time feature supported by Emscripten that allows pausing the entire program and asynchronously resuming it later.

Usage in C / C++ with Emscripten #

If you wanted to use Asyncify to implement an asynchronous sleep for the last example, you could do it like this:

#include <stdio.h>
#include <emscripten.h>

EM_JS(void, async_sleep, (int seconds), {
    Asyncify.handleSleep(wakeUp => {
        setTimeout(wakeUp, seconds * 1000);
    });
});
…
puts("A");
async_sleep(1);
puts("B");

EM_JS is a macro that allows defining JavaScript snippets as if they were C functions. Inside, use a function Asyncify.handleSleep() which tells Emscripten to suspend the program and provides a wakeUp() handler that should be called once the asynchronous operation has finished. In the example above, the handler is passed to setTimeout(), but it could be used in any other context that accepts callbacks. Finally, you can call async_sleep() anywhere you want just like regular sleep() or any other synchronous API.

When compiling such code, you need to tell Emscripten to activate the Asyncify feature. Do that by passing -s ASYNCIFY as well as -s ASYNCIFY_IMPORTS=[func1, func2] with an array-like list of functions that might be asynchronous.

emcc -O2 \
    -s ASYNCIFY \
    -s ASYNCIFY_IMPORTS=[async_sleep] \
    ...

This lets Emscripten know that any calls to those functions might require saving and restoring the state, so the compiler will inject supporting code around such calls.

Now, when you execute this code in the browser you'll see a seamless output log like you'd expect, with B coming after a short delay after A.

A
B

You can return values from Asyncify functions too. What you need to do is return the result of handleSleep(), and pass the result to the wakeUp() callback. For example, if, instead of reading from a file, you want to fetch a number from a remote resource, you can use a snippet like the one below to issue a request, suspend the C code, and resume once the response body is retrieved—all done seamlessly as if the call were synchronous.

EM_JS(int, get_answer, (), {
     return Asyncify.handleSleep(wakeUp => {
        fetch("answer.txt")
            .then(response => response.text())
            .then(text => wakeUp(Number(text)));
    });
});
puts("Getting answer...");
int answer = get_answer();
printf("Answer is %d\n", answer);

In fact, for Promise-based APIs like fetch(), you can even combine Asyncify with JavaScript's async-await feature instead of using the callback-based API. For that, instead of Asyncify.handleSleep(), call Asyncify.handleAsync(). Then, instead of having to schedule a wakeUp() callback, you can pass an async JavaScript function and use await and return inside, making code look even more natural and synchronous, while not losing any of the benefits of the asynchronous I/O.

EM_JS(int, get_answer, (), {
     return Asyncify.handleAsync(async () => {
        let response = await fetch("answer.txt");
        let text = await response.text();
        return Number(text);
    });
});

int answer = get_answer();

Awaiting complex values #

But this example still limits you only to numbers. What if you want to implement the original example, where I tried to get a user's name from a file as a string? Well, you can do that too!

Emscripten provides a feature called Embind that allows you to handle conversions between JavaScript and C++ values. It has support for Asyncify as well, so you can call await() on external Promises and it will act just like await in async-await JavaScript code:

val fetch = val::global("fetch");
val response = fetch(std::string("answer.txt")).await();
val text = response.call<val>("text").await();
auto answer = text.as<std::string>();

When using this method, you don't even need to pass ASYNCIFY_IMPORTS as a compile flag, as it's already included by default.

Okay, so this all works great in Emscripten. What about other toolchains and languages?

Usage from other languages #

Say that you have a similar synchronous call somewhere in your Rust code that you want to map to an async API on the web. Turns out, you can do that too!

First, you need to define such a function as a regular import via extern block (or your chosen language's syntax for foreign functions).

extern {
    fn get_answer() -> i32;
}

println!("Getting answer...");
let answer = get_answer();
println!("Answer is {}", answer);

And compile your code to WebAssembly:

cargo build --target wasm32-unknown-unknown

Now you need to instrument the WebAssembly file with code for storing/restoring the stack. For C / C++, Emscripten would do this for us, but it's not used here, so the process is a bit more manual.

Luckily, the Asyncify transform itself is completely toolchain-agnostic. It can transform arbitrary WebAssembly files, no matter which compiler it's produced by. The transform is provided separately as part of the wasm-opt optimiser from the Binaryen toolchain and can be invoked like this:

wasm-opt -O2 --asyncify \
      --pass-arg=asyncify-imports@env.get_answer \
      [...]

Pass --asyncify to enable the transform, and then use --pass-arg=… to provide a comma-separated list of asynchronous functions, where the program state should be suspended and later resumed.

All that's left is to provide supporting runtime code that will actually do that—suspend and resume WebAssembly code. Again, in the C / C++ case this would be included by Emscripten, but now you need custom JavaScript glue code that would handle arbitrary WebAssembly files. We've created a library just for that.

You can find it on GitHub at https://github.com/GoogleChromeLabs/asyncify or npm under the name asyncify-wasm.

It simulates a standard WebAssembly instantiation API, but under its own namespace. The only difference is that, under a regular WebAssembly API you can only provide synchronous functions as imports, while under the Asyncify wrapper, you can provide asynchronous imports as well:

const { instance } = await Asyncify.instantiateStreaming(fetch('app.wasm'), {
    env: {
        async get_answer() {
            let response = await fetch("answer.txt");
            let text = await response.text();
            return Number(text);
        }
    }
});
…
await instance.exports.main();

Once you try to call such an asynchronous function - like get_answer() in the example above - from the WebAssembly side, the library will detect the returned Promise, suspend and save the state of the WebAssembly application, subscribe to the promise completion, and later, once it's resolved, seamlessly restore the call stack and state and continue execution as if nothing has happened.

Since any function in the module might make an asynchronous call, all the exports become potentially asynchronous too, so they get wrapped as well. You might have noticed in the example above that you need to await the result of instance.exports.main() to know when the execution is truly finished.

How does this all work under the hood? #

When Asyncify detects a call to one of the ASYNCIFY_IMPORTS functions, it starts an asynchronous operation, saves the entire state of the application, including the call stack and any temporary locals, and later, when that operation is finished, restores all the memory and call stack and resumes from the same place and with the same state as if the program has never stopped.

This is quite similar to async-await feature in JavaScript that I showed earlier, but, unlike the JavaScript one, doesn't require any special syntax or runtime support from the language, and instead works by transforming plain synchronous functions at compile-time.

When compiling the earlier shown asynchronous sleep example:

puts("A");
async_sleep(1);
puts("B");

Asyncify takes this code and transforms it to roughly like the following one (pseudo-code, real transformation is more involved than this):

if (mode == NORMAL_EXECUTION) {
    puts("A");
    async_sleep(1);
    saveLocals();
    mode = UNWINDING;
    return;
}
if (mode == REWINDING) {
    restoreLocals();
    mode = NORMAL_EXECUTION;
}
puts("B");

Initially mode is set to NORMAL_EXECUTION. Correspondingly, the first time such transformed code is executed, only the part leading up to async_sleep() will get evaluated. As soon as the asynchronous operation is scheduled, Asyncify saves all the locals, and unwinds the stack by returning from each function all the way to the top, this way giving control back to the browser event loop.

Then, once async_sleep() resolves, Asyncify support code will change mode to REWINDING, and call the function again. This time, the "normal execution" branch is skipped - since it already did the job last time and I want to avoid printing "A" twice - and instead it comes straight to the "rewinding" branch. Once it's reached, it restores all the stored locals, changes mode back to "normal" and continues the execution as if the code were never stopped in the first place.

Transformation costs #

Unfortunately, Asyncify transform isn't completely free, since it has to inject quite a bit of supporting code for storing and restoring all those locals, navigating the call stack under different modes and so on. It tries to modify only functions marked as asynchronous on the command line, as well as any of their potential callers, but the code size overhead might still add up to approximately 50% before compression.

This isn't ideal, but in many cases acceptable when the alternative is not having the functionality altogether or having to make significant rewrites to the original code.

Make sure to always enable optimizations for the final builds to avoid it going even higher. You can also check Asyncify-specific optimization options to reduce the overhead by limiting transforms only to specified functions and/or only direct function calls. There is also a minor cost to runtime performance, but it's limited to the async calls themselves. However, compared to the cost of the actual work, it's usually negligible.

Real-world demos #

Now that you've looked at the simple examples, I'll move on to more complicated scenarios.

As mentioned in the beginning of the article, one of the storage options on the web is an asynchronous File System Access API. It provides access to a real host filesystem from a web application.

On the other hand, there is a de-facto standard called WASI for WebAssembly I/O in the console and the server-side. It was designed as a compilation target for system languages, and exposes all sorts of file system and other operations in a traditional synchronous form.

What if you could map one to another? Then you could compile any application in any source language with any toolchain supporting the WASI target, and run it in a sandbox on the web, while still allowing it to operate on real user files! With Asyncify, you can do just that.

In this demo, I've compiled Rust coreutils crate with a few minor patches to WASI, passed via Asyncify transform and implemented asynchronous bindings from WASI to File System Access API on the JavaScript side. Once combined with Xterm.js terminal component, this provides a realistic shell running in the browser tab and operating on real user files - just like an actual terminal.

Check it out live at https://wasi.rreverser.com/.

Asyncify use-cases are not limited just to timers and filesystems, either. You can go further and use more niche APIs on the web.

For example, also with the help of Asyncify, it's possible to map libusb—probably the most popular native library for working with USB devices—to a WebUSB API, which gives asynchronous access to such devices on the web. Once mapped and compiled, I got standard libusb tests and examples to run against chosen devices right in the sandbox of a web page.

It's probably a story for another blog post though.

Those examples demonstrate just how powerful Asyncify can be for bridging the gap and porting all sorts of applications to the web, allowing you to gain cross-platform access, sandboxing, and better security, all without losing functionality.

Debugging memory leaks in WebAssembly using Emscripten

2020-08-13T00:00:00Z

Squoosh.app is a PWA that illustrates just how much different image codecs and settings can improve image file size without significantly affecting quality. However, it's also a technical demo showcasing how you can take libraries written in C++ or Rust and bring them to the web.

Being able to port code from existing ecosystems is incredibly valuable, but there are some key differences between those static languages and JavaScript. One of those is in their different approaches to memory management.

While JavaScript is fairly forgiving in cleaning up after itself, such static languages are definitely not. You need to explicitly ask for a new allocated memory and you really need to make sure you give it back afterwards, and never use it again. If that doesn't happen, you get leaks… and it actually happens fairly regularly. Let's take a look at how you can debug those memory leaks and, even better, how you can design your code to avoid them next time.

Suspicious pattern #

Recently, while starting to work on Squoosh, I couldn't help but notice an interesting pattern in C++ codec wrappers. Let's take a look at an ImageQuant wrapper as an example (reduced to show only object creation and deallocation parts):

liq_attr* attr;
liq_image* image;
liq_result* res;
uint8_t* result;

RawImage quantize(std::string rawimage,
                  int image_width,
                  int image_height,
                  int num_colors,
                  float dithering) {
  const uint8_t* image_buffer = (uint8_t*)rawimage.c_str();
  int size = image_width * image_height;

  attr = liq_attr_create();
  image = liq_image_create_rgba(attr, image_buffer, image_width, image_height, 0);
  liq_set_max_colors(attr, num_colors);
  liq_image_quantize(image, attr, &res);
  liq_set_dithering_level(res, dithering);
  uint8_t* image8bit = (uint8_t*)malloc(size);
  result = (uint8_t*)malloc(size * 4);

  // …

  free(image8bit);
  liq_result_destroy(res);
  liq_image_destroy(image);
  liq_attr_destroy(attr);

  return {
    val(typed_memory_view(image_width * image_height * 4, result)),
    image_width,
    image_height
  };
}

void free_result() {
  free(result);
}

JavaScript (well, TypeScript):

export async function process(data: ImageData, opts: QuantizeOptions) {
  if (!emscriptenModule) {
    emscriptenModule = initEmscriptenModule(imagequant, wasmUrl);
  }
  const module = await emscriptenModule;

  const result = module.quantize(/* … */);

  module.free_result();

  return new ImageData(
    new Uint8ClampedArray(result.view),
    result.width,
    result.height
  );
}

Do you spot an issue? Hint: it's use-after-free, but in JavaScript!

In Emscripten, typed_memory_view returns a JavaScript Uint8Array backed by the WebAssembly (Wasm) memory buffer, with byteOffset and byteLength set to the given pointer and length. The main point is that this is a TypedArray view into a WebAssembly memory buffer, rather than a JavaScript-owned copy of the data.

When we call free_result from JavaScript, it, in turn, calls a standard C function free to mark this memory as available for any future allocations, which means the data that our Uint8Array view points to, can be overwritten with arbitrary data by any future call into Wasm.

Or, some implementation of free might even decide to zero-fill the freed memory immediately. The free that Emscripten uses doesn't do that, but we are relying on an implementation detail here that cannot be guaranteed.

Or, even if the memory behind the pointer gets preserved, new allocation might need to grow the WebAssembly memory. When WebAssembly.Memory is grown either via JavaScript API, or corresponding memory.grow instruction, it invalidates the existing ArrayBuffer and, transitively, any views backed by it.

Let me use the DevTools (or Node.js) console to demonstrate this behavior:

> memory = new WebAssembly.Memory({ initial: 1 })
Memory {}

> view = new Uint8Array(memory.buffer, 42, 10)
Uint8Array(10) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
// ^ all good, we got a 10 bytes long view at address 42

> view.buffer
ArrayBuffer(65536) {}
// ^ its buffer is the same as the one used for WebAssembly memory
//   (the size of the buffer is 1 WebAssembly "page" == 64KB)

> memory.grow(1)
1
// ^ let's say we grow Wasm memory by +1 page to fit some new data

> view
Uint8Array []
// ^ our original view is no longer valid and looks empty!

> view.buffer
ArrayBuffer(0) {}
// ^ its buffer got invalidated as well and turned into an empty one

Finally, even if we don't explicitly call into Wasm again between free_result and new Uint8ClampedArray, at some point we might add multithreading support to our codecs. In that case it could be a completely different thread that overwrites the data just before we manage to clone it.

Looking for memory bugs #

Just in case, I've decided to go further and check if this code exhibits any issues in practice. This seems like a perfect opportunity to try out the new(ish) Emscripten sanitizers support that was added last year and presented in our WebAssembly talk at the Chrome Dev Summit:

In this case, we're interested in the AddressSanitizer, which can detect various pointer- and memory-related issues. To use it, we need to recompile our codec with -fsanitize=address:

emcc \
  --bind \
  ${OPTIMIZE} \
  --closure 1 \
  -s ALLOW_MEMORY_GROWTH=1 \
  -s MODULARIZE=1 \
  -s 'EXPORT_NAME="imagequant"' \
  -I node_modules/libimagequant \
  -o ./imagequant.js \
  --std=c++11 \
  imagequant.cpp \
  -fsanitize=address \
  node_modules/libimagequant/libimagequant.a

This will automatically enable pointer safety checks, but we also want to find potential memory leaks. Since we're using ImageQuant as a library rather than a program, there is no "exit point" at which Emscripten could automatically validate that all memory has been freed.

Instead, for such cases the LeakSanitizer (included in the AddressSanitizer) provides the functions __lsan_do_leak_check and __lsan_do_recoverable_leak_check, which can be manually invoked whenever we expect all memory to be freed and want to validate that assumption. __lsan_do_leak_check is meant to be used at the end of a running application, when you want to abort the process in case any leaks are detected, while __lsan_do_recoverable_leak_check is more suitable for library use-cases like ours, when you want to print leaks to the console, but keep the application running regardless.

Let's expose that second helper via Embind so that we can call it from JavaScript at any time:

#include <sanitizer/lsan_interface.h>

// …

void free_result() {
  free(result);
}

EMSCRIPTEN_BINDINGS(my_module) {
  function("zx_quantize", &zx_quantize);
  function("version", &version);
  function("free_result", &free_result);
  function("doLeakCheck", &__lsan_do_recoverable_leak_check);
}

And invoke it from the JavaScript side once we're done with the image. Doing this from the JavaScript side, rather than the C++ one, helps to ensure that all the scopes have been exited and all the temporary C++ objects were freed by the time we run those checks:

  // …

  const result = opts.zx
    ? module.zx_quantize(data.data, data.width, data.height, opts.dither)
    : module.quantize(data.data, data.width, data.height, opts.maxNumColors, opts.dither);

  module.free_result();
  module.doLeakCheck();

  return new ImageData(
    new Uint8ClampedArray(result.view),
    result.width,
    result.height
  );
}

This gives us a report like the following in the console:

Uh-oh, there are some small leaks, but the stacktrace is not very helpful as all the function names are mangled. Let's recompile with a basic debugging info to preserve them:

emcc \
  --bind \
  ${OPTIMIZE} \
  --closure 1 \
  -s ALLOW_MEMORY_GROWTH=1 \
  -s MODULARIZE=1 \
  -s 'EXPORT_NAME="imagequant"' \
  -I node_modules/libimagequant \
  -o ./imagequant.js \
  --std=c++11 \
  imagequant.cpp \
  -fsanitize=address \
  -g2 \
  node_modules/libimagequant/libimagequant.a

This looks much better:

Some parts of the stacktrace still look obscure as they point to Emscripten internals, but we can tell that the leak is coming from a RawImage conversion to "wire type" (to a JavaScript value) by Embind. Indeed, when we look at the code, we can see that we return RawImage C++ instances to JavaScript, but we never free them on either side.

As a reminder, currently there is no garbage collection integration between JavaScript and WebAssembly, although one is being developed. Instead, you have to manually free any memory and call destructors from the JavaScript side once you're done with the object. For Embind specifically, the official docs suggest to call a .delete() method on exposed C++ classes:

JavaScript code must explicitly delete any C++ object handles it has received, or the Emscripten heap will grow indefinitely.
var x = new Module.MyClass;
x.method();
x.delete();

Indeed, when we do that in JavaScript for our class:

  // …

  const result = opts.zx
    ? module.zx_quantize(data.data, data.width, data.height, opts.dither)
    : module.quantize(data.data, data.width, data.height, opts.maxNumColors, opts.dither);

  module.free_result();
  result.delete();
  module.doLeakCheck();

  return new ImageData(
    new Uint8ClampedArray(result.view),
    result.width,
    result.height
  );
}

The leak goes away as expected.

Discovering more issues with sanitizers #

Building other Squoosh codecs with sanitizers reveals both similar as well as some new issues. For example, I've got this error in MozJPEG bindings:

Here, it's not a leak, but us writing to a memory outside of the allocated boundaries 😱

Digging into the code of MozJPEG, we find that the problem here is that jpeg_mem_dest—the function that we use to allocate a memory destination for JPEG—reuses existing values of outbuffer and outsize when they're non-zero:

if (*outbuffer == NULL || *outsize == 0) {
  /* Allocate initial buffer */
  dest->newbuffer = *outbuffer = (unsigned char *) malloc(OUTPUT_BUF_SIZE);
  if (dest->newbuffer == NULL)
    ERREXIT1(cinfo, JERR_OUT_OF_MEMORY, 10);
  *outsize = OUTPUT_BUF_SIZE;
}

However, we invoke it without initialising either of those variables, which means MozJPEG writes the result into a potentially random memory address that happened to be stored in those variables at the time of the call!

uint8_t* output;
unsigned long size;
// …
jpeg_mem_dest(&cinfo, &output, &size);

Zero-initialising both variables before the invocation solves this issue, and now the code reaches a memory leak check instead. Luckily, the check passes successfully, indicating that we don't have any leaks in this codec.

Issues with shared state #

…Or do we?

We know that our codec bindings store some of the state as well as results in global static variables, and MozJPEG has some particularly complicated structures.

uint8_t* last_result;
struct jpeg_compress_struct cinfo;

val encode(std::string image_in, int image_width, int image_height, MozJpegOptions opts) {
  // …
}

What if some of those get lazily initialized on the first run, and then improperly reused on future runs? Then a single call with a sanitizer would not report them as problematic.

Let's try and process the image a couple of times by randomly clicking at different quality levels in the UI. Indeed, now we get the following report:

262,144 bytes—looks like the whole sample image is leaked from jpeg_finish_compress!

After checking out the docs and the official examples, it turns out that jpeg_finish_compress doesn't free the memory allocated by our earlier jpeg_mem_dest call—it only frees the compression structure, even though that compression structure already knows about our memory destination… Sigh.

We can fix this by freeing the data manually in the free_result function:

void free_result() {
  /* This is an important step since it will release a good deal of memory. */
  free(last_result);
  jpeg_destroy_compress(&cinfo);
}

I could keep hunting those memory bugs one by one, but I think by now it's clear enough that the current approach to memory management leads to some nasty systematic issues.

Some of them can be caught by the sanitizer right away. Others require intricate tricks to be caught. Finally, there are issues like in the beginning of the post that, as we can see from the logs, aren't caught by the sanitizer at all. The reason is that the actual mis-use happens on the JavaScript side, into which the sanitizer has no visibility. Those issues will reveal themselves only in production or after seemingly unrelated changes to the code in the future.

Building a safe wrapper #

Let's take a couple of steps back, and instead fix all of these problems by restructuring the code in a safer way. I'll use ImageQuant wrapper as an example again, but similar refactoring rules apply to all the codecs, as well as other similar codebases.

First of all, let's fix the use-after-free issue from the beginning of the post. For that, we need to clone the data from the WebAssembly-backed view before marking it as free on the JavaScript side:

  // …

  const result = /* … */;

  const imgData = new ImageData(
    new Uint8ClampedArray(result.view),
    result.width,
    result.height
  );

  module.free_result();
  result.delete();
  module.doLeakCheck();

  return new ImageData(
    new Uint8ClampedArray(result.view),
    result.width,
    result.height
  );
  return imgData;
}

Now, let's make sure that we don't share any state in global variables between invocations. This will both fix some of the issues we've already seen, as well as will make it easier to use our codecs in a multithreaded environment in the future.

To do that, we refactor the C++ wrapper to make sure that each call to the function manages its own data using local variables. Then, we can change the signature of our free_result function to accept the pointer back:

liq_attr* attr;
liq_image* image;
liq_result* res;
uint8_t* result;

RawImage quantize(std::string rawimage,
                  int image_width,
                  int image_height,
                  int num_colors,
                  float dithering) {
  const uint8_t* image_buffer = (uint8_t*)rawimage.c_str();
  int size = image_width * image_height;

  attr = liq_attr_create();
  image = liq_image_create_rgba(attr, image_buffer, image_width, image_height, 0);
  liq_attr* attr = liq_attr_create();
  liq_image* image = liq_image_create_rgba(attr, image_buffer, image_width, image_height, 0);
  liq_set_max_colors(attr, num_colors);
  liq_result* res = nullptr;
  liq_image_quantize(image, attr, &res);
  liq_set_dithering_level(res, dithering);
  uint8_t* image8bit = (uint8_t*)malloc(size);
  result = (uint8_t*)malloc(size * 4);
  uint8_t* result = (uint8_t*)malloc(size * 4);

  // …
}

void free_result() {
void free_result(uint8_t *result) {
  free(result);
}

But, since we're already using Embind in Emscripten to interact with JavaScript, we might as well make the API even safer by hiding C++ memory management details altogether!

For that, let's move the new Uint8ClampedArray(…) part from JavaScript to the C++ side with Embind. Then, we can use it to clone the data into the JavaScript memory even before returning from the function:

class RawImage {
 public:
  val buffer;
  int width;
  int height;

  RawImage(val b, int w, int h) : buffer(b), width(w), height(h) {}
};
thread_local const val Uint8ClampedArray = val::global("Uint8ClampedArray");

RawImage quantize(/* … */) {
val quantize(/* … */) {
  // …
  return {
    val(typed_memory_view(image_width * image_height * 4, result)),
    image_width,
    image_height
  };
  val js_result = Uint8ClampedArray.new_(typed_memory_view(
    image_width * image_height * 4,
    result
  ));
  free(result);
  return js_result;
}

Note how, with a single change, we both ensure that the resulting byte array is owned by JavaScript and not backed by WebAssembly memory, and get rid of the previously leaked RawImage wrapper too.

Now JavaScript doesn't have to worry about freeing data at all anymore, and can use the result like any other garbage-collected object:

  // …

  const result = /* … */;

  const imgData = new ImageData(
    new Uint8ClampedArray(result.view),
    result.width,
    result.height
  );

  module.free_result();
  result.delete();
  // module.doLeakCheck();

  return imgData;
  return new ImageData(result, result.width, result.height);
}

This also means we no longer need a custom free_result binding on the C++ side:

void free_result(uint8_t* result) {
  free(result);
}

EMSCRIPTEN_BINDINGS(my_module) {
  class_<RawImage>("RawImage")
      .property("buffer", &RawImage::buffer)
      .property("width", &RawImage::width)
      .property("height", &RawImage::height);

  function("quantize", &quantize);
  function("zx_quantize", &zx_quantize);
  function("version", &version);
  function("free_result", &free_result, allow_raw_pointers());
}

All in all, our wrapper code became both cleaner and safer at the same time.

After this I went through some further minor improvements to the code of the ImageQuant wrapper and replicated similar memory management fixes for other codecs. If you're interested in more details, you can see the resulting PR here: Memory fixes for C++ codecs.

Takeaways #

What lessons can we learn and share from this refactoring that could be applied to other codebases?

Don't use memory views backed by WebAssembly—no matter which language it's built from—beyond a single invocation. You can't rely on them surviving any longer than that, and you won't be able to catch these bugs by conventional means, so if you need to store the data for later, copy it to the JavaScript side and store it there.
If possible, use a safe memory management language or, at least, safe type wrappers, instead of operating on raw pointers directly. This won't save you from bugs on the JavaScript ↔ WebAssembly boundary, but at least it will reduce the surface for bugs self-contained by the static language code.
No matter which language you use, run code with sanitizers during development—they can help to catch not only problems in the static language code, but also some issues across the JavaScript ↔ WebAssembly boundary, such as forgetting to call .delete() or passing in invalid pointers from the JavaScript side.
If possible, avoid exposing unmanaged data and objects from WebAssembly to JavaScript altogether. JavaScript is a garbage-collected language, and manual memory management is not common in it. This can be considered an abstraction leak of the memory model of the language your WebAssembly was built from, and incorrect management is easy to overlook in a JavaScript codebase.
This might be obvious, but, like in any other codebase, avoid storing mutable state in global variables. You don't want to debug issues with its reuse across various invocations or even threads, so it's best to keep it as self-contained as possible.

Ingvar Stepanyan on web.dev

Drawing to canvas in Emscripten

Canvas via Embind #

Choosing the canvas element #

OpenGL and SDL2 #

Drawing a rectangle #

Integrating an event loop #

Unblocking event loop with Asyncify #

Unblocking event loop with "main loop" APIs #

Handling user interactions #

Drawing other shapes with SDL2_gfx #

Porting USB applications to the web. Part 2: gPhoto2

Pointing build systems to custom forks #

Dealing with dynamic loading #

Generating the settings UI #

Building a live "video" feed #

Synchronizing the USB access #

Conclusion #

WebAssembly feature detection

Picking and grouping features #

Compiling for different feature sets #

Loading the correct bundle #

Final words #

Porting USB applications to the web. Part 1: libusb

First things first: a demo #

Note on camera-specific quirks #

Important cross-platform compatibility notes #

Adding a new backend to libusb #

Storing WebUSB objects as device handles #

Async web APIs in synchronous C contexts #

Device enumeration #

Porting event loops to the web #

Build system and the first test #

Using the port #

Conclusion #

Embedding JavaScript snippets in C++ with Emscripten

emscripten::val class #

EM_JS macro #

EM_ASYNC_JS macro #

EM_ASM #

--js-library #

Conclusion #

Bundling non-JavaScript resources

Custom imports in bundlers #

Universal pattern for browsers and bundlers #

Ambiguous relative URLs #

Tooling support #

Bundlers #

WebAssembly #

C/C++ via Emscripten #

Rust via wasm-pack / wasm-bindgen #

Future features #

import.meta.resolve #

Import assertions #

Conclusion #

Using WebAssembly threads from C, C++ and Rust

How WebAssembly threads work #

Web Workers #

SharedArrayBuffer #

WebAssembly atomics #

How to use WebAssembly threads #

Feature detection #

C #

pthread_detach #

PROXY_TO_PTHREAD #

Comlink #

C++ #

Rust #

Real-world use cases #

Using asynchronous web APIs from WebAssembly

I/O in system languages #

Asynchronous model of the web #

Bridging the gap with Asyncify #

Usage in C / C++ with Emscripten #

Awaiting complex values #

Usage from other languages #

How does this all work under the hood? #

Transformation costs #

Real-world demos #

Debugging memory leaks in WebAssembly using Emscripten

`SharedArrayBuffer` #