Daniel Isaksson on web.dev

The Hobbit Experience 2014

2014-11-18T00:00:00Z

In time for the new Hobbit movie “The Hobbit: The Battle of the Five Armies” we have worked on extending last year’s Chrome Experiment, A Journey through Middle-earth with some new content. The main focus this time has been to widen the use of WebGL as more browsers and devices can view the content and to work with the WebRTC capabilities in Chrome and Firefox. We had three goals with this years experiment:

P2P gameplay using WebRTC and WebGL on Chrome for Android
Make a multi-player game that is easy to play and that is based on touch input
Host on Google Cloud Platform

Defining the game #

The game logic is built on a grid-based setup with troops moving on a game board. This made it easy for us to try out the gameplay on paper as we were defining the rules. Using a grid-based setup also helps with collision detection in the game to keep a good performance since you only have to check for collisions with objects in the same or neighbouring tiles. We knew from the beginning that we wanted to focus the new game around a battle between the four main forces of Middle-earth, Humans, Dwarves, Elves and Orcs. It also had to be casual enough to be played within a Chrome Experiment and not have too many interactions to learn. We started by defining five Battlegrounds on the Middle-earth map that serve as game-rooms where multiple players can compete in a peer-to-peer battle. Showing multiple players in the room on a mobile screen, and allowing users to select who to challenge was a challenge in itself. To make the interaction and the scene easier we decided to only have one button to challenge and accept and only use the room to show events and who is the current king of the hill. This direction also resolved a few issues on the match-making side and allowed us to match the best candidates for a battle. In our previous Chrome experiment Cube Slam we learned that it takes a lot of work to handle latency in a multi-player game if the game’s result is relying on it. You constantly have to make assumptions of where the opponent’s state will be, where the opponent thinks that you are and sync that with animations on different devices. This article explains these challenges in more detail. To make it a bit easier we made this game turn-based.

Parts of the game #

To make this multi-player game there are a few key parts that we had to build:

A server side player management API handles users, match-making, sessions and game statistics.
Servers to help establishing the connection between the players.
An API for handling the AppEngine Channels API signaling used to connect and communicate with all the players in the game rooms.
A JavaScript Game engine that handles the syncing of the state and the RTC messaging between the two players/peers.
The WebGL game view.

Player management #

To support a large number of players we use many parallel game-rooms per Battleground. The main reason to limiting the number of players per game-room is to allow new players to reach the top of the leaderboard in reasonable time. The limit is also connected to the size of the json object describing the game-room sent through the Channel API that has a limit of 32kb. We have to store players, rooms, scores, sessions and their relationships in the game. To do this, first we used NDB for entities and used the query interface to deal with relationships. NDB is an interface to the Google Cloud Datastore. Using NDB worked great in the beginning but we soon ran into a problem with how we needed to use it. The query was run against the "committed" version of the database (NDB Writes is explained at great length in this in-depth article) which can have a delay of several seconds. But the entities themselves didn't have that delay as they respond directly from the cache. It might be a bit easier to explain with some example code:

// example code to explain our issue with eventual consistency
def join_room(player_id, room_id):
    room = Room.get_by_id(room_id)
    
    player = Player.get_by_id(player_id)
    player.room = room.key
    player.put()
    
    // the player Entity is updated directly in the cache
    // so calling this will return the room key as expected
    player.room // = Key(Room, room_id)

    // Fetch all the players with room set to 'room.key'
    players_in_room = Player.query(Player.room == room.key).fetch()
    // = [] (an empty list of players)
    // even though the saved player above may be expected to be in the
    // list it may not be there because the query api is being run against the 
    // "committed" version and may still be empty for a few seconds

    return {
        room: room,
        players: players_in_room,
    }

After adding unit tests we could see the issue clearly and we moved away from the queries to instead keep the relationships in a comma separated list in memcache. This felt like a bit of a hack but it worked and the AppEngine memcache has a transaction-like system for the keys using the excellent “compare and set”-feature so now the tests passed again.

Unfortunately memcache is not all rainbows and unicorns but comes with a few limits, the most notable ones being the 1MB value size (can’t have too many rooms related to a battleground) and key expiration, or as the docs explains it:

We did consider using another great key-value store, Redis. But at the time setting up a scalable cluster was a bit daunting and since we’d rather focus on building the experience than maintaining servers we didn’t go down that path. On the other hand the Google Cloud Platform recently released a simple Click-to-deploy feature, with one of the options being a Redis Cluster so that would have been a very interesting option.

Finally we found Google Cloud SQL and moved the relationships into MySQL. It was a lot of work but eventually it worked great, the updates are now fully atomic and the tests still pass. It also made implementing the match-making and score-keeping a lot more reliable.

Over time more of the data slowly has moved over from NDB and memcache to SQL but in general the player, battleground and room entities are still stored in NDB while the sessions and relationships between them all are stored in SQL.

We also had to keep track on who was playing who and pair up players against each other using a matching mechanism that took into consideration the players skill level and experience. We based the match-making on the open-source library Glicko2.

Since this is a multi-player game we want to inform the other players in the room about events like “who entered or left”, “who won or lost” and if there is a challenge to accept. To handle this we built in the ability to receive notifications into the Player Management API.

Setting up WebRTC #

When two players are matched up for a battle a signaling service is used to get the two matched peers talking to each other and to help start a peer connection.

There are several third-party libraries you can use for the signaling service and that also simplifies setting up WebRTC. Some options are PeerJS, SimpleWebRTC, and PubNub WebRTC SDK. PubNub uses a hosted server solution and for this project we wanted to host on the Google Cloud Platform. The other two libraries use node.js servers that we could have installed on Google Compute Engine but we would also have to make sure it could handle thousands of concurrent users, something we already knew the Channel API can do.

One of the main advantages of using the Google Cloud Platform in this case is scaling. Scaling the resources needed for an AppEngine project is easily handled through the Google Developers Console and no extra work is needed to scale the signaling service when using the Channels API.

There were some concerns about latency and how robust the Channels API is but we had previously used it for the CubeSlam project and it had proven to work for millions of users in that project so we decided to use it again.

Since we didn’t choose to use a third-party library to help with WebRTC we had to build our own. Luckily we could re-use a lot of the work we did for the CubeSlam project. When both players have joined a session the session is set to “active”, and both players will then use that active session id to initiate the peer-to-peer connection through the Channel API. After that all communication between the two players will be handled over a RTCDataChannel.

We also need STUN and TURN servers to help establishing the connection and cope with NATs and firewalls. Read more in depth about setting up WebRTC in the HTML5 Rocks article WebRTC in the real world: STUN, TURN, and signaling.

The number of TURN servers used also need to be able to scale depending on the traffic. To handle this we tested the Google Deployment manager. It allows us to dynamically deploy resources on Google Compute Engine and install TURN servers using a template. It’s still in alpha but for our purposes it’s worked flawlessly. For TURN server we use coturn, which is a very fast, efficient and seemingly reliable implementation of STUN/TURN.

The Channel API #

The Channel API is used to send all communication to and from the game room on the client side. Our player Management API is using the Channel API for its notifications about game events.

Working with the Channels API had a few speedbumps. One example is that since the messages can come unordered we had to wrap all messages in an object and sort them. Here’s some example code on how it works:

var que = [];  // [seq, packet...]
var seq = 0;
var rcv = -1;

function send(message) {
  var packet = JSON.stringify({
    seq: seq++,
    msg: message
  });
  channel.send(packet);
}

function recv(packet) {
  var data = JSON.parse(packet);

  if (data.seq &lt;= rcv) {
    // ignoring message, older or already received
  } else if (data.seq > rcv + 1) {
    // message from the future. queue it up.
    que.push(data.seq, packet);
  } else {
    // message in order! update the rcv index and emit the message
    rcv = data.seq;
    emit('message', data.message);

    // and now that we have updated the `rcv` index we 
    // will check the que for any other we can send
    setTimeout(flush, 10);
  }
}

function flush() {
  for (var i=0; i&lt;que.length; i++) {
    var seq = que[i];
    var packet = que[i+1];
    if (data.seq == rcv + 1) {
      recv(packet);
      return; // wait for next flush
    }
  }
}

We also wanted to keep the different APIs of the site modular and separated from the hosting of the site and started up with using the modules built into GAE. Unfortunately after getting it all to work in dev we realized that the Channel API does not work with modules at all in production. Instead we moved to using separate GAE instances and ran into CORS problems that forced us to use an iframe postMessage bridge.

Game engine #

To make the game-engine as dynamic as possible we built the front end application using the entity-component-system (ECS) approach. When we started the development the wireframes and functional specification was not set, so it was very helpful to be able to add features and logic as the development progressed. For example, the first prototype used a simple canvas-render-system to display the entities in a grid. A couple of iterations later, a system for collisions was added, and one for AI-controlled players. In the middle of the project we could switch to a 3d-renderer-system without changing the rest of the code. When the networking parts was up and running the ai-system could be modified to use remote commands.

So the basic logic of the multiplayer is to send the configuration of the action-command to the other peer through DataChannels and let the simulation act as if it’s an AI-player. On top of that there’s logic to decide which turn it is, if the player presses pass/attack-buttons, queue commands if they come in while the player still looking at the previous animation etc.

If it was just two users switching turns, both peers could share the responsibility to pass the turn to the opponent when they were done, but there is a third player involved. The AI-system became handy again (not just for testing), when we needed to add enemies like spiders and trolls. To make them fit into the turn-based flow it had to be spawned and executed exactly the same on both sides. This was solved by letting one peer control the turn-system and send the current status to the remote peer. Then when it’s the spiders turn, the turn manager let the ai-system create a command which is sent to the remote user. Since the game-engine is just acting on commands and entity-id:s the game will be simulated just the same on both sides. All units can also have the ai-component which enables easy automated testing.

It was optimal to have a simpler canvas-renderer in the beginning of development while focusing on the game logic. But the real fun started when the 3d version was implemented and the scenes came to life with environments and animations. We use three.js as 3d-engine, and it was easy to get to a playable state because of the architecture.

The mouse position is sent more frequently to the remote user and a 3d-light subtle hints about where the cursor is at the moment.

The Front-end of Middle-earth

2013-12-11T00:00:00Z

In our first article about the development of the Chrome Experiment A Journey Through Middle-earth we focused on WebGL development for mobile devices. In this article we discuss the challenges, problems and solutions we encountered when creating the rest of the HTML5 front-end.

Three versions of the same site #

Let’s start by talking a bit about adapting this experiment to work on both desktop computers and mobile devices from a screen-size and device-capabilities perspective.

The whole project is based on a very “cinematic” style, where we design-wise wanted to keep the experience within a landscape-oriented fixed frame to keep the magic from the movie. Since a large chunk of the project consists of interactive mini “games” it wouldn’t make sense to let them overflow the frame either.

We can take the landing page as an example of how we adapt the design for different sizes.

The eagles just dropped us at the landing page.

The site has three different modes: desktop, tablet and mobile. Not just to handle layout, but because we need to handle runtime-loaded assets and add various performance optimizations. With devices that have a higher resolution than desktop computers and laptops but have worse performance than phones, it's not an easy task to define the ultimate set of rules.

We’re using user-agent data to detect mobile devices and a viewport-size test to target tablets among those (645px and higher). Each different mode can in fact render all resolutions, because the layout is based on media queries or relative/percentage positioning with JavaScript.

Since the designs in this case aren’t based on grids or rules and are quite unique between the different sections it really depends on the specific element and scenario as to what breakpoints or styles to use. It happened more than once that we had set up the perfect layout with nice sass-mixins and media-queries, and then we needed to add an effect based on the mouse position or dynamic objects, and ended up rewriting everything in JavaScript.

We also add a class with the current mode in the head tag so we can use that info in our styles, like in this example (in SCSS):

.loc-hobbit-logo {

  // Default values here.

  .desktop & {
     // Applies only in desktop mode.
  }

 .tablet &, .mobile & {
   
   // Different asset for mobile and tablets perhaps.

   @media screen and (max-height: 760px), (max-width: 760px) {
     // Breakpoint-specific styles.
   }

   @media screen and (max-height: 570px), (max-width: 400px) {
     // Breakpoint-specific styles.
   }
 }
}

We support all sizes down to about 360x320, which has been pretty challenging when making a immersive web experience. On desktop we have a minimum size before we show scrollbars because we want you to experience the site in a larger viewport if possible. On mobile devices we decided to allow both landscape and portrait mode all the way up to the interactive experiences, where we ask you to turn the device to landscape. The argument against this was that it’s not as immersive in portrait as in landscape; but the site scaled pretty well so we kept it.

It’s important to note that layout shouldn’t be mixed up with feature detection like input type, device orientation, sensors etc. Those features can exist in all of these modes and should span across all. Supporting mouse and touch at the same time is one example. Retina compensation for quality but most of all performance is another, sometimes lesser quality is better. As an example the canvas is half the resolution in the WebGL experiences on retina displays, which would otherwise have to render four times the number of pixels

We frequently used the emulator tool in DevTools during development, especially in Chrome Canary which has new improved features and lots of presets. It is a good way of quickly validating design. We still needed to test on real devices regularly. One reason was because the site is adapting to fullscreen. Pages with vertical scroll hide the browser UI when scrolling in most cases (Safari on iOS7 has problems with this currently) but we had to fit everything independent of the that. We also used a preset in the emulator and changed the screen size setting to simulate the loss of available space. Testing on real devices is also important for monitoring memory-consumption and performance

Handling the state #

After the landing page we land at the map of Middle-earth. Did you notice the URL changing? The site is a single page application that uses the History API to handle routing.

Each section of the site is its own object inheriting a boilerplate of functionality such as DOM-elements, transitions, loading of assets, disposing etc. When you explore different parts of the site, sections are initiated, elements are added to and removed from the DOM and assets for the current section are loaded.

Since the user can hit the browser’s back button or navigate via the menu at any time, everything that is created needs to be disposed of at some point. Timeouts and animations need to be stopped and discarded or they will cause unwanted behaviour, errors, and memory leaks. This is not always an easy task, especially when deadlines are approaching and you need to get everything in there as fast as possible.

Showing off the locations #

To show off the beautiful settings and the characters of Middle-earth we built a modular system of image and text components that you can drag or swipe horizontally. We haven’t enabled a scrollbar here since we want to have different speeds on different ranges, like in image sequences where you stop the motion sideways until the clip has played out.

Thranduil's Hall timeline

The timeline #

When development started we didn’t know the content of the modules for each location. What we knew was that we wanted a templated way of showing different types of media and information in a horizontal timeline that would give us the freedom to have six different location presentations without having to rebuild everything six times. To manage this we created a timeline controller that handle the panning of its modules based on settings and the modules’ behaviours.

Modules and behaviour components #

The different modules we added support for are image-sequence, still image, parallax scene, focus-shift scene and text.

The parallax scene module has an opaque background with a custom number of layers that listens to the viewport progress for exact positions.

The focus-shift scene is a variant of the parallax bucket, with the addition that we use two images for each layer which fades in and out to simulate a focus change. We tried to use the blur filter, but it’s still to expensive, so we’ll wait for CSS shaders for this.

The content in the text module is drag-enabled with the TweenMax plugin Draggable. You can also use the scrollwheel or two-finger swipe to scroll vertically. Note the throw-props-plugin that adds the fling-style physics when you swipe and release.

The modules can also have different behaviours that are added as a set of components. They all have their own target selectors and settings. Translate to move an element, scale to zoom, hotspots for info overlay, debug metrics for testing visually, a start-title overlay, a flare layer, and some more. These will be appended to the DOM or controlling their target element inside the module.

With this in place we can create the different locations with just a config file that defines what assets to load and setup the different kinds of modules and components.

Image sequences #

The most challenging of the modules from a performance and a download-size aspect is the image sequence. There’s a bunch to read about this topic. On mobile and tablets we replace this with a still image. It’s too much data to decode and store in memory if we want decent quality on mobile. We tried multiple alternative solutions; using a background image and a spritesheet first, but it led to memory problems and lag when the GPU needed to swap between spritesheets. Then we tried swapping img elements, but it was also too slow. Drawing a frame from a spritesheet to a canvas was the most performant, so we began optimizing that. To save computation time each frame, the image data to write into the canvas is pre-processed via a temporary canvas and saved with putImageData() to an array, decoded and ready to use. The original spritesheet can then be garbage collected, and we store only the minimum amount of data needed in memory. Maybe it’s actually less to store undecoded images, but we get better performance while scrubbing the sequence this way. The frames are pretty small, just 640x400, but those will just be visible during scrubbing. When you stop, a high-res image loads and quickly fades in.

var canvas = document.createElement('canvas');
canvas.width = imageWidth;
canvas.height = imageHeight;

var ctx = canvas.getContext('2d');
ctx.drawImage(sheet, 0, 0);

var tilesX = imageWidth / tileWidth;
var tilesY = imageHeight / tileHeight;

var canvasPaste = canvas.cloneNode(false);
canvasPaste.width = tileWidth;
canvasPaste.height = tileHeight;

var i, j, canvasPasteTemp, imgData, 
var currentIndex = 0;
var startIndex = index * 16;
for (i = 0; i < tilesY; i++) {
  for (j = 0; j < tilesX; j++) {
    // Store the image data of each tile in the array.
    canvasPasteTemp = canvasPaste.cloneNode(false);
    imgData = ctx.getImageData(j * tileWidth, i * tileHeight, tileWidth, tileHeight);
    canvasPasteTemp.getContext('2d').putImageData(imgData, 0, 0);

    list[ startIndex + currentIndex ] = imgData;

    currentIndex++;
  }
}

The sprite-sheets are generated with Imagemagick. Here is a simple example on GitHub that shows how to create a spritesheet of all images inside a folder.

Animating the modules #

To place the modules on the timeline, a hidden representation of the timeline, displayed offscreen, keeps track on the ‘playhead’ and the width of the timeline. This can be done with just code, but it was good with a visual representation when developing and debugging. When running for real it’s just updated on resize to set dimensions. Some modules fills the viewport and some have their own ratio, so it was a little tricky to scale and position everything in all resolutions so everything is visible and not cropped too much. Each module has two progress indicators, one for the visible position on screen and one for the duration of the module itself. When making parallax movement it’s often hard to calculate start- and end-position of objects to sync with the expected position when it’s in view. It’s good to know exactly when a module enters the view, plays its internal timeline and when it animates out of view again.

Each module has a subtle black layer on top that adjusts its opacity so it’s fully transparent when it’s in the center position. This helps you to focus on one module at a time, which enhances the experience.

Page performance #

Moving from a functioning prototype to a jank-free release version means going from guessing to knowing of what happens in the browser. This is where Chrome DevTools is your best friend.

We have spent quite a lot of time optimising the site. Forcing hardware-acceleration is one of the most important tools of course to get smooth animations. But also hunting colorful columns and red rectangles in Chrome DevTools. There are many good articles about the topics, and you should read them all. The reward for removing skipping frames is instant, but so is the frustration when they return again. And they will. It's an ongoing process that needs iterations.

I like to use TweenMax from Greensock for tweening properties, transforms and CSS. Think in containers, visualise your structure as you add new layers. Keep in mind that existing transforms can be overwritten by new transforms. The translateZ(0) that forced hardware acceleration in your CSS class is replaced by a 2D matrix if you tween 2D values only. To keep the layer in acceleration mode in those cases, use the property “force3D:true” in the tween to make a 3D matrix instead of a 2D matrix. It’s easy to forget when you combine CSS and JavaScript tweens to set styles.

Don’t force hardware acceleration where it’s not needed. GPU memory can quickly fill up and cause unwanted results when you want to hardware-accelerate many containers, especially on iOS where memory have more constraints. To load smaller assets and scale them up with css and disable some of the effects in mobile mode made huge improvements.

Memory leaks was another field we needed to improve our skills in. When navigating between the different WebGL experiences a lot of objects, materials, textures and geometry are created. If those are not ready for garbage collection when you navigate away and remove the section they will probably cause the device to crash after a while when it runs out of memory.

Exiting a section with a failing dispose function.

Much better!

To find the leak it was pretty straight forward workflow in DevTools, recording the timeline and capturing heap snapshots. It’s easier if there are specific objects, like 3D geometry or a specific library, that you can filter out. In the example above it turned out that the 3D scene was still around and also an array that stored geometry was not cleared. If you find it hard to locate where the object lives, there is a nice feature that let you view this called retaining paths. Just click the object you want to inspect in the heap snapshot and you get the information in a panel below. Using a good structure with smaller objects helps when locating your references.

The scene was referenced in the EffectComposer.

In general, it's healthy to think twice before you manipulate the DOM. When you do, think about efficiency. Don't manipulate the DOM inside a game loop if you can help it. Store references in variables for reuse. If you need to search for an element, use the shortest route by storing references to strategic containers and searching inside the nearest ancestor element.

Delay reading dimensions of newly added elements or when removing/adding classes if you experience layout bugs. Or make sure Layout is triggered. Sometimes the browser batch changes to styles, and will not update after the next layout trigger. This can really be a big problem sometimes, but it’s there for a reason, so try to learn how it’s working behind the scenes and you will gain a lot.

Fullscreen #

When available, you have the option to put the site in fullscreen-mode in the menu via the Fullscreen API. But on devices there is also the browsers decision to put it into fullscreen. Safari on iOS had previously a hack to let you control that, but that is not available anymore so you have to prepare your design to work without it when making a non-scrolling page. We can probably expect updates on this in future updates, since it has broke a lot of web-apps.

Assets #

Animated instructions for the experiments.

Throughout the site we have a lot of different types of assets, we use images (PNG and JPEG), SVG (inline and background), spritesheets (PNG), custom icon fonts and Adobe Edge animations. We use PNGs for assets and animations (spritesheets) where the element can't be vector based, otherwise we try to use SVGs as much as possible.

The vector format means no loss of quality, even if we scale it. 1 file for all devices.

Small file size.
We can animate each part separately (perfect for advanced animations). As an example we hide the "subtitle" of the Hobbit logo (the desolation of Smaug) when it's scaled down.
It can be embedded as an SVG HTML tag or used as a background-image with no extra loading (it’s loaded the same time as the html page).

Icon typefaces have the same advantages as SVG when it comes to scalability and are used instead of SVG for small elements like icons on which we only need to be able to change the colour (hover, active, etc.). The icons are also very easy to reuse, you just need to set the CSS "content" property of an element.

Animations #

In some cases animating SVG elements with code can be very time consuming, especially when the animation needs to be changed a lot during the design process. To improve the workflow between designers and developers we use Adobe Edge for some animations (the instructions before the games). The animation workflow is really close to Flash and that helped the team but there are a few drawbacks, especially with integrating the Edge animations in our asset loading process since it comes with it’s own loaders and implementation logic.

I still feel we have a long way to go before we have a perfect workflow for handling assets and handmade animations on the web. We’re looking forward to seeing how tools like Edge will evolve. Feel free to add suggestions on other animation tools and workflows in the comments.

Conclusion #

Now when all the parts of the project are released and we look at the final result I must say we are quite impressed with the state of modern mobile browsers. When we started off this project we had much lower expectations on how seamless, integrated and performant we would be able to make it. It's been a great learning experience for us and all the time spent iterating and testing (a lot) has improved our understanding of how modern browsers work. And that's what it will take if we want to shorten the production time on these types of projects, going from guessing to knowing.

The Hobbit Experience

2013-11-20T00:00:00Z

Historically, bringing interactive, web-based, multimedia-heavy experiences to mobiles and tablets has been a challenge. The main constraints have been performance, API availability, limitations in HTML5 audio on devices and the lack of seamless inline video playback.

Earlier this year we started a project with friends from Google and Warner Bros. to make a mobile-first web experience for the new Hobbit movie, The Hobbit: The Desolation of Smaug. Building a multimedia-heavy mobile Chrome Experiment has been a really inspiring and challenging task.

The experience is optimized for Chrome for Android on the new Nexus devices where we now have access to WebGL and Web Audio. However a large portion of the experience is accessible on non-WebGL devices and browsers as well thanks to hardware-accelerated compositing and CSS animations.

The whole experience is based on a map of Middle-earth and the locations and characters from the Hobbit movies. Using WebGL made it possible for us to dramatize and explore the rich world of the Hobbit trilogy and let the users control the experience.

Challenges of WebGL on mobile devices #

First, the term "mobile devices" is very broad. The specs for devices vary a lot. So as a developer you need to decide if you want to support more devices with a less complex experience or, as we did in this case, limit the supported devices to those able to display a more realistic 3D world. For “Journey through Middle-earth” we focused on Nexus devices and five popular Android smartphones.

In the experiment, we used three.js as we’ve done for some of our previous WebGL projects. We started implementation by building an initial version of the Trollshaw game that would run well on the Nexus 10 tablet. After some initial testing on the device, we had a list of optimizations in mind that looked much like what we normally would use for a low-spec laptop:

Use low-poly models
Use low-res textures
Reduce the number of drawcalls as much as possible by merging geometry
Simplify materials and lighting
Remove post effects and turn off antialiasing
Optimise Javascript performance
Render the WebGL canvas at half size and scale up with CSS

After applying these optimizations to our first rough version of the game, we had a steady frame rate of 30FPS that we were happy with. At that point our goal was to improve the visuals without negatively impacting frame rate. We tried many tricks: some turned out to really have an impact on performance; a few didn’t have as big an effect as we’d hoped.

Use low-poly models #

Let’s start with the models. Using low-poly models certainly helps download time, as well as the time it takes to initialize the scene. We found that we could increase the complexity quite a lot without affecting performance much. The troll models we use in this game are about 5K faces and the scene is around 40K faces and that works fine.

One of the trolls of Trollshaw forest

For another (not yet released) location in the experience we saw more impact on performance from reducing polygons. In that case we loaded lower-polygon objects for mobile devices than the objects we loaded for desktop. Creating different sets of 3D models requires some extra work and isn’t always required. It really depends on how complex your models are to begin with.

When working on big scenes with a lot of objects, we tried to be strategic on how we divide the geometry. This enabled us to switch less important meshes on and off quickly, to find a setting that worked for all mobile devices. Then, we could choose to merge the geometry in JavaScript at runtime for dynamic optimization or to merge it in pre-production to save requests.

Use low-res textures #

To reduce load time on mobile devices, we chose to load different textures that were half the size of the textures on desktop. It turns out that all devices can handle texture sizes up to 2048x2048px and most can handle 4096x4096px. Texture lookup on the individual textures doesn’t seem to be a problem once they are uploaded to GPU. The total size of the textures must fit in the GPU memory to avoid having textures constantly up- and downloaded, but this is probably not a big problem for most web-experiences. However, combining textures into as few spritesheets as possible is important to reduce the number of drawcalls - this is something that has a big impact on performance on mobile devices.

Texture for one of the trolls of Trollshaw forest
(original size 512x512px)

Simplify materials and lighting #

The choice of materials can also greatly affect performance and must be managed wisely on mobile. Using MeshLambertMaterial (per vertex light calculation) in three.js instead of MeshPhongMaterial (per texel light calculation) is one thing we used to optimise performance. Basically we tried to use as simple shaders with as few lighting calculations as possible.

To see how the materials you use affect a scene’s performance, you can override the materials of the scene with a MeshBasicMaterial . This will give you a good comparison.

scene.overrideMaterial = new THREE.MeshBasicMaterial({color:0x333333, wireframe:true});

Optimise JavaScript performance #

When building games for mobile, the GPU isn’t always the biggest hurdle. A lot of time is spent on the CPU, especially physics and skeletal animations. One trick that helps sometimes, depending on the simulation, is to only run these expensive calculations every other frame. You can also use available JavaScript optimisation techniques when it comes to object pooling, garbage collection and object creation.

Updating pre-allocated objects in loops instead of creating new objects is an important step to avoid garbage collection "hiccups" during the game.

For example, consider code like this:

var currentPos = new THREE.Vector3();

function gameLoop() {
  currentPos = new THREE.Vector3(0+offsetX,100,0);
}

An improved version of this loop avoids creating new objects that must be garbage collected:

var originPos = new THREE.Vector3(0,100,0);
var currentPos = new THREE.Vector3();
function gameLoop() {
  currentPos.copy(originPos).x += offsetX;
  //or
  currentPos.set(originPos.x+offsetX,originPos.y,originPos.z);
}

As much as possible, event handlers should only update properties, and let the requestAnimationFrame render-loop handle updating the stage.

Another tip is to optimise and/or pre-calculate ray-casting operations. For example, if you need to attach an object to a mesh during a static path movement, you can "record" the positions during one loop and then read from this data instead of ray-casting against the mesh. Or as we do in the Rivendell experience, ray-cast to look for mouse interactions with a simpler low-poly invisible mesh. Searching for collisions on a high-poly mesh is very slow and should be avoided in a game-loop in general.

Render the WebGL canvas at half size and scale up with CSS #

The size of the WebGL canvas is probably the single most effective parameter you can tweak to optimise performance. The bigger the canvas you use to draw your 3D scene, the more pixels have to be drawn on every frame. This of course affects performance.The Nexus 10 with its high-density 2560x1600 pixel display has to push 4 times the number of pixels as a low-density tablet. To optimise this for mobile we use a trick where we set the canvas to half the size (50%) and then scale it up to its intended size (100%) with hardware-accelerated CSS 3D transforms. The downside of this is a pixelated image where thin lines can become a problem but on a high-res screen the effect isn’t that bad. It’s absolutely worth the extra performance.

The same scene without canvas scaling on the Nexus 10 (16FPS) and scaled to 50% (33FPS).

Objects as building blocks #

To be able to create the big maze of the Dol Guldur castle and the never ending valley of Rivendell we made a set of building block 3D models that we re-use. Re-using objects lets us ensure that objects are instantiated and uploaded at the start of the experience, and not in the middle of it.

3D object building blocks used in the maze of Dol Guldur.

In Rivendell we have a number of ground sections that we constantly reposition in Z-depth as the user’s journey progresses. As the user passes sections, these are repositioned in the far distance.

For the Dol Guldur castle we wanted the maze to be regenerated for every game. To do this we created a script that regenerates the maze.

Merging the whole structure into one big mesh from the beginning results in a very big scene and poor performance. To address this, we decided to hide and show the building blocks depending on whether they are in view. Right from the start, we had an idea about using a 2D raycaster script but in the end we used the built-in three.js frustrum culling. We reused the raycaster script to zoom in on the "danger" the player is facing.

The next big thing to handle is user interaction. On desktop you have mouse and keyboard input; on mobile devices your users interact with touch, swipe, pinch, device orientation etc.

Using touch interaction in mobile web experiences #

Adding touch support isn’t difficult. There are great articles to read about the topic. But there are some small things that can make it more complicated.

You can have both touch and mouse. The Chromebook Pixel and other touch enabled laptops have both mouse and touch support. One common mistake is to check if the device is touch enabled and then only add touch event listeners and none for mouse.

Don’t update rendering in event listeners. Save the touch events to variables instead and react to them in the requestAnimationFrame render loop. This improves performance and also coalesce conflicting events. Make sure that you re-use objects instead of creating new objects in the event listeners.

Remember that it’s multitouch: event.touches is an array of all touches. In some cases it’s more interesting to look at event.targetTouches or event.changedTouches instead and just react to the touches you are interested in. To separate taps from swipes we use a delay before we check if the touch has moved (swipe) or if it’s still (tap). To get a pinch we measure the distance between the two initial touches and how that changes over time.

In a 3D world you have to decide how your camera reacts to mouse vs. swipe actions. One common way of adding camera movement is to follow the mouse movement. This can be done with either direct control using mouse position or with a delta movement (position change). You don’t always want the same behaviour on a mobile device as a desktop browser. We tested extensively to decide what felt right for each version.

When dealing with smaller screens and touchscreens you’ll find that the user’s fingers and UI interaction graphics are often in the way of what you want to show. This is something we are used to when designing native apps but haven’t really had to think about before with web experiences. This is a real challenge for designers and UX designers.

Summary #

Our overall experience from this project is that WebGL on mobile works really well, especially on newer, high-end devices. When it comes to performance it seems like polygon count and texture size mostly affect download and initialization times and that materials, shaders and the size of the WebGL canvas are the most important parts to optimise for mobile performance. However, it’s the sum of the parts that affects the performance so everything you can do to optimise counts.

Targeting mobile devices also means that you have to get used to thinking about touch interactions and that it’s not only about the pixel size - it’s the physical size of the screen as well. In some cases we had to move the 3D camera closer to actually see what was going on.

The experiment is launched and it’s been a fantastic journey. Hope you enjoy it!

Want to try it out? Take your own Journey to Middle-Earth.